Smart Control of a Microrobot for Navigation on Fluid Surface and Simulation of its Application in Microplastics Removal

Document Type : Research Paper

Authors

1 Department of Agricultural Machinery Engineering, Faculty of Agricultural, College of Agriculture and Natural Resources, University of Tehran, Karaj, Iran

2 Department of Medical Nanotechnology, School of Advanced Technologies in Medicine, Tehran University of Medical Sciences, Tehran, Iran.

Abstract

Microplastic contamination of food and beverages has become a global concern in recent years. As a novel approach, magnetic microrobots offer promising potential to address microplastic separation and degradation. However, achieving precise, intelligent, and automated navigation control for microrobots in such tasks remains a significant challenge. This level of control is typically achieved by modeling the complex dynamics of microrobots, the environment, and the actuation system. In this study, an alternative approach was presented using a model-free deep reinforcement learning algorithm (DRL) to navigate a magnetic microrobot on fluid surfaces. In order to simulate the process of reaching a microplastic particle on the fluid surface, the DRL system was implemented to train the microrobot to autonomously navigate from an initial position within the real-world environment to a specified target position. A magnetic actuation system based on two permanent magnets and one-axis Helmholtz coils was constructed to manipulate the position of the microrobot. During the training phase, the microrobot demonstrated high accuracy and speed in achieving the desired position. The evaluation results of the trained model also confirmed the microrobot's success in all episodes, with an average reward of 39.02 out of 40 and a standard deviation of 0.71. These findings indicate that the control system could effectively learn an optimal policy by employing DRL without any prior knowledge of environmental dynamics or the actuation system.

Keywords

Main Subjects

Smart Control of a Microrobot for Navigation on Fluid Surface and Simulation of its Application in Microplastics Removal

 

EXTENDED ABSTRACT

 

Introduction

Over the past few decades. the presence of microplastics in foods and beverages has caused irreversible damage and disease to humans and other organisms. To address this issue, it is necessary to develop a sustainable and efficient method of separation and degradation of microplastics. There has been considerable interest in using microrobots in order to achieve this objective. Nevertheless, this approach faces a critical challenge in terms of automating, intelligent, and precise control and navigation. As traditional methods are typically based on approximate and empirical approaches, they require complex dynamic modeling of the environment and actuation systems in order to analyze microrobot behavior. As a result of remarkable advancements in artificial intelligence technology, reinforcement learning algorithms (RL) have been introduced as a potential alternative method of addressing the challenge of microrobot navigation control. The RL makes it possible to train agents by enabling them to interact with real-world environments. The objective of the microrobot was to reach the target point, simulating the process of approaching microplastic particles floating on the surface of a fluid. In this study, the hypothesis is to create a high-performance control system using RL, eliminating the need to develop a specialized modeling of the magnetic field or fluid dynamics.

Material and methods

In this study, a magnetic actuation system was constructed to control a disk-shaped magnetic microrobot in a real-world environment. Changing the angle of the magnets affects the magnetic field and gradient within the workspace, which in turn, affects the position of the microrobot floating on the fluid surface in the xy plane. In order to align the microrobot in the Z-direction, a Helmholtz coil was used to generate a uniform magnetic field. In order to detect and determine the microrobot's position in the xy plane, an image processing procedure was employed. In this study, a Soft Actor-Critic algorithm (SAC) was utilized for microrobot control. Due to its sample-efficient capability, SAC is considered as a suitable choice, particularly in real-world scenarios. The training process was conducted through two repetitions. After observing the microrobot's state, the agent took action, and then the control system provided a feedback in the form of a reward or penalty primarily based on the microrobot's distance from the target position.

Results and discussion

Through 10,000 training steps, the SAC algorithm evaluated all available actions within the environment to determine the optimal policy for microrobot control. As a result, the agent enhances its actions in order to achieve an optimal control strategy. During microrobot training, there was a noticeable trend of shorter episode lengths and higher average rewards. These results show that implementing this optimal policy for the microrobot resulted in following a shorter and quicker path to reach the microplastic position.

Conclusions

It was demonstrated that the SAC algorithm could effectively achieve an optimal control strategy for microrobot navigation without requiring any prior knowledge of environmental dynamics, microrobot behavior, or actuation systems.

References

Agrahari, V., Agrahari, V., Chou, M.-L., Chew, C. H., Noll, J., & Burnouf, T. (2020). Intelligent micro-/nanorobots as drug and cell carrier devices for biomedical therapeutic advancement: Promising development opportunities and translational challenges. Biomaterials, 260, 120163.
Amoudruz, L., & Koumoutsakos, P. (2022). Independent Control and Path Planning of Microswimmers with a Uniform Magnetic Field. Advanced Intelligent Systems, 4(3), 2100183.
Bae, H., Paludan, M., Knoblauch, J., & Jensen, K. H. (2021). Neural networks and robotic microneedles enable autonomous extraction of plant metabolites. Plant Physiology, 186(3), 1435–1441.
Behrens, M. R., & Ruder, W. C. (2022). Smart Magnetic Microrobots Learn to Swim with Deep Reinforcement Learning. Advanced Intelligent Systems, 4(10), 2270049.
Beladi-Mousavi, S. M., Hermanová, S., Ying, Y., Plutnar, J., & Pumera, M. (2021). A Maze in Plastic Wastes: Autonomous Motile Photocatalytic Microrobots against Microplastics. ACS Applied Materials & Interfaces, 13(21), 25102–25110.
Bellman, R. (1957). A Markovian decision process. Journal of Mathematics and Mechanics, 679–684.
Borra, F., Biferale, L., Cencini, M., & Celani, A. (2022). Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number. Physical Review Fluids, 7(2), 023103.
Cai, M., Wang, Q., Qi, Z., Jin, D., Wu, X., Xu, T., & Zhang, L. (2022). Deep Reinforcement Learning Framework-Based Flow Rate Rejection Control of Soft Magnetic Miniature Robots. IEEE Transactions on Cybernetics, 1–13.
Campuzano, S., Orozco, J., Kagan, D., Guix, M., Gao, W., Sattayasamitsathit, S., Claussen, J. C., Merkoçi, A., & Wang, J. (2012). Bacterial Isolation by Lectin-Modified Microengines. Nano Letters, 12(1), 396–401.
Choi, J., Hwang, J., Kim, J. young, & Choi, H. (2021). Recent Progress in Magnetically Actuated Microrobots for Targeted Delivery of Therapeutic Agents. Advanced Healthcare Materials, 10(6), 1–24.
Colabrese, S., Gustavsson, K., Celani, A., & Biferale, L. (2017). Flow Navigation by Smart Microswimmers via Reinforcement Learning. Physical Review Letters, 118(15), 158004.
Dan, J., Shi, S., Sun, H., Su, Z., Liang, Y., Wang, J., & Zhang, W. (2022). Micro/nanomotor technology: the new era for food safety control. Critical Reviews in Food Science and Nutrition, 1–21.
de Jesus, J. C., Kich, V. A., Kolling, A. H., Grando, R. B., Cuadros, M. A. de S. L., & Gamarra, D. F. T. (2021). Soft Actor-Critic for Navigation of Mobile Robots. Journal of Intelligent & Robotic Systems, 102(2), 31.
Ding, Z., Huang, Y., Yuan, H., & Dong, H. (2020). Introduction to Reinforcement Learning. In Deep Reinforcement Learning (pp. 47–123). Springer Singapore.
Dong, X., & Sitti, M. (2020). Controlling two-dimensional collective formation and cooperative behavior of magnetic microrobot swarms. The International Journal of Robotics Research, 39(5), 617–638.
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021). Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9), 2419–2468.
Gustavsson, K., Biferale, L., Celani, A., & Colabrese, S. (2017). Finding efficient swimming strategies in a three-dimensional chaotic flow by reinforcement learning. The European Physical Journal E, 40(12), 110.
Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., & Levine, S. (2018). Learning to walk via deep reinforcement learning. ArXiv Preprint ArXiv:1812.11103.
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In J. Dy & A. Krause (Eds.), Proceedings of the 35th International Conference on Machine Learning (Vol. 80, pp. 1861–1870). PMLR.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., & Abbeel, P. (2018). Soft actor-critic algorithms and applications. ArXiv Preprint ArXiv:1812.05905.
Huang, L., Rogowski, L., Kim, M. J., & Becker, A. T. (2017). Path planning and aggregation for a microrobot swarm in vascular networks using a global input. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 414–420.
Jiang, J., Yang, L., & Zhang, L. (2023). DQN-based on-line Path Planning Method for Automatic Navigation of Miniature Robots. 2023 IEEE International Conference on Robotics and Automation (ICRA), 5407–5413.
Jiang, J., Yang, Z., Ferreira, A., & Zhang, L. (2022). Control and Autonomy of Microrobots: Recent Progress and Perspective. Advanced Intelligent Systems, 4(5), 2100279.
Khalesi, R., Yousefi, M., Nejat Pishkenari, H., & Vossoughi, G. (2022). Robust independent and simultaneous position control of multiple magnetic microrobots by sliding mode controller. Mechatronics, 84, 102776.
Kober, J., Bagnell, J. A., & Peters, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11), 1238–1274.
Lee, J., & Ha, J.-I. (2017). Direction Priority Control Method for Magnetic Manipulation System in Current and Voltage Limits. IEEE Transactions on Industrial Electronics, 64(4), 2914–2923.
Leslie, H. A., van Velzen, M. J. M., Brandsma, S. H., Vethaak, A. D., Garcia-Vallejo, J. J., & Lamoree, M. H. (2022). Discovery and quantification of plastic particle pollution in human blood. Environment International, 163, 107199.
Lynch, K. M., & Park, F. C. (2017). Modern robotics. Cambridge University Press.
Magnetics, Magnetop. (2023). Neodymium Cube - 20mm x 20mm x 20mm - N42. https://www.magnetop.com.au/neodymium-cube-20mm-x-20mm-x-20mm-n42
Mayorga‐Martinez, C. C., Castoralova, M., Zelenka, J., Ruml, T., & Pumera, M. (2023). Swarming Magnetic Microrobots for Pathogen Isolation from Milk. Small, 19(6), 2205047.
Medina-Sánchez, M., & Schmidt, O. G. (2017). Medical microbots need better imaging and control. Nature, 545(7655), 406–408.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533.
Nauber, R., Goudu, S. R., Goeckenjan, M., Bornhäuser, M., Ribeiro, C., & Medina-Sánchez, M. (2023). Medical microrobots in reproductive medicine from the bench to the clinic. Nature Communications, 14(1), 728.
Parmar, J., Vilela, D., Villa, K., Wang, J., & Sánchez, S. (2018). Micro- and Nanomotors as Active Environmental Microcleaners and Sensors. Journal of the American Chemical Society, 140(30), 9317–9331.
Pawashe, C., Floyd, S., Diller, E., & Sitti, M. (2012). Two-Dimensional Autonomous Microparticle Manipulation Strategies for Magnetic Microrobots in Fluidic Environments. IEEE Transactions on Robotics, 28(2), 467–477.
Qiu, J., Huang, W., Xu, C., & Zhao, L. (2020). Swimming strategy of settling elongated micro-swimmers by reinforcement learning. Science China Physics, Mechanics & Astronomy, 63(8), 284711.
Rahmer, J., Stehning, C., & Gleich, B. (2017). Spatially selective remote magnetic actuation of identical helical micromachines. Science Robotics, 2(3).
Salmani, Y., Mousazadeh, H., Alimardani, R., Jafarbiglu, H., Omrani, E., Makhsoos, A., & Kiapey, A. (2018). Evaluation of a Navigation Algorithm for Robot Boat and Comparison to Simulation Results. Iranian Journal of Biosystem Engineering, 49(3), 353–366. (In Persian)
Sehyuk Yim, Gultepe, E., Gracias, D. H., & Sitti, M. (2014). Biopsy using a Magnetic Capsule Endoscope Carrying, Releasing, and Retrieving Untethered Microgrippers. IEEE Transactions on Biomedical Engineering, 61(2), 513–521.
Singh, B., Kumar, R., & Singh, V. P. (2022). Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review, 55(2), 945–990.
Sitti, M. (2017). Mobile Microrobotics (R. Arkin (ed.)). The MIT Press.
Sun, M., Chen, W., Fan, X., Tian, C., Sun, L., & Xie, H. (2020). Cooperative recyclable magnetic microsubmarines for oil and microplastics removal from water. Applied Materials Today, 20, 100682.
Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
TowerPro. (2023). MG996R. https://www.towerpro.com.tw/product/mg996r/
Tsang, Alan C. H., Demir, E., Ding, Y., & Pak, O. S. (2020). Roads to Smart Artificial Microswimmers. Advanced Intelligent Systems, 2(8), 1900137.
Tsang, Alan Cheng Hou, Tong, P. W., Nallan, S., & Pak, O. S. (2020). Self-learning how to swim at low Reynolds number. Physical Review Fluids, 5(7), 074101.
Urso, M., Ussia, M., & Pumera, M. (2021). Breaking Polymer Chains with Self‐Propelled Light‐Controlled Navigable Hematite Microrobots. Advanced Functional Materials, 31(28).
Urso, M., Ussia, M., & Pumera, M. (2023). Smart micro- and nanorobots for water purification. Nature Reviews Bioengineering.
Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1).
Vilela, D., Stanton, M. M., Parmar, J., & Sánchez, S. (2017). Microbots Decorated with Silver Nanoparticles Kill Bacteria in Aqueous Media. ACS Applied Materials & Interfaces, 9(27), 22093–22100.
Villa, K., Vyskočil, J., Ying, Y., Zelenka, J., & Pumera, M. (2020). Microrobots in Brewery: Dual Magnetic/Light‐Powered Hybrid Microrobots for Preventing Microbial Contamination in Beer. Chemistry – A European Journal, 26(14), 3039–3043.
Wang, L., Kaeppler, A., Fischer, D., & Simmchen, J. (2019). Photocatalytic TiO 2 Micromotors for Removal of Microplastics and Suspended Matter. ACS Applied Materials & Interfaces, 11(36), 32937–32944.
Wang, Q., Li, T., Fang, D., Li, X., Fang, L., Wang, X., Mao, C., Wang, F., & Wan, M. (2020). Micromotor for removal/detection of blood copper ion. Microchemical Journal, 158, 105125.
Wang, X., Hu, C., Pane, S., & Nelson, B. J. (2022). Dynamic Modeling of Magnetic Helical Microrobots. IEEE Robotics and Automation Letters, 7(2), 1682–1688.
Watkins, C. J. C. H. (1989). Learning from delayed rewards.
Wright, S. L., Thompson, R. C., & Galloway, T. S. (2013). The physical impacts of microplastics on marine organisms: A review. Environmental Pollution, 178, 483–492.
Xu, T., Yu, J., Yan, X., Choi, H., & Zhang, L. (2015). Magnetic Actuation Based Motion Control for Microrobots: An Overview. Micromachines, 6(9).
Yang, G.-Z., Bellingham, J., Dupont, P. E., Fischer, P., Floridi, L., Full, R., Jacobstein, N., Kumar, V., McNutt, M., Merrifield, R., Nelson, B. J., Scassellati, B., Taddeo, M., Taylor, R., Veloso, M., Wang, Z. L., & Wood, R. (2018). The grand challenges of Science Robotics. Science Robotics, 3(14).
Yang, Y., Bevan, M. A., & Li, B. (2020a). Efficient Navigation of Colloidal Robots in an Unknown Environment via Deep Reinforcement Learning. Advanced Intelligent Systems, 2(1), 1900106.
Yang, Y., Bevan, M. A., & Li, B. (2020b). Micro/Nano Motor Navigation and Localization via Deep Reinforcement Learning. Advanced Theory and Simulations, 3(6), 2000034.
Yang, Y., Bevan, M. A., & Li, B. (2022). Hierarchical Planning with Deep Reinforcement Learning for 3D Navigation of Microrobots in Blood Vessels. Advanced Intelligent Systems, 4(11), 2200168.
Yousefi, M., & Nejat Pishkenari, H. (2021). Independent position control of two identical magnetic microrobots in a plane using rotating permanent magnets. Journal of Micro-Bio Robotics, 17(1), 59–67.
Zhang, H., & Yu, T. (2020). Taxonomy of Reinforcement Learning Algorithms. In Deep Reinforcement Learning (pp. 125–133). Springer Singapore.
Zhou, H., Mayorga‐Martinez, C. C., & Pumera, M. (2021). Microplastic Removal and Degradation by Mussel‐Inspired Adhesive Magnetic/Enzymatic Microrobots. Small Methods, 5(9), 2100230.
Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A., Kumar, V., & Levine, S. (2020). The ingredients of real-world robotic reinforcement learning.
Zou, Z., Liu, Y., Young, Y.-N., Pak, O. S., & Tsang, A. C. H. (2022). Gait switching and targeted navigation of microswimmers via deep reinforcement learning. Communications Physics, 5(1), 158.
Iranian Journal of Biosystems Engineering
Volume 54, Issue 3
October 2023
Pages 75-94

Files

History

  • Receive Date: 17 October 2023
  • Revise Date: 02 January 2024
  • Accept Date: 07 January 2024

Share

How to cite

Statistics