A METHOD OF CONTROLLING A MOBILE ROBOT USING NATURAL LANGUAGE SEMANTICS
Abstract
A large number of different interfaces can be used to control robots, from traditional remotes to augmented reality technologies. However, all such interfaces have a number of limitations, which are particularly acute in service robotics. They are associated with long-term training of a human operator, non-intuitive control for humans, and the need for full human involvement. On the other hand, a new direction has emerged today, related to large language models that are capable of processing natural language and then translating it into robot control commands. There are a number of works demonstrating the possibility of using language models in tasks of planning robot actions. Based on the analysis of existing work, a new method of controlling a mobile robot is proposed, combining the advantages of other methods. The method allows you to plan scenarios for the robot, receiving a natural language mission, the robot's TOP, and information from its sensors. The article also describes the sequence of configuring the system using a large language model to solve this problem. Three variants of instructions for the neural network are presented, which gradually improve the achievability of the generated scenarios. After that, various missions are described, which are set as part of experimental studies - a total of 100 missions were tested, divided into 4 levels of difficulty in equal proportions. The complexity of the missions ranged from describing objects in the robot's field of view to interacting with complex missions involving synonyms of objects and implicitly defined goals. At the end of the work, the results of the evaluation of the algorithm and three variants of the instruction are presented. The conclusion can be considered that the use of language models to assign scenarios to robots is possible, including with a sufficiently high achievement. The model with the most advanced instruction reached 91% of correctly formed scenarios, which suggests the applicability of the developed method for controlling a mobile robot in natural language
References
1. Sergeev A.V., Guk M.Yu. Upravlenie mobil'nym robotom kosmicheskogo naznacheniya s primeneniem virtual'noy real'nosti i silomomentnogo ochuvstvleniya [Control of a mobile space robot using virtual re-ality and force-moment sensing], Pilotiruemye polety v kosmos [Manned Space Flights], 2018,
No. 4, pp. 44.
2. Zhao W.X. et al. A survey of large language models, arXiv preprint arXiv:2303.18223, 2023, Vol. 1, No. 2.
3. Vaswani A. et al. Attention is all you need, Advances in neural information processing systems, 2017, Vol. 30.
4. Wang L. et al. A survey on large language model based autonomous agents, Frontiers of Computer Science, 2024, Vol. 18, No. 6, pp. 186345.
5. Chen Qian, Wei Liu, et al. Chatdev: Communicative agents for software development, In Proc. of ACL, 2024.
6. Nian Li, Chen Gao, Mingyu Li, Yong Li, and Qingmin Liao. Econagent: large language model-empowered agents for simulating macroeconomic activities, In Proc. of ACL, 2024.
7. Smirnova E.Yu., Spasskiy B.A. Organizatsiya supervizornogo upravleniya v stsenariyakh ekstremal'noy robototekhniki [Organization of supervisory control in scenarios of extreme robotics], Robototekhnika i tekhnicheskaya kibernetika [Robotics and Technical Cybernetics], 2020, Vol. 8, No. 4, pp. 245-258.
8. Ma Y. et al. A survey on vision-language-action models for embodied ai, arXiv preprint arXiv:2405.14093, 2024.
9. Yao S. et al. React: Synergizing reasoning and acting in language models, The eleventh international conference on learning representations, 2022.
10. Ahn M. et al. Do as i can, not as i say: Grounding language in robotic affordances, arXiv preprint arXiv:2204.01691, 2022.
11. Anwar A. et al. Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation, 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 2838-2845.
12. Huang S. et al. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model, arXiv preprint arXiv:2305.11176. – 2023.
13. Hazra R., Dos Martires P. Z., De Raedt L. Saycanpay: Heuristic planning with large language models using learnable domain knowledge, Proceedings of the AAAI Conference on Artificial Intelligence, 2024, Vol. 38, No. 18, pp. 20123-20133.
14. Song C.H. et al. Llm-planner: Few-shot grounded planning for embodied agents with large language models, Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 2998-3009.
15. Devlin J. et al. Bert: Pre-training of deep bidirectional transformers for language understanding, Pro-ceedings of the 2019 conference of the North American chapter of the association for computational lin-guistics: human language technologies. Vol. 1 (long and short papers), 2019, pp. 4171-4186.
16. Driess D, et al. Palm-e: An embodied multimodal language model, 2023.
17. Chowdhery A, et al. Palm: Scaling language modeling with pathways, Journal of Machine Learning Research, 2023, Vol. 24, No. 240, pp. 1-113.
18. Lapkin YA.D., Matveev V.D. Sravnitel'nyy analiz bol'shikh yazykovykh modeley dlya avtonomnykh robototekhnicheskikh sistem [Comparative analysis of large language models for autonomous robotic systems], Ekstremal'naya robototekhnika [Experimental Robotics], 2025, No. 1 (36), pp. 110-115. EDN FXTPGG.
19. Gemma Team: Aishwarya Kamath, et al. Google DeepMind. Gemma 3 Technical Report, arXiv:2503.19786, 2025.
20. Tianyi Zhang, et al. BERTScore: Evaluating Text Generation with BERT, arXiv:1904.09675, 2019.
21. Rviz User’s Guide, RViz 1.9.0 documentation. Available at: https://docs.ros.org/en/jade/api/rviz/ html/user_guide/ (accessed 10 February 2026).
