APPLICATION OF A SURROUND-VIEW CAMERA SYSTEM FOR MOBILE ROBOT MOVEMENT SAFETY
Abstract
This paper addresses the problem of ensuring the motion safety of a mobile robotic platform in an environment with known object classes, based on object detection results from a surround-view camera system. While solutions based on optical flow or other methods for detecting moving objects in a camera's field of view are well-known, this work proposes to use the results from a dedicated object detector for safety assurance. Here, we propose to utilize object positions, calculated from the output of a neural network detector, for safety purposes. The surround-view camera system (SVS) consists of 4 cameras with wide-angle lenses. Several approaches for feeding objects into the neural network detector are considered. Differences in quality and performance for these approaches are demonstrated, and methodological recommendations for their use are formulated. To calculate object positions in the camera coordinate system and, through camera positions, in the robot's coordinate frame, 2 solutions based on camera calibration and certain assumptions are proposed. In the first case, the robot's position on the surface is assumed to be horizontal, and the camera's height above the surface is assumed constant and known. In the second case, one of the metric dimensions of objects for each class are assumed to be known in advance. The proposed solution has been deployed and tested on a Rockchip 3588-based computing platform, demonstrating high performance in terms of detection count (from 8 to 19 objects per frame on average, depending on settings) and processing speed (0.29 s for 8 objects per frame and about 1.05 s for the result of 19 objects per frame). Regarding the accuracy of distance estimation to objects, for the first method, the standard deviation ranged from 4.2 to 7.9 mm, and for the second method, from 3.8 to 7.6 mm. The standard deviation of object size estimation for the first method ranged from 0.64 to 2.02 mm, and for the second method from 1.63 to 2.22 mm, respectively. The obtained results allow us to confidently state that the proposed object detection algorithm is applicable for ensuring the safety and navigation of a mobile robot using surround-view system cameras
References
1. Polovko S.A., Smirnova E.Yu., Stepanov D.N. Intellektual'nye sistemy tekhnicheskogo zreniya dlya be-zopasnosti i navigatsii [Intelligent machine vision systems for safety and navigation], Mekhatronika, avtomatizatsiya, upravlenie [Mechatronics, Automation, Control], 2009, No. 3, pp. 33-39.
2. Asvadi A., et al. 3D Lidar-based static and moving obstacle detection in driving environments: An ap-proach based on voxels and multi-region ground planes, Robotics and Autonomous Systems, 2016, Vol. 83, pp. 299-311.
3. Sekine T. The image of automobile safety, International Association of Traffic and Safety Sciences, In-terdisciplinary Wisdom of IATSS, 2015, pp. 62.
4. Hamsini S., Kathiresh M. Automotive safety systems, Automotive Embedded Systems: Key Technolo-gies, Innovations, and Applications. Cham: Springer International Publishing, 2021, pp. 1-18.
5. Kazak A.N., et al. Service robotics development trends in the Russian Federation, AIP Conference Pro-ceedings. AIP Publishing LLC, 2021, Vol. 2402, No. 1, pp. 070020.
6. Scaramuzza D., Martinelli A., Siegwart R. A toolbox for easily calibrating omni-directional cameras, 2006 IEEE/RSJ International Conference on Intelligent Ro-bots and Systems. IEEE, 2006, pp. 5695-5701.
7. Park K.T., et al. Training YOLOv8 Object Detection Model with Synthetic Images, 2024 24th Interna-tional Conference on Control, Automation and Systems (IC-CAS). IEEE, 2024, pp. 524-528.
8. OpenImages V7 Description. Available at: https://storage.googleapis.com/openimages/web/ factsfig-ures_v7.html (accessed 6 February 2011).
9. Meng M., et al. Distortion-aware room layout estimation from a single fisheye image, 2021 IEEE Inter-national Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 2021, pp. 441-449.
10. Meng M., et al. Distortion-aware room layout estimation from a single fisheye image, 2021 IEEE Inter-national Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 2021, pp. 441-449.
11. Yan F., et al. Monocular catadioptric panoramic depth estimation via improved end-to-end neural net-work model, Frontiers in Neurorobotics, 2023, Vol. 17, pp. 1278986.
12. Simioni E., et al. Geometrical calibration for the panrover: A stereo omnidirectional system for planetary rover, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sci-ences, 2020, Vol. 43, pp. 1151-1158.
13. Coates A.J., et al. The PanCam instrument for the ExoMars rover, Astrobiology, 2017, Vol. 17, No. 6-7, pp. 511-541.
14. Ullah H., et al. Automatic 360 mono-stereo panorama generation using a cost-effective multi-camera system, Sensors, 2020, Vol. 20, No. 11, pp. 3097.
15. Cui H., Zhao Z., Zhang F. Research on Panorama Generation from a Multi-Camera System by Object-Distance Estimation, Applied Sciences, 2023, Vol. 13, No. 22, pp. 12309.
16. Prudnikov N.V., Shlishevskiy V.B. Panoramnye optiko-elektronnye ustroystva krugovogo i sektornogo obzora [Panoramic optoelectronic devices for circular and sectorial surveillance], Vestnik SGUGiT [Bul-letin of SSUGT], 2016, No. 1 (33), pp. 148-161.
17. Kakani V., Kim H. Adaptive self-calibration of fisheye and wide-angle cameras, TENCON 2019-2019 IEEE Region 10 Conference (TENCON). IEEE, 2019, pp. 976-981.
18. Douskos V., Kalisperakis I., Karras G. Automatic calibration of digital cameras using planar chess-board patterns, Proceedings of the 8th Conference on Op-tical, 2007, pp. 9-12.
19. Jiang G., Quan L. Detection of concentric circles for camera calibration, Tenth IEEE International Con-ference on Computer Vision (ICCV'05). Vol. 1. IEEE, 2005, pp. 333-340.
20. Kannala J., Brandt S.S. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses, IEEE transactions on pattern analysis and machine intelligence, 2006, Vol. 28, No. 8, pp. 1335-1340.
21. Liu W., et al. Ssd: Single shot multibox detector, European conference on computer vision. Cham: Springer International Publishing, 2016, pp. 21-37.
22. Duan K., et al. Centernet: Keypoint triplets for object detection, Proceedings of the IEEE/CVF interna-tional conference on computer vision, 2019, pp. 6569-6578.
23. Alhasanat M.N., et al. Retinanet-based approach for object detection and dis-tance estimation in an im-age, International Journal on Communications Antenna and Propagation (IRECAP), 2021, Vol. 11, No. 1, pp. 1-9.
24. Tan M., Pang R., Le Q.V. Efficientdet: Scalable and efficient object de-tection, Proceedings of the IEEE/CVF conference on computer vision and pat-tern recognition, 2020, pp. 10781-10790.
25. Redmon J., Divvala S., Girshick R., Farhadi A. You Only Look Once: Unified, Real-Time Ob-ject De-tection, arXiv:1506.02640 [cs.CV], 2015. Available at: https://arxiv.org/abs/1506.02640 (accessed ac-cessed 13 February).
26. Ren S., He K., Girshick R., Sun J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, arXiv:1506.01497 [cs.CV], 2015. Available at: https://arxiv.org/pdf/1506.01497 (accessed 06 February 2026).
27. He K., Gkioxari G., Dollár P., Girshick R. Mask R-CNN // arXiv:1703.06870 [cs.CV], 2017. Available at: https://arxiv.org/pdf/1703.06870 (accessed accessed 7 February).
28. Sun P., et al. Sparse r-cnn: End-to-end object detection with learnable proposals, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14454-14463.
29. Shridhar A., Tomson P., Innes M. Interoperating deep learning models with onnx. Jl, Proceedings of the JuliaCon Conferences, 2020, Vol. 1, No. 1, pp. 59.
30. Jegham N., et al. Yolo evolution: A comprehensive benchmark and archi-tectural review of yolov12, yolo11, and their previous versions, arXiv preprint arXiv:2411.00201, 2024.
31. Ultralytics | Revolutionizing the World of Vision AI. Available at: https://www.ultralytics.com/ (дата обращения 02.02.2026).
32. GitHub – Pelochus/ezrknn-toolkit2. Making Rockchip’s RKNN-Toolkit-2 easier for SBCs like Orange Pi 5 or Radxa Rock 5. Available at: https://github.com/Pelochus/ezrknn-toolkit2 (accessed 2 February).
