METHOD FOR GENERATING A MOBILE ROBOT OCCUPANCY MAP FROM MULTISPECTRAL VISION SYSTEM DATA
Abstract
This paper addresses the problem of generating occupancy maps of the surrounding space for robotic
platforms using data from a multispectral vision system. The aim of the work is to qualitatively improve
the generated passability model by combining geometric and semantic data from lidars and stereo
cameras, as well as direct velocity measurements from millimeter-wave radars. The presented algorithms
and their modifications are universal to the data source and do not require physical synchronization of
sensors. The paper solves the problem of constructing both static a priori and real-time dynamic occupancy
maps. An approach for combining an a priori semantic map with the one generated in the runtime is
proposed. Approaches for accumulating and updating semantic information in the maps are described.
The problem of detecting dynamic obstacles in occupancy maps based on a modified particle filtering
algorithm is also considered. The combined method described in the paper increases the accuracy of dynamic
obstacle detection and enables correct obstacle detection even if the dynamics detection algorithm
fails. Metrics for quantifying occupancy maps are defined. The developed algorithm has been tested on
Semantic KITTI, nuScenes open datasets in the automotive data domain, and on a small service cleaning
robot both in the CARLA simulator and in real-world conditions with active pedestrian traffic. The software
implementation of the algorithm runs in real time on Jetson AGX Xavier and Jetson AGX Orin embedded
computers.
References
lidarov [Modified algorithm for constructing an occupancy map using a point cloud from several
lidars], Programmnye produkty i sistemy [Software products and systems], 2020, Vol. 33, No. 2.
2. Fleischmann P., Berns K. A Stereo Vision Based Obstacle Detection System for Agricultural Applications,
Field and Service Robotics: Results of the 10th International Conference, ed. by
D.S. Wettergreen, T.D. Barfoot. Cham: Springer International Publishing, 2016, pp. 217-231.
3. Huang J. et al. An online multi-lidar dynamic occupancy mapping method, 2019 IEEE Intelligent
Vehicles Symposium (IV). IEEE, 2019, pp. 517-522.
4. Jaspers H., Himmelsbach M., Wuensche H.J. Multi-modal local terrain maps from vision and lidar,
2017 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2017, pp. 1119-1125.
5. Cho H. [et al.]. A multi-sensor fusion system for moving object detection and tracking in urban driving
environments, 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE.
2014, pp. 1836-1843.
6. Nuss D. et al. A random finite set approach for dynamic occupancy grid maps with real-time application,
The International Journal of Robotics Research, 2018, Vol. 37, No. 8, pp. 841-866.
7. Danescu R., Nedevschi S. A Particle-Based Solution for Modeling and Tracking Dynamic Digital Elevation
Maps, IEEE Transactions on Intelligent Transportation Systems, 2014, Vol. 3, No. 15,
pp. 1002-1015.
8. Mohajerin N., Rohani M. Multi-step prediction of occupancy grid maps with recurrent neural networks,
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019,
pp. 10600-10608.
9. Liu R., Wang J., Zhang B. High definition map for automated driving: Overview and analysis, The
Journal of Navigation, 2020, Vol. 73, No. 2, pp. 324-341.
10. Nikdel P., Chen M., Vaughan R. Recognizing and Tracking High-Level, Human-Meaningful Navigation
Features of Occupancy Grid Maps, 2020 17th Conference on Computer and Robot Vision (CRV).
IEEE, 2020, pp. 62-69.
11. Roddick T., Cipolla R. Predicting semantic map representations from images using pyramid occupancy
networks, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
2020, pp. 11138-11147.
12. Li Y. [et al.]. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion,
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp.
9087-9098.
13. Zhang Y., Zhu Z., Du D. Occformer: Dual-path transformer for vision-based 3d semantic occupancy
prediction, Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023,
pp. 9433-9443.
14. Gan L. [et al.]. Bayesian spatial kernel smoothing for scalable dense semantic mapping, IEEE Robotics
and Automation Letters, 2020, Vol. 5, No. 2, pp. 790-797.
15. Shepel' I.O. Algoritm kompleksirovaniya neskol'kikh istochnikov dannykh v obshchuyu kartu
zanyatosti [Algorithm for integrating several data sources into a common employment map], Izvestiya
YuFU. Tekhnicheskie nauki [Izvestiya SFedU. Engineering Sciences], 2021, No. 3 (220), pp. 64-71.
16. Shepel' I.O. Obnaruzhenie dinamicheskikh ob"ektov na karte zanyatosti s nakopleniem na osnove
fil'tra chastits [Detection of dynamic objects on an occupancy map with accumulation based on a particle
filter], Izvestiya YuFU. Tekhnicheskie nauki [Izvestiya SFedU. Engineering Sciences], 2022, No.
3, pp. 192-201.
17. Shepel I. [et al.]. Occupancy Grid Generation With Dynamic Obstacle Segmentation in Stereo Images,
IEEE Transactions on Intelligent Transportation Systems, 2022, Vol. 23, No. 9, pp. 14779-14789.
18. Shepel' I.O. Postroenie modeli prokhodimosti okruzhayushchey sredy po oblaku tochek stereokamery
s ispol'zovaniem ierarkhicheskoy karty vysot [Construction of a model of environmental cross-country
ability using a point cloud of a stereo camera using a hierarchical height map], Inzhenernyy vestnik
Dona [Engineering Bulletin of the Don], 2018, 1 (48).
19. Thrun S. Learning occupancy grid maps with forward sensor models, Autonomous robots, 2003,
Vol. 15, No. 2, pp. 111-127.
20. Collins T., Collins J., Ryan D. Occupancy grid mapping: An empirical evaluation, 2007 mediterranean
conference on control & automation. IEEE, 2007, pp. 1-6.
21. Behley J. [et al.]. SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences,
Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
22. Caesar H. [et al.]. nuScenes: A Multimodal Dataset for Autonomous Driving, Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 06.2020.
23. Dosovitskiy A. [et al.]. CARLA: An Open Urban Driving Simulator, Proceedings of the 1st Annual
Conference on Robot Learning, 2017, pp. 1-16.
24. Mihelich P., Konolige K., Leibs J. stereo_image_proc / Open Source Robotics Foundation, 2024.
Available at: https: // github. com / ros - perception/image_pipeline/tree/rolling/stereo_image_proc.
25. Zhu Y. [et al.]. Improving semantic segmentation via video propagation and label relaxation, Proceedings
of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8856-8865.
