Search

Search Results

• APPLICATION OF A SURROUND-VIEW CAMERA SYSTEM FOR MOBILE ROBOT MOVEMENT SAFETY

  I.S. Fomin , А. А. Baseltsev

  2026-04-29

  Abstract ▼

  This paper addresses the problem of ensuring the motion safety of a mobile robotic platform in an environment with known object classes, based on object detection results from a surround-view camera system. While solutions based on optical flow or other methods for detecting moving objects in a camera's field of view are well-known, this work proposes to use the results from a dedicated object detector for safety assurance. Here, we propose to utilize object positions, calculated from the output of a neural network detector, for safety purposes. The surround-view camera system (SVS) consists of 4 cameras with wide-angle lenses. Several approaches for feeding objects into the neural network detector are considered. Differences in quality and performance for these approaches are demonstrated, and methodological recommendations for their use are formulated. To calculate object positions in the camera coordinate system and, through camera positions, in the robot's coordinate frame, 2 solutions based on camera calibration and certain assumptions are proposed. In the first case, the robot's position on the surface is assumed to be horizontal, and the camera's height above the surface is assumed constant and known. In the second case, one of the metric dimensions of objects for each class are assumed to be known in advance. The proposed solution has been deployed and tested on a Rockchip 3588-based computing platform, demonstrating high performance in terms of detection count (from 8 to 19 objects per frame on average, depending on settings) and processing speed (0.29 s for 8 objects per frame and about 1.05 s for the result of 19 objects per frame). Regarding the accuracy of distance estimation to objects, for the first method, the standard deviation ranged from 4.2 to 7.9 mm, and for the second method, from 3.8 to 7.6 mm. The standard deviation of object size estimation for the first method ranged from 0.64 to 2.02 mm, and for the second method from 1.63 to 2.22 mm, respectively. The obtained results allow us to confidently state that the proposed object detection algorithm is applicable for ensuring the safety and navigation of a mobile robot using surround-view system cameras

• SPIKING REPRESENTATIONS COMPARISON FOR LOCALIZATION AND NAVIGATION IN THE KEYFRAME MAP

  I.S. Fomin , V.D. Matveev , А.Е. Arkhipov

  273-284

  2025-10-01

  Abstract ▼

  The task of navigating a mobile robotic platform in a known environment has been efficiently solved for a long time and using a flat passability map, which is built using lidar. Nevertheless, situations regularly arise when, for one reason or another, the platform is not equipped with lidar or other active navigation tools. At the same time, a camera is usually installed on the robotic platform, designed for visual monitoring of the situation by the operator, which can also be used for navigation when moving the robot in a known environment. There are well-known examples of navigation algorithms based on the use of sequences of keyframes, for example, visual SLAM. At the same time, various variants of video images (blurred, masked, etc.) are considered as keyframes. In this paper, a cognitive (non-metric, non-spatial) map of keyframes representing a spiking representation of the observed images is considered as a base for navigation. The possibility of using neuromorphic information control elements developed at the RTC to compare the current spiking representation with all spiking representations of a key sequence is analyzed. It is shown that by such a comparison, the keyframe closest to the current one can be determined, and parameters for the shift of spiking representations can also be selected, which is an analog of localization and navigation for a cognitive map. The description of a software tool for emulating the construction of a map and moving in it for experimental testing of the proposed algorithms is given. Data collection and experimental evaluation of the quality of localization and navigation algorithms have been performed. To do this, we have collected several keyframe maps with different patterns of movement between frames. When determining the position of the frame in the map, the quality was from 70 to 98%, when determining the direction of displacement between frames, the accuracy was from 94 to 97%. The results obtained are assessed as sufficient to solve the tasks assigned to the algorithm.

• COMBINING SEGMENTATION, TRACKING, AND CLASSIFICATION MODELS TO SOLVE VIDEO ANALYTICS PROBLEMS

  V.D. Matveev, А. Е. Arkhipov, I. S. Fomin

  2025-04-27

  Abstract ▼

  The task of detecting obstacles in front of a mobile robot has been successfully solved long ago using
  laser and ultrasonic sensors. However, obstacles that are not detected by these types of sensors may endanger
  the safety of the robot. To detect them in the work, it is proposed to use a technical vision system
  (STZ), the information from which is processed by a semantic segmentation neural network, which returns
  the mask of the obstacle on the frame and its class. The basis for such a network was the SAM universal segmentation
  network, which requires further development to be applied to the semantic segmentation task.
  The peculiarity of this network is its universal applicability, that is, the ability to select any objects in any
  filming situation. At the same time, SAM does not predict the semantics of the object. In this paper, an additional
  module is proposed that makes it possible to implement semantic segmentation by classifying the features
  of the selected objects. The possibility of using such a module to solve the problem of supplementing the
  network output with new information is substantiated. The classification result is then fed into the same filtering
  algorithm as the masks to ensure consistency between the result of the universal network and the complementary
  module. After integrating the module with the model, a new semantic segmentation model was
  obtained, called RTC-SAM in the work. It was used to perform semantic segmentation of a publicly available
  dataset with images of an open area. The 45% result obtained by the IoU metric exceeds the result of existing
  methods by 13%. The images of the results of using the new network shown in the work make it possible to
  verify its performance. It also describes the testing of the developed solution with a study of the performance
  of the developed model on a PC and a mobile computer. The algorithm on the mobile computer shows insufficient
  speed to enter real-time mode – more than 3.5 seconds to process one frame. In this regard, one of
  the directions of further research in the field of improving system performance.

• INTEGRATION OF SEGMENTATION, TRACKING AND CLASSIFICATION MODELS TO SOLVE VIDEO ANALYTICS PROBLEMS

  А.Е. Arkhipov, I.S. Fomin, V.D. Matveev

  2024-04-16

  Abstract ▼

  The integration of several models into one technical vision system will allow solving more
  complex tasks. In particular, for mobile robotics and unmanned aerial vehicles (UAVs), the lack of
  data sets for various conditions is an urgent problem. In the work, the integration of several models
  is proposed as a solution to this problem: segmentation, maintenance and classification. The segmentation
  model allows you to select arbitrary objects from frames, which allows it to be used in nondeterministic
  and dynamic environments. The classification model allows you to determine the objects necessary for navigation or other use, which are then accompanied by a third model. The paper
  describes an algorithm for model aggregation. In addition to models, the key element is the correction
  of model predictions, which allows you to segment and accompany various objects reliably
  enough. The procedure for correcting model predictions solves the following tasks: adding new objects
  to accompany, validating segmented object masks and clarifying the associated masks. The
  versatility of this solution is confirmed by working in difficult conditions, for example, underwater
  photography or images from UAVs. An experimental study of each of the models was carried out in
  an open area and indoors. The data sets used make it possible to assess the applicability of models
  for mobile robotics tasks, that is, to identify possible obstacles in the robot's path, for example, a
  curb, as well as moving objects such as a person or a car. They demonstrated a sufficiently high
  quality of work. For most classes, the indicators exceeded 80% by various metrics. The main errors
  are related to the size of the objects. The conducted experiments clearly demonstrate the versatility of
  this solution without additional training of models. Additionally, a study of performance on a personal
  computer with various input parameters and resolution was conducted. Increasing the number of
  models significantly increases the computational load and does not reach real time. Therefore, one of
  the directions of further research is to increase the speed of the system