INTEGRATION OF SEGMENTATION, TRACKING AND CLASSIFICATION MODELS TO SOLVE VIDEO ANALYTICS PROBLEMS
Abstract
The integration of several models into one technical vision system will allow solving more
complex tasks. In particular, for mobile robotics and unmanned aerial vehicles (UAVs), the lack of
data sets for various conditions is an urgent problem. In the work, the integration of several models
is proposed as a solution to this problem: segmentation, maintenance and classification. The segmentation
model allows you to select arbitrary objects from frames, which allows it to be used in nondeterministic
and dynamic environments. The classification model allows you to determine the objects necessary for navigation or other use, which are then accompanied by a third model. The paper
describes an algorithm for model aggregation. In addition to models, the key element is the correction
of model predictions, which allows you to segment and accompany various objects reliably
enough. The procedure for correcting model predictions solves the following tasks: adding new objects
to accompany, validating segmented object masks and clarifying the associated masks. The
versatility of this solution is confirmed by working in difficult conditions, for example, underwater
photography or images from UAVs. An experimental study of each of the models was carried out in
an open area and indoors. The data sets used make it possible to assess the applicability of models
for mobile robotics tasks, that is, to identify possible obstacles in the robot's path, for example, a
curb, as well as moving objects such as a person or a car. They demonstrated a sufficiently high
quality of work. For most classes, the indicators exceeded 80% by various metrics. The main errors
are related to the size of the objects. The conducted experiments clearly demonstrate the versatility of
this solution without additional training of models. Additionally, a study of performance on a personal
computer with various input parameters and resolution was conducted. Increasing the number of
models significantly increases the computational load and does not reach real time. Therefore, one of
the directions of further research is to increase the speed of the system
References
Available at: http://arxiv.org/abs/2304.11968.
2. Cheng H.K., et al. Tracking anything with decoupled video segmentation, IEEE/CVF International
Conference on Computer Vision, 2023, pp. 1316-1326.
3. Zhu J., et al. Tracking anything in high quality, CoRR, 2023, Vol. abs/2307.13974. Available
at: http://arxiv.org/abs/2307.13974.
4. Liu Y., et al. MobileSAM-Track: Lightweight One-Shot Tracking and Segmentation of Small
Objects on Edge Devices, Remote Sensing, 2023, Vol. 15, No. 24, pp. 5665.
5. Cheng Y., et al. Segment and track anything, CoRR, 2023, Vol. abs/2305.06558. Available at:
http://arxiv.org/abs/2305.06558.
6. Kirillov A., et al. Segment anything, CoRR, 2023, Vol. abs/2304.02643. Available at:
http://arxiv.org/abs/2304.02643
7. Cheng B., Schwing A., Kirillov A. Per-pixel classification is not all you need for semantic segmentation,
Advances in Neural Information Processing Systems, 2021, Vol. 34, pp. 17864-17875.
8. Yang Z., Yang Y. Decoupling features in hierarchical propagation for video object segmentation,
Advances in Neural Information Processing Systems, 2022, Vol. 35, pp. 36324-36336.
9. Yang Z., Wei Y., Yang Y. Associating objects with transformers for video object segmentation,
Advances in Neural Information Processing Systems, 2021, Vol. 34, pp. 2491-2502.
10. Cherti M., et al. Reproducible scaling laws for contrastive language-image learning,
IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2818-2829.
11. Awadalla A., et al. Openflamingo: An open-source framework for training large autoregressive
vision-language models, CoRR, 2023, Vol. abs/2308.01390. Available at: http://arxiv.org/abs/
2308.01390.
12. Li J., Li D., Xiong C., Hoi S. Blip: Bootstrapping language-image pre-training for unified vision-
language understanding and generation, International Conference on Machine Learning,
2022, pp. 12888-12900.
13. Radford A., et al. Learning transferable visual models from natural language supervision, International
conference on machine learning, 2021, pp. 8748-8763.
14. Mueller M., Smith N., Ghanem B. A benchmark and simulator for uav tracking //Computer
Vision–ECCV 2016: 14th European Conference. 2016. – P. 445-461.
15. Github: fbrs_interactive_segmentation. Available at: https://github.com/SamsungLabs/fbrs_
interactive_segmentation.
16. Sofiiuk K., Petrov I., Barinova O., Konushin A. F-BRS: Rethinking Backpropagating Refinement
for Interactive Segmentation, IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2020, pp. 8623-8632.
17. Fomin I., Arhipov A. Selection of Neural Network Algorithms for the Semantic Analysis of
Local Industrial Area, International Russian Automation Conference, 2021, pp. 380-385.
18. Miao J., et al. VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild, IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 2021, pp. 4133-4143.
19. Zhang C., et al. Faster Segment Anything: Towards Lightweight SAM for Mobile Applications,
CoRR, 2023, Vol. abs/2306.14289. Available at: http://arxiv.org/abs/2306.14289.
20. Wang A., et al. RepViT-SAM: Towards Real-Time Segmenting Anything, CoRR, 2023,
Vol. abs/2312.05760 Available at: http://arxiv.org/abs/2312.05760.
