MULTIMODAL HUMAN ACTION RECOGNITION BASED ON INDEPENDENTLY TRAINED VIDEO AND AUDIO MODELS WITH LATE FUSION OF PREDICTIONS
Abstract
The relevance of this research is driven by the need to improve the accuracy of human action recognition systems in video by integrating audiovisual information, which compensates for the limitations of each individual modality. The aim of this work is to develop and investigate methods for late fusion of predictions from independently trained video and audio models for recognizing six classes of human actions (Archery, BalanceBeam, BasketballDunk, BoxingSpeedBag, FloorGymnastics, SoccerPenalty). The study employs 3D convolutional neural networks with Squeeze-and-Excitation blocks for video processing and a hybrid CNN-GRU architecture for audio analysis, along with seven prediction fusion methods: arithmetic mean, weighted average, majority voting, confidence-based fusion, stacking, product of probabilities, and the Borda count. Experiments demonstrate that the arithmetic mean and product of probabilities methods achieve the highest accuracy of 91.67%, outperforming the individual modalities by 2.33% for color and by 5.6% for grayscale video models. Confidence-based fusion exhibited the best results for grayscale models by automatically adapting the contribution of each modality based on the entropy of their predictions. Special attention is paid to the strict separation of training, validation, and test sets based on filming environments to prevent data leakage and to ensure objective evaluation of model generalization capability. The obtained results confirm the practical value of the proposed approach for video surveillance systems and multimedia platforms, offering an optimal balance between accuracy, computational complexity, and robustness to the absence of one modality. This research contributes to the development of multimodal machine learning methods and provides practical guidelines for selecting a fusion strategy depending on available computational resources and input data types.
References
1. Sun Z., Ke Q., Rahmani H., Bennamoun M., Wang G., Liu J. Human Action Recognition from Various Data Modalities: A Review, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
2. Carreira J., Zisserman A. Quo vadis, action recognition? A new model and the kinetics dataset, Pro-ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299-6308.
3. Bayoudh K., Knani R., Hamdaoui F., Mtibaa A. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets, The Visual Computer, 2022, Vol. 38. No. 8,
pp. 2939-2970.
4. Feichtenhofer C., Pinz A., Zisserman A. Convolutional two-stream network fusion for video action recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1933-1941.
5. Snoek C.G., Worring M., Smeulders A.W. Early versus late fusion in semantic video analysis, Proceed-ings of the 13th Annual ACM International Conference on Multimedia, 2005, pp. 399-402.
6. Pawlowski M., Wroblewska A., Sysko-Romanczuk S. Effective Techniques for Multimodal Data Fusion: A Comparative Analysis, Sensors, 2023, Vol. 23, No. 5, pp. 2381.
7. Caba Heilbron J., Escorcia V., Ghanem B., Niebles J.C. ActivityNet: A large-scale video benchmark for human activity understanding, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 961-970.
8. Ngiam J., Khosla A., Kim M., Nam J., Lee H., Ng A.Y. Multimodal deep learning, Proceedings of the 28th International Conference on Machine Learning (ICML), 2011, Vol. 2, No. 3, pp. 4.
9. Demšar J. Statistical comparisons of classifiers over multiple data sets, The Journal of Machine Learn-ing Research, 2006, Vol. 7, pp. 1-30.
10. He K., Zhang X., Ren S., Sun J. Deep residual learning for image recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778.
11. Hershey S., Chaudhuri S., Ellis D. P., Gemmeke J. F., Jansen A., Moore R. C., Xu R.C. CNN architec-tures for large-scale audio classification, ICASSP 2017 – IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 131-135.
12. Wang L., Xiong Y., Wang Z., Qiao Y., Lin D., Tang X., Van Gool L. Temporal segment networks: To-wards good practices for deep action recognition, Proceedings of the European Conference on Comput-er Vision (ECCV), 2018, pp. 20-36.
13. Soomro K., Zamir A.R., Shah M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. Orlando: CRCV, 2012, 12 p.
14. Bradski G., Kaehler A. Learning OpenCV: Computer Vision with the OpenCV Library. Sebastopol: O’Reilly Media, 2008, 580 p.
15. McFee B., Raffel C., Liang D. et al. librosa: Audio and music signal analysis in python, Proceedings of the 14th Python in Science Conference, 2015, pp. 18-25.
16. Hu J., Shen L., Sun G. Squeeze-and-excitation networks, Proceedings of the IEEE Conference on Com-puter Vision and Pattern Recognition, 2018, pp. 7132-7141.
17. Cho K., Van Merriënboer B., Gulcehre C., Bahdanau D., Bougares F., Schwenk H., Bengio Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. 2014. arXiv:1406.1078.
18. Tompson J., Goroshin R., Jain A., LeCun Y., Bregler C. Efficient object localization using convolutional networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 648-656.
19. Ioffe S., Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covari-ate shift, Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015, pp. 448-456.
20. Szegedy C., Vanhoucke V., Ioffe S., Shlens J., Wojna Z. Rethinking the inception architecture for comput-er vision, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818-2826.
21. Loshchilov I., Hutter F. Decoupled weight decay regularization, International Conference on Learning Representations, 2019.
22. Shorten C., Khoshgoftaar T.M. A survey on image data augmentation for deep learning, Journal of Big Data, 2019, Vol. 6, pp. 1-48.
23. Park D.S., Chan W., Zhang Y., Chiu C.C., Zoph B., Cubuk E.D., Le Q.V. SpecAugment: A simple data augmentation method for automatic speech recognition, arXiv:1904.08779, 2019.
24. Perez L., Wang J. The effectiveness of data augmentation in image classification using deep learning, arXiv:1712.04621, 2017.
25. Smith L.N., Topin N. Super-convergence: Very fast training of neural networks using large learning rates, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, 2019, Vol. 11006, pp. 1100612.
26. Piczak K.J. ESC: Dataset for environmental sound classification, Proceedings of the 23rd ACM Interna-tional Conference on Multimedia, 2015, pp. 1015-1018.
27. Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R. Dropout: a simple way to pre-vent neural networks from overfitting, The Journal of Machine Learning Research, 2014, Vol. 15,
No. 1, pp. 1929–1958.
28. Gemmeke J.F., Ellis D.P., Freedman D., Jansen A., Lawrence W., Moore R.C., Vincent M. Audio set: An ontology and human-labeled dataset for audio events, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 776-780.
29. Hinton G., Vinyals O., Dean J. Distilling the knowledge in a neural network, arXiv:1503.02531, 2015.
30. Wolpert D.H. Stacked generalization, Neural Networks, 1992, Vol. 5, No. 2, pp. 241-259
