Search

Search Results

• MULTIMODAL HUMAN ACTION RECOGNITION BASED ON INDEPENDENTLY TRAINED VIDEO AND AUDIO MODELS WITH LATE FUSION OF PREDICTIONS

  V.Z. Gergov

  51-65

  2026-09-10

  Abstract ▼

  The relevance of this research is driven by the need to improve the accuracy of human action recognition systems in video by integrating audiovisual information, which compensates for the limitations of each individual modality. The aim of this work is to develop and investigate methods for late fusion of predictions from independently trained video and audio models for recognizing six classes of human actions (Archery, BalanceBeam, BasketballDunk, BoxingSpeedBag, FloorGymnastics, SoccerPenalty).
  The study employs 3D convolutional neural networks with Squeeze-and-Excitation blocks for video processing and a hybrid CNN-GRU architecture for audio analysis, along with seven prediction fusion methods: arithmetic mean, weighted average, majority voting, confidence-based fusion, stacking, product of probabilities, and the Borda count. Experiments demonstrate that the arithmetic mean and product of probabilities methods achieve the highest accuracy of 91.67%, outperforming the individual modalities by 2.33% for color and by 5.6% for grayscale video models. Confidence-based fusion exhibited the best results for grayscale models by automatically adapting the contribution of each modality based on the entropy of their predictions. Special attention is paid to the strict separation of training, validation, and test sets based on filming environments to prevent data leakage and to ensure objective evaluation of model generalization capability. The obtained results confirm the practical value of the proposed approach for video surveillance systems and multimedia platforms, offering an optimal balance between accuracy, computational complexity, and robustness to the absence of one modality. This research contributes to the development of multimodal machine learning methods and provides practical guidelines for selecting a fusion strategy depending on available computational resources and input data types.

• MODERN METHODS OF HYPERSPECTRAL IMAGE PROCESSING: SYSTEM ANALYSIS, ALGORITHMS AND PROSPECTS FOR APPLICATION IN CONSTRUCTION DIAGNOSTICS

  М. А. Filonova , S. N. Shirobokova

  188-208

  2026-07-07

  Abstract ▼

  The relevance of this study is determined by the growing interest in hyperspectral imaging as a tool for non-destructive testing of building materials and structures, as well as by the insufficient systematization of modern methods for processing such data. The aim of the work is to provide a systematic analysis of HSI processing algorithms, identify their advantages and limitations, and determine the prospects for their application in construction diagnostics. The study examines the specific features of hyperspectral data, including high dimensionality, noise, calibration errors, atmospheric distortions, and the shortage of labeled datasets. The evolution of approaches is shown: from classical machine learning methods and manual feature engineering to deep neural networks. Dimensionality reduction methods, kNN classifiers, Bayesian models, logistic regression, Random Forest, SVM, and MLP are analyzed, along with methods for incorporating spectral-spatial context. Special attention is paid to modern deep learning architectures: 1D, 2D, and 3D CNNs, RNNs, LSTM/GRU models, hybrid CNN–RNN models, transformers, and CNN–Transformer schemes. Transfer learning, semi-supervised learning, self-supervised learning, few-shot learning, meta-learning, and domain adaptation are considered separately as ways to overcome the limited availability of labeled data. The approaches are compared in terms of data requirements, computational complexity, robustness to noise, and their ability to account for spectral and spatial dependencies. It is shown that the most promising models for construction diagnostics are hybrid models that combine local convolutional features, the global context of attention mechanisms, and the possibility of fine-tuning on small specialized datasets. The paper summarizes the current state of the field and forms a basis for selecting methods for defect detection, moisture assessment, corrosion analysis, and evaluation of degradation in building structures. The conclusions formulated in the study can be used when designing experimental protocols and selecting architectures for further applied research in the field of building monitoring

• ACCELERATION OF THE DIRECT PASSAGE IN THE IMPLEMENTATION OF CNN ON A LIMITED COMPUTING RESOURCE

  А.Е. Shchelkunov, V.V. Kovalev, I. V. Sidko, N. Е. Sergeev

  2022-04-21

  Abstract ▼

  The work is devoted to the optimization of the neural network architecture for its launch on
  a limited computing resource. Several optimization approaches are considered, estimates of the
  complexity and execution time of the forward pass of the neural network are given. Comparative
  estimates of the complexity of the network using different optimization approaches are given.
  The paper presents an analysis of the selected network architecture, and estimates of the computational
  complexity of individual components (modules) of the architecture are obtained. An analysis
  of possible optimization methods for each module was made. The parameters of the considered
  modules, the sizes of the input and output tensors are described. Several architectures were tested
  to optimize the feature extraction module, ResNet 50, ResNet 18, MobileNet v3 small, MobileNet
  v3 large. A comparative analysis of the computational complexity and execution time of the forward
  pass for each architecture is presented. Forward pass times were measured on Nvidia's
  Jetson AGX Xaver embedded computing device. Estimates of the execution time of the direct pass
  for each module of the considered neural networks are presented. The paper presents the results of
  comparing neural network accuracy estimates before and after architecture optimization. The test
  data set consists of 100 video recordings. 5 different typical objects are involved in test videos,
  10 different scenarios are recorded for each object class. For each of the developed architectures,
  accuracy estimates were obtained, and a comparative analysis was made.