METHOD AND ALGORITHM FOR EXTRACTING FEATURES FROM DIGITAL SIGNALS BASED ON NEURAL NETWORKS TRANSFORMER
Cite as: Z.A. Ponimash M.V. Potanin. Method and algorithm for extracting features from digital signals based on neural networks transformer // Izvestiya SFedU. Engineering Sciences – 2024. – N. 6. - P. 52-64. doi: 10.18522/2311-3103-2024-6-52-64
Abstract
Recently, neural network models have become one of the most promising directions in the field of automatic
feature extraction from digital signals. Traditional approaches, such as statistical, time-domain,
frequency-domain, and time-frequency analysis, require significant expert knowledge and often prove insufficiently
effective when dealing with non-stationary and complex signals, such as biomedical signals (ECG,
EEG, EMG) or industrial signals (e.g., currentgrams). These methods have several limitations when it comes
to analyzing multichannel data with varying frequency structures or when signal labeling is too laborintensive
or expensive. Modern neural network architectures, such as transformers, have demonstrated high
efficiency in automatic feature extraction from complex data. Transformers have outperformed traditional
convolutional and recurrent neural networks in many key metrics, particularly in tasks involving time series
forecasting, multimodal data classification, and feature extraction from sequences. Their ability to model
complex temporal dependencies and nonlinear relationships in data makes them ideal for tasks such as noise
filtering and multimodal signal processing. This paper proposes a method for feature extraction from digital
signals based on a modified transformer architecture that incorporates a nonlinear layer after the selfinspection
module. This approach improved the ability of the model to detect complex and nonlinear dependencies
in the data, which is particularly important when dealing with biomedical and signals obtained from
industrial systems. A description of the architecture and the experiments performed are presented, demonstrating
the high performance of the model in solving signal classification, prediction and filtering problems.
It is expected that the model can be applied to a wide range of applications including disease and fault
diagnosis, signal parameter prediction and system modelling.
References
for pattern recognition], 1974, 368 p.
2. Jiang A.Q., Sablayrolles A., Roux A., Mensch A., Savary B., Bamford C., Singh Chaplot D., et al.
Mixtral of Experts. Available at: https://arxiv.org/abs/2401.04088 (accessed 22 October 2024).
3. Bai J., Bai S., Chu Y., Cui Z., Dang K., Deng X., Fan Y., Ge W., et al. Qwen Technical Report. Available
at: https://arxiv.org/abs/2309.16609 (accessed 21 October 2024).
4. Yang A., Yang B., Hui B., Zheng B., Yu B., Zhou C., Li C., Li C., Liu D., et al. Technical Report.
Available at: https://arxiv.org/abs/2407.10671 (accessed 21 October 2024).
5. Mnih V., Badia A. P., Mirza M., Graves A., Lillicrap T. P., Harley T., Silver D., Kavukcuoglu K.
Asynchronous Methods for Deep Reinforcement Learning. Available at: https://arxiv.org/abs/
1602.01783 (accessed 22 October 2024).
6. Jin Y., Li D., A Y., Shi J., Hao P., Sun F., Zhang J., Fang B. RobotGPT: Robot Manipulation Learning
from ChatGPT. Available at: https://arxiv.org/abs/2312.01421 (accessed 19 October 2024).
7. Fan A., Lewis M., Dauphin Y. Hierarchical Neural Story Generation. Available at:
https://arxiv.org/abs/1805.04833 (accessed 20 October 2024).
8. Holtzman A., Buys J., Du L., Forbes M., Choi Y. The Curious Case of Neural Text Degeneration.
Available at: https://arxiv.org/abs/1904.09751 (accessed 20 October 2024).
9. Meister C., Vieira T., Cotterell R. If beam search is the answer, what was the question?. Available at:
https://arxiv.org/abs/2010.02650 (accessed 21 October 2024).
10. Liu Z., Wang Y., Vaidya S., Ruehle F., Halverson J., Soljacic M., Hou T. Y., Tegmark M. KAN: Kolmogorov-
Arnold Networks, arXiv preprint. Available at: https://arxiv.org/abs/2404.19756 (accessed
20 October 2024).
11. Sinii V., Nikulin A., Kurenkov V., Zisman I., Kolesnikov S. In-Context Reinforcement Learning for
Variable Action Spaces, arXiv preprint. Available at: https://arxiv.org/abs/2312.13327 (accessed
22 October 2024).
12. Ponimash Z.A. Raspoznavanie radiotekhnicheskikh signalov s pomoshch'yu neyronnykh setey
[Recognition of radio-technical signals using neural networks]. Available at: https://habr.com/ru/ articles/
318832/ (accessed 20 October 2024).
13. Akhmed A.F., Abdullakh Kh.N., Albaker B.M. Klassifikatsiya signalov EKG na osnove tsifrovoy
obrabotki signalov, tekhnicheskogo vybora funktsiy i radiochastotnogo klassifikatora [ECG Signal
Classification Based on Digital Signal Processing, Feature Selection, and Radio Frequency Classifier],
RENSIT/RENSIT [RENSIT], 2024, Vol. 16, No. 1. DOI: 10.17725/rensit.2024.16.053.
14. Deych A.M. Metody identifikatsii dinamicheskikh ob"ektov [Methods for identifying dynamic objects],
1979.
15. McLean S.M. Fast algorithms for linear prediction and system identification filters with linear phase,
IEEE Transactions on Acoustics, Speech, and Signal Processing, 1982. Dec. DOI:
10.1109/TASSP.1982.1163987.
16. Alekseev D.M., Shumilin A.S., Minyuk A.N., Ponimash Z.A. Sistema avtomaticheskogo poiska
uchastkov epilepticheskoy aktivnosti v sostave oblachnoy platformy khraneniya, sistematizatsii i
obrabotki meditsinskikh dannykh [Automatic search system for epileptic activity areas as part of a
cloud platform for storing, systematizing and processing medical data], Sovremennye naukoemkie
tekhnologii [Modern science-intensive technologies], 2019. Available at: http://www.toptechnologies.
ru/ru/article/view?id=37371 (accessed 18 June 2024).
17. Alekseev D.M., Shumilin A.S., Minyuk A.N., Ponimash Z.A. Ansambl' klassifikatorov: realizatsiya,
otsenka effektivnosti i integratsiya v oblachnuyu platformu khraneniya, sistematizatsii i obrabotki
meditsinskikh dannykh [Ensemble of classifiers: implementation, efficiency evaluation and integration
into a cloud platform for storing, systematizing and processing medical data], Sovremennye
naukoemkie tekhnologii [Modern science-intensive technologies], 2019, No. 9, pp. 20-25.
18. Alekseev D.M., Minyuk A.N., Ponimash Z.A., Shumilin A.S. Razrabotka i opisanie struktury i
funktsionala oblachnoy platformy khraneniya, sistematizatsii i obrabotki meditsinskikh dannykh:
integratsiya sistemy avtomaticheskogo poiska uchastkov epilepticheskoy aktivnosti [Development and
description of the structure and functionality of a cloud platform for storing, systematizing and processing
medical data: integration of a system for automatic search for areas of epileptic activity],
Sistemy upravleniya i informatsionnye tekhnologii [Control Systems and Information Technologies],
2019, No. 3 (77), pp. 52-55.
19. Shilian Zheng, Xiaoyu Zhou, Shichuan Chen, Peihan Qi, and Xiaoniu Yang. DemodNet: Learning Soft
Demodulation from Hard Information Using Convolutional Neural Network, 2020.
20. Zahorian S.A., Zimmer A.M., and Meng F. Vowel Classification for Computer based Visual Feedback
for Speech Training for the Hearing Impaired in ICSLP, 2002.
21. Ponimash Z.A. Razrabotka algoritma klassifikatsii signalov na osnove neyronnykh setey [Development
of a signal classification algorithm based on neural networks], 2018. Available at:
https://hub.lib.sfedu.ru/repository/material/800841304/ (accessed 18 June 2024).
22. Fernandez, Santiago, Graves Alex, Schmidhuber Jürgen. Sequence labelling in structured domains
with hierarchical recurrent neural networks, Proceedings of IJCAI, 2007.
23. Turker I., Serkan K., Levent E., Askar M., Gabbouj M. Real-Time Motor Fault Detection by 1D Convolutional
Neural Networks, IEEE Transactions on Industrial Electronics, 2016.
24. Yihong D., Ying P., Muqiao Y., Songtao L., Qingjiang S. Signal Transformer: Complex-valued Attention
and Meta-Learning for Signal Recognition. https://doi.org/10.48550/arXiv.2106.04392, 2021.
Available at: https://arxiv.org/abs/2106.04392 (accessed 18 June 2024).
25. Jiang Albert Q., Sablayrolles Alexandre, Mensch Arthur, Bamford Chris, Devendra Singh Chaplot,
Diego de las Casas, et al. Mistral 7B. 10.10.23. Available at: https://doi.org/10.48550/
arXiv.2310.06825 (accessed 18 June 2024).
26. Ponimash Z.A., Nosko V.I. Algoritm ranzhirovaniya tekstov na baze svertochnoy neyronnoy seti [Text
ranking algorithm based on a convolutional neural network], Mater. Vserossiyskoy nauchnotekhnicheskoy
konferentsii s mezhdunarodnym uchastiem [Proceedings of the All-Russian scientific
and technical conference with international participation]: in 2 vol. Rostov-on-Don; Taganrog:
Izd-vo YuFU, 2022.
27. Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y.
Hou, Max Tegmark. KAN: Kolmogorov-Arnold Networks. https://doi.org/10.48550/arXiv.2404.19756.
2021. Available at: https://arxiv.org/abs/2404.19756 (accessed 25 October 2024).
