ON THE SIMILARITY FUNCTION OF GRAPHIC REPRESENTATIONS OF EXECUTIVE FILES IN THE OBFUSCING TRANSFORMATION EVALUATION MODEL
Abstract
Obfuscation of program code is used to complicate its analysis in a model when the analyst has full access to the program. Obfuscation is usually divided into cryptographically secure and heuristically resistant. In the first case, the complexity of the analysis is comparable to the difficulty of solving some known mathematical problem. In the second case, the resistance is usually justified by the lack of effective techniques for analyzing the obfuscation method known at the time of its creation. Cryptographically secure obfuscation has not yet found practical application, while heuristically resistant is widely used. Previously, the authors proposed a model for assessing the efficiency and resistance of heuristic obfuscating transformations based on the use of a similarity function. In this paper, such a similarity function is constructed using machine learning methods based on a comparison of the graphical representation of program executable files. In particular, the comparison is performed using a convolutional network with four convolutional layers, an RMSprop optimizer, an NLLLoss loss function, and two outputs of a fully connected layer. The proposed function is used in the implementation of a model for evaluating the efficiency and resistance of obfuscating transformations. In addition to the similarity function, the implementation of the model also includes: a basic set of obfuscating transformations provided by the Hikari obfuscator; a set of obfuscating transformation sequences based on the basic set; a test set of programs for training models based on the CoreUtils, PolyBench and HashCat program sets; approximation of the most "understandable" version of the program using the smallest version of the program (searched among the versions obtained using various optimization options of the GCC, Clang and AOCC compilers); a program deobfuscation scheme based on the optimizing compiler from LLVM. The results of an experimental study with the implemented model showed that it is impractical to use the constructed similarity function in the framework of the evaluation model due to its low accuracy, but it is possible to use it when constructing more complex functions.
References
1. Varnovsky N. et al. The current state of art in program obfuscations: definitions of obfuscation security, Proceedings of the Institute for system programming of the RAS, 2014, Vol. 26, No. 3.
2. Garg S. et al. Candidate Indistinguishability Obfuscation and Functional Encryption for all Circuits, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, 2013.
3. Xu H. et al. Layered obfuscation: a taxonomy of software obfuscation techniques for layered security, Vol. 3, 2020.
4. BinShamlan M. H. B.M.A..Z.A.A. The impact of control flow obfuscation technique on software protection against human attacks, First International Conference of Intelligent Computing and Engineering (ICOICE), 2019.
5. Halevi S. et al. Implementing BP-obfuscation using graph-induced encoding, Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017.
6. Collberg C. T.C..L.D. A taxonomy of obfuscating transformations, 1997.
7. Nagra J. C.C. Surreptitious software: obfuscation, watermarking, and tamperproofing for software protection. Pearson Education, 2009.
8. Banescu S. P.A. A tutorial on software obfuscation, 2018, Vol. 108.
9. De Sutter B. et al. Evaluation methodologies in software protection research, 2024, Vol. 57, No. 4.
10. .Madou M. et al. On the effectiveness of source code transformations for binary obfuscation, Proceedings of the International Conference on Software Engineering Research and Practice (SERP06), 2006.
11. Manikyam R. et al. Comparing the effectiveness of commercial obfuscators against MATE attacks, Proceedings of the 6th Workshop on Software Security, Protection, and Reverse Engineering, 2016.
12. Schrittwieser S., Katzenbeisser S., Kinder J., Merzdovnik G., Weippl E. Protecting software through obfuscation: Can it keep pace with progress in code analysis?, ACM Computing Surveys (CSUR), 2016, Vol. 49, pp. 1-37.
13. Borisov P.D., Kosolapov Yu.V. Sposob kolichestvennogo sravneniya obfustsiruyushchikh preobrazovaniy [Method for quantitative comparison of obfuscating transformations], Informatika i avtomatizatsiya [Computer Science and Automation], 2024, Vol. 23, pp. 684-726.
14. Gulwani S., Polozov O., Singh R., others. Program synthesis, Foundations and Trends® in Programming Languages, 2017, Vol. 4, pp. 1-119.
15. Holder W., McDonald J.T., Andel T.R. Evaluating optimal phase ordering in obfuscation executives, Proceedings of the 7th Software Security, Protection, and Reverse Engineering/Software Security and Protection Workshop, 2017, pp. 1-12.
16. Borisov P.D., Kosolapov Y.V. On the Automatic Analysis of the Practical Resistance of Obfuscating Transformations, Automatic Control and Computer Sciences, 2020, Vol. 54, pp. 619-629.
17. Borisov P.D., Kosolapov Y.V. On the Characteristics of Symbolic Execution in the Problem of Assessing the Quality of Obfuscating Transformations, Automatic Control and Computer Sciences, 2022, Vol. 56, pp. 595-605.
18. Mohsen R., Pinto A. Evaluating Obfuscation Security: A Quantitative Approach October 2015.
19. Lekssays A., Falah B., Abufardeh S. A Novel Approach for Android Malware Detection and Classification using Convolutional Neural Networks, ICSOFT, 2020, pp. 606-614.
20. Kiger J., Ho S.S., Heydari V. Malware binary image classification using convolutional neural networks, International Conference on Cyber Warfare and Security, 2022, Vol. 17, pp. 469-478.
21. Jiang H., Polsani H., Liu Y. DeepGray: Malware Classification Using Grayscale Images with Deep Learning, The International FLAIRS Conference Proceedings, 2024, Vol. 37.
22. Ben Abdel Ouahab I., Bouhorma M., Boudhir A.A., El Aachak L. Classification of grayscale malware images using the K-nearest neighbor algorithm, Innovations in Smart Cities Applications Edition 3: The Proceedings of the 4th International Conference on Smart City Applications 4, 2020, pp. 1038-1050.
