THE TECHNIQUE OF AUTOMATED IMAGE RESTORATION USING CONVOLUTIONAL NEURAL NETWORKS
Cite as: G.A. Khrishkevich, D.A. Andreev, L.V. Motaylenko, I.V. Bruttan, O.N. Timofeeva. The technique of automated image restoration using convolutional neural networks // Izvestiya SFedU. Engineering Sciences – 2024. – N. 6. - P. 65-76. doi: 10.18522/2311-3103-2024-6-65-76
Keywords:
Image, convolutional neural network, restoration, fresco
Abstract
The task of restoring lost fragments of monumental painting is relevant in the context of preserving
cultural heritage sites. Modern artificial intelligence technologies, including convolutional neural networks
(CNN), significantly expand the possibilities of restoration, allowing for the automation of complex
image restoration processes. In particular, the restoration of lost elements of frescoes requires precise
analysis tools that can predict missing fragments with minimal errors, while preserving the artistic style of
the original. The purpose of this study is to develop a technique of automated restoration of lost fragments
of monumental painting images using CNN (using frescoes as an example). This goal was achieved by
solving the following problems: obtaining fresco images using appropriate methodological and technical
tools, applying the U-Net architecture for image segmentation and reconstruction, predicting lost areas
based on color characteristic analysis. The photogrammetry method and the designed device, which were
used to perform multi-angle shooting, provided high-quality source data for subsequent processing. Adaptation
of the U-Net architecture to the image segmentation task has proven its effectiveness in identifying
key structural elements of frescoes, which contributed to the accurate reconstruction of lost areas.
To predict the lost areas, color characteristics were analyzed in the HSL system, which allowed the CNN
to predict the missing colors with a high degree of accuracy. Brief conclusions of the study show that the
proposed technique allows restoring both the shape and color of lost fragments of frescoes. The proposed
technique is planned to be used for the restoration of other types of art works, which makes it promising
for further research
References
1. Sikorskiy O.S. Obzor svertochnykh neyronnykh setey dlya zadachi klassifikatsii izobrazheniy [Review
of convolutional neural networks for image classification], Novye informatsionnye tekhnologii v
avtomatizirovannykh sistemakh [New information technologies in automated systems], 2017, No. 20,
pp. 37-42.
2. Sulavko A.E., Lozhnikov P.S.., Choban A.G., Stadnikov D.G., Nigrey A.A., Inivatov D.P. Evaluation of
EEG identification potential using statistical approach and convolutional neural networks, Information
and Control Systems, 2020, No. 6 (109), pp. 37-49.
3. Markin E.I., Zuparova V.V., Martyshkin A.I. Issledovanie vozmozhnosti primeneniya neyronnykh
setey dlya vosstanovleniya izobrazheniya litsa v sistemakh raspoznavaniya [Study of the possibility of
using neural networks for facial image restoration in recognition systems], Tr. ISP RAN [Proceedings
of ISP RAS], 2022, Vol. 34, Issue 6, pp. 117-126.
4. Vishniakou U.A., Shaya B.H. Voice Detection Using Convolutional Neural Network, Doklady BGUIR
[Reports of BSUIR], 2023, Vol. 21, No. 2, pp. 114-120.
5. Guseynov S.B., Mengel' V.V., Orozkozhoev D.S. Ispol'zovanie intellektual'nykh sistem pri
raspoznavanii teksta na izobrazhenii [Using intelligent systems in recognizing text in an image],
Izvestiya Kyrgyzskogo gosudarstvennogo tekhnicheskogo universiteta im. I. Razzakova [Bulletin of the
Kyrgyz State Technical University named after I. Razzakov], 2022, No. 1 (61), pp. 46-52.
6. Borisov A.A., Vasil'ev Yu.A., Vladzimirskiy A.V., Omelyanskaya O.V., Semenov S.S., Arzamasov K.M.
Ispol'zovanie neyronnykh setey dlya poiska narusheniy ukladki patsienta na rentgenogrammakh
organov grudnoy kletki [Using neural networks to search for patient positioning disorders on chest Xrays],
Programmnye sistemy: teoriya i prilozheniya [Software systems: theory and applications], 2023,
Vol. 14, No. 3 (58), pp. 95-113.
7. Akutin A.S., Goryakin M.V., Zubavlenko R.A., Pechenkin V.V., Solopekin D.A. Ispol'zovanie
iskusstvennykh neyronnykh setey dlya vypolneniya segmentatsii rentgenogramm tazobedrennogo
sustava pri lechenii osteoartrita [Using artificial neural networks to perform segmentation of hip joint
radiographs in the treatment of osteoarthritis], Modelirovanie, optimizatsiya i informatsionnye
tekhnologii [Modeling, optimization and information technology], 2024, Vol. 12, No. 1 (44). Available
at: https://moitvivt.ru/ru/journal/pdf?id=1486 (accessed 12 September 2024).
8. Doroshkin N.V., Obraztsov V.A. Raspoznavanie obrazov na meditsinskikh izobrazheniyakh s pomoshch'yu
neyronnoy seti [Pattern recognition in medical images using a neural network], Voprosy ustoychivogo
razvitiya obshchestva [Issues of sustainable development of society], 2022, No. 4, pp. 941-948.
9. Kakumani A.K., Sree L.P. BRB U-Net: Bottleneck Residual Blocks in U-Net for Light-Weight Semantic
Segmentation, International Journal of Computer Applications, 2022, Vol. 184, No. 33, pp. 63-67.
10. Kovalev V.Yu., Shishkin A.G. Raspoznavanie stilya proizvedeniy zhivopisi po ikh izobrazheniyam s
pomoshch'yu glubokikh neyronnykh setey [Recognition of the style of works of art from their images
using deep neural networks], Informatika i sistemy upravleniya [Computer Science and Control Systems],
2020, No. 3 (65), pp. 74-86.
11. Filina O.S. Vozmozhnosti primeneniya komp'yuternykh graficheskikh prilozheniy v restavratsii i
rekonstruktsii russkoy ikony [Possibilities of using computer graphics applications in the restoration
and reconstruction of Russian icons], Innovatsionnyy diskurs razvitiya sovremennoy nauki: Sb. statey
V Mezhdunarodnoy nauchno-prakticheskoy konferentsii [Innovative discourse on the development of
modern science: Collection of articles from the V International Scientific and Practical Conference].
Petrozavodsk, 2021, pp. 166-175.
12. Rassolova V.M., Pozdnyakova E.D., Lotoreva V.V. Issledovanie effektivnosti arkhitektur svertochnykh
neyronnykh setey v reshenii zadachi klassifikatsii pri maloy obuchayushchey vyborke [Study of the efficiency
of convolutional neural network architectures in solving the classification problem with a
small training sample], Auditorium, 2023, No. 2 (38), pp. 73-81.
13. Buryy V.P., Borisova N.L. Raboty po restavratsii fragmentov drevnerusskikh stenopisey v MGKhPA
im. S.G. Stroganova. Dekorativnoe iskusstvo i predmetno-prostranstvennaya sreda [Works on the restoration
of fragments of ancient Russian murals at the Stroganov Moscow State Academy of Art and
Industry. Decorative art and subject-spatial environment], Vestnik RGKhPU im. S.G. Stroganova [Bulletin
of the Stroganov Russian State University of Art and Industry], 2019, No. 3-1, pp. 21-36.
14. Ivanov P.V., Tarasov A.V. Fotogrammetriya – novaya zhizn' staroy tekhnologii (printsipy i prilozhenie
dlya poiskov MPI) [Photogrammetry - a new life for an old technology (principles and application for
mineral exploration)], Nauchno-metodicheskie osnovy prognoza, poiskov i otsenki mestorozhdeniy
blagorodnykh, tsvetnykh metallov i almazov: Sbornik tezisov dokladov VII nauchno-prakticheskoy
konferentsii [Scientific and methodological foundations for forecasting, prospecting and evaluating
deposits of precious, non-ferrous metals and diamonds: Collection of abstracts of reports of the
VII scientific and practical conference]. Moscow, 2017, pp. 118.
15. Khrishkevich G.A. Informatsionnye protsessy vosstanovleniya izobrazheniy monumental'noy zhivopisi
[Information processes for restoring images of monumental painting], Molodezh'-nauke: Mater.
molodezhnykh nauchno-prakticheskikh konferentsiy Pskovskogo gosudarstvennogo universiteta po
itogam nauchno-issledovatel'skoy raboty v 2021/2022 uchebnom godu [Youth for Science: Proceedings
of youth scientific and practical conferences of Pskov State University based on the results of research
work in the 2021/2022 academic year]. Pskov, 2022, Vol. I, pp. 134-137.
16. Hao X., Yin L., Li X. et al. A Multi-Objective Semantic Segmentation Algorithm Based on Improved
U-Net Networks, Remote Sensing, 2023, 15 (7), 1838.
17. Li M., Liu X., Zhang A. et al. Testing of the stochastic parallel gradient descent algorithm to the alignment
of a two-mirror telescope, Journal of Optical Technology, 2020, 87 (5), pp. 276-283
18. Mallaev A.R., Juraev F.D., Ochilov M.A. Improvement of control models of closed systems using
neural networks, Innovatsion texnologiyalar,2023, No. 3, pp. 17-26.
19. Proskuryakov N.E., Yakovlev B.S., Arkhangel'skaya N.N. Opredelenie parametrov, vliyayushchikh na
vremya rascheta srednego tsveta izobrazheniya [Determination of parameters affecting the time of calculating
the average color of an image], Problemy mashinovedeniya: Materialy III Mezhdunarodnoy
nauchno-tekhnicheskoy konferentsii [Problems of Mechanical Engineering: Proceedings of the III International
Scientific and Technical Conference]. Omsk, 2019, Part II, pp. 342-350.
20. Zotkina A.A., Kholkina V.M. Raspoznavanie izobrazheniy s pomoshch'yu svertochnykh neyronnykh
setey [Image recognition using convolutional neural networks], Sovremennye informatsionnye
tekhnologii [Modern information technologies], 2023, No. 38 (38), pp. 60-63.
of convolutional neural networks for image classification], Novye informatsionnye tekhnologii v
avtomatizirovannykh sistemakh [New information technologies in automated systems], 2017, No. 20,
pp. 37-42.
2. Sulavko A.E., Lozhnikov P.S.., Choban A.G., Stadnikov D.G., Nigrey A.A., Inivatov D.P. Evaluation of
EEG identification potential using statistical approach and convolutional neural networks, Information
and Control Systems, 2020, No. 6 (109), pp. 37-49.
3. Markin E.I., Zuparova V.V., Martyshkin A.I. Issledovanie vozmozhnosti primeneniya neyronnykh
setey dlya vosstanovleniya izobrazheniya litsa v sistemakh raspoznavaniya [Study of the possibility of
using neural networks for facial image restoration in recognition systems], Tr. ISP RAN [Proceedings
of ISP RAS], 2022, Vol. 34, Issue 6, pp. 117-126.
4. Vishniakou U.A., Shaya B.H. Voice Detection Using Convolutional Neural Network, Doklady BGUIR
[Reports of BSUIR], 2023, Vol. 21, No. 2, pp. 114-120.
5. Guseynov S.B., Mengel' V.V., Orozkozhoev D.S. Ispol'zovanie intellektual'nykh sistem pri
raspoznavanii teksta na izobrazhenii [Using intelligent systems in recognizing text in an image],
Izvestiya Kyrgyzskogo gosudarstvennogo tekhnicheskogo universiteta im. I. Razzakova [Bulletin of the
Kyrgyz State Technical University named after I. Razzakov], 2022, No. 1 (61), pp. 46-52.
6. Borisov A.A., Vasil'ev Yu.A., Vladzimirskiy A.V., Omelyanskaya O.V., Semenov S.S., Arzamasov K.M.
Ispol'zovanie neyronnykh setey dlya poiska narusheniy ukladki patsienta na rentgenogrammakh
organov grudnoy kletki [Using neural networks to search for patient positioning disorders on chest Xrays],
Programmnye sistemy: teoriya i prilozheniya [Software systems: theory and applications], 2023,
Vol. 14, No. 3 (58), pp. 95-113.
7. Akutin A.S., Goryakin M.V., Zubavlenko R.A., Pechenkin V.V., Solopekin D.A. Ispol'zovanie
iskusstvennykh neyronnykh setey dlya vypolneniya segmentatsii rentgenogramm tazobedrennogo
sustava pri lechenii osteoartrita [Using artificial neural networks to perform segmentation of hip joint
radiographs in the treatment of osteoarthritis], Modelirovanie, optimizatsiya i informatsionnye
tekhnologii [Modeling, optimization and information technology], 2024, Vol. 12, No. 1 (44). Available
at: https://moitvivt.ru/ru/journal/pdf?id=1486 (accessed 12 September 2024).
8. Doroshkin N.V., Obraztsov V.A. Raspoznavanie obrazov na meditsinskikh izobrazheniyakh s pomoshch'yu
neyronnoy seti [Pattern recognition in medical images using a neural network], Voprosy ustoychivogo
razvitiya obshchestva [Issues of sustainable development of society], 2022, No. 4, pp. 941-948.
9. Kakumani A.K., Sree L.P. BRB U-Net: Bottleneck Residual Blocks in U-Net for Light-Weight Semantic
Segmentation, International Journal of Computer Applications, 2022, Vol. 184, No. 33, pp. 63-67.
10. Kovalev V.Yu., Shishkin A.G. Raspoznavanie stilya proizvedeniy zhivopisi po ikh izobrazheniyam s
pomoshch'yu glubokikh neyronnykh setey [Recognition of the style of works of art from their images
using deep neural networks], Informatika i sistemy upravleniya [Computer Science and Control Systems],
2020, No. 3 (65), pp. 74-86.
11. Filina O.S. Vozmozhnosti primeneniya komp'yuternykh graficheskikh prilozheniy v restavratsii i
rekonstruktsii russkoy ikony [Possibilities of using computer graphics applications in the restoration
and reconstruction of Russian icons], Innovatsionnyy diskurs razvitiya sovremennoy nauki: Sb. statey
V Mezhdunarodnoy nauchno-prakticheskoy konferentsii [Innovative discourse on the development of
modern science: Collection of articles from the V International Scientific and Practical Conference].
Petrozavodsk, 2021, pp. 166-175.
12. Rassolova V.M., Pozdnyakova E.D., Lotoreva V.V. Issledovanie effektivnosti arkhitektur svertochnykh
neyronnykh setey v reshenii zadachi klassifikatsii pri maloy obuchayushchey vyborke [Study of the efficiency
of convolutional neural network architectures in solving the classification problem with a
small training sample], Auditorium, 2023, No. 2 (38), pp. 73-81.
13. Buryy V.P., Borisova N.L. Raboty po restavratsii fragmentov drevnerusskikh stenopisey v MGKhPA
im. S.G. Stroganova. Dekorativnoe iskusstvo i predmetno-prostranstvennaya sreda [Works on the restoration
of fragments of ancient Russian murals at the Stroganov Moscow State Academy of Art and
Industry. Decorative art and subject-spatial environment], Vestnik RGKhPU im. S.G. Stroganova [Bulletin
of the Stroganov Russian State University of Art and Industry], 2019, No. 3-1, pp. 21-36.
14. Ivanov P.V., Tarasov A.V. Fotogrammetriya – novaya zhizn' staroy tekhnologii (printsipy i prilozhenie
dlya poiskov MPI) [Photogrammetry - a new life for an old technology (principles and application for
mineral exploration)], Nauchno-metodicheskie osnovy prognoza, poiskov i otsenki mestorozhdeniy
blagorodnykh, tsvetnykh metallov i almazov: Sbornik tezisov dokladov VII nauchno-prakticheskoy
konferentsii [Scientific and methodological foundations for forecasting, prospecting and evaluating
deposits of precious, non-ferrous metals and diamonds: Collection of abstracts of reports of the
VII scientific and practical conference]. Moscow, 2017, pp. 118.
15. Khrishkevich G.A. Informatsionnye protsessy vosstanovleniya izobrazheniy monumental'noy zhivopisi
[Information processes for restoring images of monumental painting], Molodezh'-nauke: Mater.
molodezhnykh nauchno-prakticheskikh konferentsiy Pskovskogo gosudarstvennogo universiteta po
itogam nauchno-issledovatel'skoy raboty v 2021/2022 uchebnom godu [Youth for Science: Proceedings
of youth scientific and practical conferences of Pskov State University based on the results of research
work in the 2021/2022 academic year]. Pskov, 2022, Vol. I, pp. 134-137.
16. Hao X., Yin L., Li X. et al. A Multi-Objective Semantic Segmentation Algorithm Based on Improved
U-Net Networks, Remote Sensing, 2023, 15 (7), 1838.
17. Li M., Liu X., Zhang A. et al. Testing of the stochastic parallel gradient descent algorithm to the alignment
of a two-mirror telescope, Journal of Optical Technology, 2020, 87 (5), pp. 276-283
18. Mallaev A.R., Juraev F.D., Ochilov M.A. Improvement of control models of closed systems using
neural networks, Innovatsion texnologiyalar,2023, No. 3, pp. 17-26.
19. Proskuryakov N.E., Yakovlev B.S., Arkhangel'skaya N.N. Opredelenie parametrov, vliyayushchikh na
vremya rascheta srednego tsveta izobrazheniya [Determination of parameters affecting the time of calculating
the average color of an image], Problemy mashinovedeniya: Materialy III Mezhdunarodnoy
nauchno-tekhnicheskoy konferentsii [Problems of Mechanical Engineering: Proceedings of the III International
Scientific and Technical Conference]. Omsk, 2019, Part II, pp. 342-350.
20. Zotkina A.A., Kholkina V.M. Raspoznavanie izobrazheniy s pomoshch'yu svertochnykh neyronnykh
setey [Image recognition using convolutional neural networks], Sovremennye informatsionnye
tekhnologii [Modern information technologies], 2023, No. 38 (38), pp. 60-63.
Published
2025-01-19
Issue
Section
SECTION I. INFORMATION PROCESSING ALGORITHMS
