ON THE ACCURACY AND COMPLEXITY OF THE MULTI-STAGE METHOD FOR CORRECTING DISTORTED TEXTS DEPENDING ON THE DEGREE OF DISTORTION

Abstract

One of the main factors that significantly complicate the understanding, translation and
analysis of texts obtained by automatic recognition of speech or images of texts is the presence of
distortions in the form of erroneous symbols, words and phrases. Until recently, there were no
effective software tools for correcting texts with significant distortions, although this task is relevant
both for Russian and other common languages in the context of the active use of recognition
systems in advanced augmented reality systems. The authors proposed a new multi-stage method
for correcting distorted texts, which significantly increases the accuracy of the correction (in
terms of the number of correctly corrected words in the text) and is based on the sequential detection
of errors and their correction. In this paper, we evaluate the accuracy and computational
complexity of the proposed method for correcting distorted texts at various levels of distortion, and
determine its place among other modern approaches to correction. The most typical errors of
recognition systems are: – replacing a word with a similar sound or graphic spelling; – replacing
several words with one; – replacing one word with several; – omission of words; – insertion or
deletion of short words (including prepositions and conjunctions). As a result of recognition, a
distorted text is obtained, which consists mainly of dictionary words, even in places of distortion.
With a large number of distortions, the texts become almost unreadable. Due to the fact that it is
problematic to select texts with a wide range of distortion levels in the required amount based on
the results of real machine recognition of speech and images of texts, software modeling of distortions
was used. A text distortion technique has been proposed and implemented that simulates the
results of recognition systems in a wide range of distortions; distorted texts have been prepared in
the required amount. Within the framework of the proposed multi-stage correction method, nondictionary
word forms and words are considered distorted if the probability of their occurrence in
the text in accordance with the chosen language model is less than a given threshold. For such
distorted words, a list of possible variants of words is built, which includes only those word forms
from the dictionary that are at a certain Levenshtein distance from the word under study. The corrected
text from the tables of word variants is obtained by searching for the most probable chain
of word forms. The correction method consists of several stages, at each stage only those fragments
of the text that remain distorted after the previous stage are corrected. According to the
results of the experiments on the correction of distorted texts, it was concluded that the proposed
correction method showed good results with an average value of F-measure >50 % in the distortion
range from 0 to 75 %. Linguistic experts confirmed the fruitfulness of the proposed approach
to correction and its preference over other modern approaches, fixing that with a level of distortion
of up to 50 % of words, the corrected text is read with much less effort than a distorted one,
and with a level of distortion of up to 70% of words, the corrected text also allows you to highlight
useful information about the content.