A TRANSFORMER-BASED ALGORITHM FOR CLASSIFYING LONG TEXTS

Abstract

The article is devoted to the urgent problem of representing and classifying long text documents using
transformers. Transformers-based text representation methods cannot effectively process long sequences
due to their self-attention process, which scales quadratically with the sequence length. This limitation
leads to high computational complexity and the inability to apply such models for processing long
documents. To eliminate this drawback, the article developed an algorithm based on the SBERT transformer,
which allows building a vector representation of long text documents. The key idea of the algorithm
is the application of two different procedures for creating a vector representation: the first is based
on text segmentation and averaging of segment vectors, and the second is based on concatenation of segment
vectors. This combination of procedures allows preserving important information from long documents.
To verify the effectiveness of the algorithm, a computational experiment was conducted on a group
of classifiers built on the basis of the proposed algorithm and a group of well-known text vectorization
methods, such as TF-IDF, LSA, and BoWC. The results of the computational experiment showed that
transformer-based classifiers generally achieve better classification accuracy results compared to classical
methods. However, this advantage is achieved at the cost of higher computational complexity and,
accordingly, longer training and application times for such models. On the other hand, classical text
vectorization methods, such as TF-IDF, LSA, and BoWC, demonstrated higher speed, making them more
preferable in cases where pre-encoding is not allowed and real-time operation is required. The proposed
algorithm has proven its high efficiency and led and led to an increase in the classification accuracy of the
BBC dataset by 0.5% according to the F1 criterion.