COMPARATIVE ANALYSIS OF METHODS OF VECTORIZATION OF HIGH DIMENSIONAL TEXT DATA

Abstract

The presented publication is devoted to an overview of the problem of presenting textual information
for the subsequent implementation of cluster analysis in the framework of processing
and managing high-dimensional information. Modern requirements for analytical, search and
recommendation information systems demonstrate the weak formation of a holistic solution that
can provide a sufficient level of speed and quality of the results obtained within the framework of
the current information technology market. The search for a solution to the presented problem
entails the need to conduct an objective analysis of existing solutions for representing textual information
in vector space, in order to form a holistic view of the advantages and disadvantages of
the analyzed approaches, as well as the formation of criteria that allow one to implement their
own approach, devoid of identified weaknesses. The presented work is analytical, and allows you
to get an idea of the current state and elaboration of the identified problem within a limited subject
area. Clustering of text data is the automatic formation of subsets, the elements of which are instances
of documents of some researched, unstructured sample of a fixed dimension. This process
can be classified as unsupervised learning, which implies the absence of an expert who personally
assigns class indices to the original sample of documents. However, the implementation of cluster
analysis of text data without any pre-processing is impossible. To do this, it is necessary to ensure
standardization and reduction of input data to a single format and form. Within the framework of
this stage of the implementation of cluster analysis, the presented publication discusses methods
for preprocessing text data. The novelty of the presented publication lies in the formation of the
theoretical basis of the main methods of text data vectorization, by systematizing and objectifying
the proposed assumptions, by conducting a series of experimental studies. The main difference of
this work from the already published scientific works is the systematization and analysis of modern
solutions, as well as the hypotheses about the relevance and effectiveness of our own hybridized
approach designed for text data vectorization.