Improvements to the Retrieval Process for Retrieval Augmented Generation Tasks

In recent years, there has been an active development of LLM models such as GPT-4, LLama and Gemini. These models show outstanding abilities in generating useful texts, but they also face the problem of hallucination - generating texts that do not correspond to reality. To solve this problem, a RAG system is applied, which allows the LLM to generate answers based on useful information found by another model, the retrieval model. In this paper, a new method for training retrieval models on a Russian-language corpus of texts was developed and investigated. As a result, a model was created that demonstrates higher quality compared to its OpenAI counterparts and the classical BM25 algorithm. In addition, the conducted studies of the learning process of such a model have provided a deeper understanding of LLM learning mechanisms and contribute to further discoveries in this area.