RAG-pipeline Optimization: HSE Data Case

This work explores a method of accessing large language models, which involves retrieval-augmented generation (RAG). The study builds all the stages of the RAG pipeline and optimizes them using data from the Higher School of Economics (HSE). The results of the work will be applied to develop a chatbot assistant for HSE University students. RAG is built on the Russian large language model GigaChat using the model's API through the GigaChain SDK. Various methods of text processing, vector databases, prompting techniques, and large language model generation methods were employed in the study. The final result of the work is an automated hyperparameters tuning pipeline for a RAG built on a given knowledge base. As part of the research, a parser of the knowledge database based on relevant sources about the educational process at the University was also developed, a set of validation metrics for assessing the quality of answers was formed, and the RAG automatic validation framework (RAGAS) was used. The analysis carried out allowed us to draw conclusions about the influence of each stage of the pipeline on the final answers. As a result of the work, the main metrics of response quality were significantly improved (+30 percentage points in response correctness). The research also identified shortcomings of the current approach, namely the low quality of the initial data. The findings will be used for further research and development of the chat assistant within the project "Using Large Language Models to Assist HSE Students." Directions for improvement and growth points of the work were formulated.