Work with Long Contexts in Large Language Models (in Russian)

The research field of large language models is actively evolving. One area of focus is the expansion of the model's context window. Another is improving the model's quality for a specific language. This work combines these two directions and addresses the task of summarization. This task was chosen because it is important for businesses, and models are already capable of generating text summaries with a certain level of quality instead of humans. The goal of this work is to explore the possibility of adapting large language models for summarizing long texts in Russian. To achieve this, the following tasks are undertaken: determining quality metrics for the results, collecting data with long texts, preparing models, and evaluating their quality. As a result of this work, quality metrics for summarization of diplomas and books (analogous to MMLU) were developed, over 10,000 diplomas from SPbU from the past 8 years were collected, and five models were trained using LoRA and LongLoRA methods. An analysis of the results of the trained models was conducted, and recommendations for further work were provided. All artifacts of this work are publicly available on GitHub, Hugging Face, and Yandex Disk.