An automatic tool for the quality benchmark of large language models on Russian-language queries

Large language models (LLMs) are an area of growing research interest due to their extensive practical application. Design of a set of tasks for testing LLMs is at the heart of understanding their capabilities. Salient characteristics of such datasets include compactness, representativeness, and interpretability. Numerous studies have attempted to determine the standards for measuring the performance of large language models primarily focusing on complex evaluation metrics. However, there is a lack of standardised evaluation suites that provide quick and interpretable results and are devised specifically for the Russian language. The purpose of the present study is to develop an automatic tool for the accuracy benchmark of LLMs on Russian language queries. The current study combines NLP-methods and linguistic analysis on phonological, lexical and syntactic levels. The research project might complement contemporary evaluation approaches of large language models, providing a relatively compact and representative automatic test for the Russian language, without resorting to large sets of complex tasks.