Evaluation of the Quality of Large Language Models for the Development of a Smart Search System Based on Knowledge in a Question-Answer Format for Company Employees

This work explores methods for evaluating the quality of large language models (LLMs) in a question-answering system (QA-system) built using Retrieval-Augmented Generation (RAG). An extensive dataset comprising question-answer pairs from insurance agents, extracted from various messengers, was collected for this research. Initially, a RAG system was constructed using the llama.index framework, and questions were passed through this system to generate model responses. The primary objective was to evaluate the quality of these responses. Manual annotation was conducted, and popular evaluation metrics such as ROUGE, BLEU, METEOR, and BERTScore were utilized. The analysis revealed that BERTScore demonstrated the highest correlation with human evaluations, leading to the hypothesis that neural network-based metrics, particularly those leveraging LLMs, provide the most human-like assessment of question-answer systems. The study tested various LLMs, including GPT-3.5, GPT-4, and Mistral 7B. The Low-Rank Adaptation (LoRA) method was employed to fine-tune the LLMs on the specific dataset for the question-answering system. The effectiveness of the fine-tuned model was evaluated using the selected metric. Experimental results confirmed that fine-tuning the model using LoRA significantly improves the quality of its responses. Additionally, the use of LLM-based evaluation metrics demonstrated performance on par with human evaluation, suggesting that these metrics can be used to reduce the time required for assessment.