Methods for Evaluating the Conversational Quality of a Virtual Assistant

One of the main functions of a virtual assistant is engaging in conversation with the user on arbitrary topics. For this purpose, the Alice virtual assistant uses dialogue models — large language models fine-tuned on the task of generating the next response in a conversation. Being able to assess the quality of these models is critically important in their development. The most accurate method of evaluating a dialogue model is collecting human judgements of its responses. However, collecting such evaluations is costly in both time and resources. Therefore, fully automated approximate evaluation methods are needed as well. We have developed three methods of evaluating the quality of Alice's dialogue models in the Kazakh language: using a crowdsourcing project to collect human evaluations of model responses, prompting a large language model to evaluate model responses with no additional training, and training a binary classifier to evaluate response quality. We use a complex binary metric that accounts for the nuances of virtual assistant conversations to evaluate Alice's responses. Our work is the first to tackle the problem of evaluating the quality of dialogue models in Kazakh.