Specifics of LLM chat bot evaluation: case of chat-bot summarizing telegram channel contents

Student: Garri Dassayev

Faculty: St. Petersburg School of Physics, Mathematics, and Computer Science

Educational Programme: UX Analytics and Information System Design (Master)

Year of Graduation: 2024

Telegram users face the challenge of processing a large flow of information from subscriptions to Telegram channels in the context of the rapid growth of Telegram's user base. It now exceeds 800 million people. This study demonstrates the development of a chatbot that summarizes content from Telegram channels based on large language models (LLMs) and delivers it to the user. Additionally, the work emphasizes how the quality of content summarization can be evaluated in the context of LLMs. The following artifacts are presented as a result: a working chatbot for summarizing content from Telegram channels, recommendations for comparing the quality of LLM performance with a benchmark, and a publicly available dataset of Telegram channel posts with calculated summarization quality metrics.

Student Theses at HSE must be completed in accordance with the University Rules and regulations specified by each educational programme.

Summaries of all theses must be published and made freely available on the HSE website.

The full text of a thesis can be published in open access on the HSE website only if the authoring student (copyright holder) agrees, or, if the thesis was written by a team of students, if all the co-authors (copyright holders) agree. After a thesis is published on the HSE website, it obtains the status of an online publication.

Student theses are objects of copyright and their use is subject to limitations in accordance with the Russian Federation’s law on intellectual property.

In the event that a thesis is quoted or otherwise used, reference to the author’s name and the source of quotation is required.

Search all student theses