Investigation of Error Sources in a Cohort Selection Service Based on a Multimodal Large Language Models

This thesis research investigates the sources of errors that can arise when working with large multimodal language models (MLLM) for data analysis in healthcare, using the example of cohort selection of patients for a medical study. The relevance of the research topic is due to the increasing role of large multimodal language models in solving problems in various fields. MLLM represent a cutting-edge approach in IT technologies, allowing processing and analyzing data from various sources, such as text, images, audio and video. However, despite their impressive capabilities, multimodal language models can generate various types of errors. The goal of the thesis is to identify and study the sources of errors in the task of patient cohort selection, compile a classification of errors, calculate metrics and evaluate the quality of the model's performance, and select a prompt engineering method to reduce model hallucinations.