Multimodal Emotion Recognition Based on Lightweight Neural Descriptors

Emotion recognition is a crucial aspect of human-computer interaction, with broad applications ranging from psychological analysis and healthcare to advanced security systems. Its significance lies in its ability to accurately interpret human emotions, which is essential for designing more empathetic and intuitive user interfaces. This thesis specifically focuses on the Expression Classification (EXPR) task within the context of the ABAW5 competition. The ABAW (Affective Behavior Analysis in-the-wild) competitions have played a pivotal role in advancing the field of emotion recognition by providing large-scale datasets such as Aff-Wild2 and promoting the development of innovative solutions. These competitions have significantly contributed to research and practical applications in affective computing. The primary objectives of this research are: • Literature Review. Conduct a comprehensive literature review, focusing on recent innovations in emotion recognition, particularly those involving multimodal data and deep learning; • Multimodal Emotion Recognition Model Development. Develop a multimodal emotion recognition model that integrates visual, auditory, and textual data, utilizing lightweight neural descriptors to enhance computational efficiency and model scalability; • Experimental Studies. Conduct experimental studies to evaluate the performance of the developed model against established benchmarks. The challenge involves developing a system that can accurately interpret a wide range of human emotions across diverse real-world environments, while operating within the constraints of limited computational resources.