Efficient Neural Network Algorithms for Multimodal Emotion Recognition

This thesis examines multimodal emotion recognition (MER) and proposes an efficient neural network ensemble that uses facial expressions and speech patterns to improve accuracy in real-world scenarios. Motivated by the limitations of unimodal approaches, we explore the fusion of visual and audio information, a strategy widely recognized for its potential to capture nuanced and complex emotional states. A comprehensive literature review reveals the increasing significance of deep learning techniques for multimodal fusion and temporal modeling. Our proposed model includes an ensemble pre-trained neural networks for feature extraction from video, then early fusion with Transformer encoders or multilayer perceptron (MLP). To ensure robustness to varying frame rates and computational efficiency, we implement smoothing and an adaptive frame rate technique based on multiple testing corrections. This allows us to dynamically adjust the level of detail in our analysis, striking a balance between accuracy and computational cost. We conduct experiments on the Aff-Wild2 dataset, a large-scale dataset of in-the-wild videos exhibiting spontaneous emotions. Results show that our proposed model, specifically our multimodal ensemble with MLP fusion, outperforms several existing methods, particularly in expression classification. We also demonstrate the effectiveness of smoothing techniques, like mean and median filtering, in improving the stability and accuracy of frame-level emotion predictions. The findings of this thesis contribute to the advancement of emotion recognition technology by demonstrating the power of multimodal fusion, temporal modeling, smoothing and adaptive frame rate techniques. The developed model holds significant potential for applications in human-computer interaction, mental health assessment, and other domains where understanding emotional states is crucial.