Emotions and Modality: Creating Emotional Speech Dataset with Multimodal Annotation

The results of theoretical investigations which explain the nature of human emotions are being widely used in applied fields, particularly in the field of human–computer interaction. Is there a chance that computers will be able to learn to recognize emotions? Research in the field of affective computing, to which this work belongs, is looking for an answer to this question. This research focuses on the problem of multimodal emotional data. The main purpose of the study is to develop and test a procedure for collecting and annotating a multimodal dataset of natural emotional speech in Russian, which can be applied to machine learning tasks. The author discusses the concepts of "multimodality" and "emotion" and provides an overview of various approaches to creating emotional corpora. A data collection method based on the Autobiographic Mood Induction Procedure is proposed and tested. To annotate the collected data, a multimodal emotional annotation system is proposed. Multimodal annotation implies conducting emotional assessment in four conditions, in each of which a limited number of communication channels are available to the annotators for perception. Using these methods, a dataset of emotional narratives in Russian has been created. Emotional data with multimodal annotation is analyzed in terms of the importance of different modalities for the recognition of six basic emotions. The conclusion is made about some features of recognizing joy, sadness, anger, surprise, disgust and fear, and the possibilities of using the collected dataset in automatic emotion recognition are discussed.