And Where to Laugh: Do Neural Networks Really Solve the Humor Detection Task

Humor, like many things in the human psyche, is a difficult-to-formalize category. Many problems that cannot be solved with the help of clear algorithms are being tried to solve with the help of deep learning of neural network models. Humor detection is one of these tasks. The main difficulty in training models for binary text classification is to contrast humorous texts in the dataset with non—humorous ones similar to them. We have created A_HDE (Advanced Humor Detection Examples) dataset with texts in Russian for training and testing humor detection algorithms. The dataset consists of 1,608 humorous texts and their minimally modified 1,590 non-humorous versions. Based on modern work in psychology and linguistics, we have limited the range of humorous mechanisms that we have included texts with in the dataset. The RuBERT and Conversational RuBERT models trained on A_HDE and FUN [Blinov et al. 2019] showed low results on the A_HDE test part (Matthew's Correlation Coefficient < 0.1199). We found that the Conversational RuBERT model recognized one of the humorous mechanisms — Garden Path [Dynel 2012] — better than the others. The dataset is not suitable for full-fledged training, but it can be used as a benchmark for humor detection models or for research on the interpretability of neural networks.