Extending the Ability to Diagnose Data Missingness Mechanism through Machine Learning

In sociological research, there are often gaps in data due to various reasons, for example, the refusal of respondents to answer a particular question. When dealing with incomplete data, there is a need to determine whether omissions can be avoided without changing data structure or whether data imputation is needed. Using missing data mechanisms allows for researchers to realize which missing data mechanism they deal with: MCAR, MAR or MNAR, and it is possible to ignore missing data in a particular variable when analyzing. Current methods of diagnosing have limitations in distinguishing between random and non-random omissions. The purpose of this study is to examine the use of machine learning techniques to enhance the capabilities of diagnosing missing data mechanisms. By training several machine learning models on the generated subsamples with gaps, the confusion matrix and quality metrics such as accuracy for models were calculated. In the result, the HistGragientBoostingClassifier showed the ability to correctly classify 57% of the sample observations, in particular, it identified the MNAR well. This allowed us to conclude that machine learning methods are applicable both for diagnosing mechanisms of missing data in general and for separating non-random missing data from random ones, and to highlight the potential of work in this direction to improve the quality of diagnostics.