Novelty and Outlier Detection

This master’s thesis focuses on enhancing anomaly detection methodologies within the credit scoring processes. Rapid anomaly detection is crucial for minimizing response times to failures and thereby preventing potential losses. The primary objective is to develop an automated system capable of real-time anomaly identification and alerting relevant personnel to possible issues. The study encompasses a comprehensive examination of existing anomaly detection techniques, particularly within time series data, and their applicability to credit scoring. It involves the development and implementation of detection algorithms and models, rigorous testing on real-world data, and an evaluation of the system’s effectiveness. The research begins by reviewing related work and defining key concepts such as anomalies and novelty detection. Anomalies, significant deviations from typical data patterns, can manifest as single data points, within specific contexts, or across entire datasets. Various detection algorithms, including statistical, nearest neighbor-based, and classification-based techniques, are discussed. The methodology includes understanding the data, focusing on metrics related to the frequency of certain rejection causes. The study excludes technical metrics to concentrate on business process anomalies that affect lending decisions. Data is aggregated in various time intervals to detect patterns indicative of anomalies. The implementation section details the construction of the model, addressing issues such as metric quantity and system limitations. The Isolation Forest model is used due to its robustness in detecting anomalies. The model is tested through back-testing and real-time data processing, with results showing significant improvements in anomaly detection. Finally, the thesis concludes by emphasizing the practical importance of the developed system for the banking sector. The system's ability to detect anomalies promptly reduces risks and potential losses. Future enhancements, such as incorporating additional data on system interconnections and release information from related applications, are proposed to improve detection accuracy and reliability further.