# Are Concept Drift Detectors Reliable Alarming Systems? -- A Comparative Study

**Authors:** Lorena Poenaru-Olaru, Luis Cruz, Arie van Deursen, Jan S. Rellermeyer  
**arXiv:** [2211.13098](https://arxiv.org/abs/2211.13098)  
**Date:** 2022-11-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.13098v1/

## Key points

- Study explores reliability of concept drift detectors in identifying drift in machine learning models over time
- Machine learning models are replacing traditional business logic in production systems, making their lifecycle management a concern
- Concept drift detectors are used to identify shifts in data patterns that can impact model performance
- Study compares performance of error rate-based and data distribution-based concept drift detectors on synthetic and real-world datasets
- Findings provide practical guidelines for using concept drift detectors effectively
- Analysis determines suitability of each detector group as an alarming system for real-time production systems
- Study contributes to addressing concerns related to managing machine learning model lifecycles

## Summary

In this study titled "Are Concept Drift Detectors Reliable Alarming Systems? -- A Comparative Study," authors Lorena Poenaru-Olaru, Luis Cruz, Arie van Deursen, and Jan S. Rellermeyer explore the reliability of concept drift detectors in identifying drift in machine learning models over time. As machine learning models increasingly replace traditional business logic in production systems, their lifecycle management becomes a significant concern. These models are constantly evaluated on new streaming data which often exhibits shifting data patterns known as concept drift. Concept drift can significantly impact the performance of machine learning models, making it crucial to identify when it occurs. Concept drift detectors are used to identify these shifts in data patterns. The authors assess the reliability of concept drift detectors by investigating how late they report drifts and how many false alarms they signal. The study compares the performance of popular drift detectors belonging to two different groups: error rate-based detectors and data distribution-based detectors. The evaluation is conducted on both synthetic and real-world datasets. For synthetic data, the researchers specifically investigate the performance of detectors in identifying two types of concept drift: abrupt and gradual. The findings aim to help practitioners understand which specific drift detector should be employed in different situations. To achieve this goal, the authors provide a list of important observations made throughout the study that can serve as practical guidelines for using concept drift detectors effectively. Additionally, based on empirical results, the suitability of each concept drift detection group as an alarming system is analyzed. This analysis provides insights into whether error rate-based or data distribution-based detectors are more suitable for detecting concept drift in real-time production systems. Overall, this comparative study contributes to addressing concerns related to managing machine learning model lifecycles by assessing the reliability and performance of various concept drift detectors.