On the Hardness and Necessity of Supervised Concept Drift Detection

Fabian Hinder, Valerie Vaquet, Johannes Brinkrolf, Barbara Hammer

2023

Abstract

The notion of concept drift refers to the phenomenon that the distribution generating the observed data changes over time. If drift is present, machine learning models can become inaccurate and need adjustment. Many technologies for learning with drift rely on the interleaved test-train error to detect drift and trigger model updates. This type of drift detection is also used for monitoring systems aiming to detect anomalies. In this work, we analyze the relationship between concept drift and change of loss on a theoretical level. We focus on the sensitivity, specificity, and localization of change points in drift detection, putting an emphasize on the detection of real concept drift. With this focus, we compare the supervised and unsupervised setups which are already studied in the literature. We show that, unlike the unsupervised case, there is no universal supervised drift detector and that the assumed correlation between model loss and concept drift is invalid. We support our theoretical findings with empirical evidence for a combination of different models and data sets. We find that many state-of-the-art supervised drift detection methods suffer from insufficient sensitivity and specificity, and that unsupervised drift detection methods are a promising addition to existing supervised approaches.

Paper Citation

in Harvard Style

Hinder F., Vaquet V., Brinkrolf J. and Hammer B. (2023). On the Hardness and Necessity of Supervised Concept Drift Detection. In Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-626-2, pages 164-175. DOI: 10.5220/0011797500003411

in Bibtex Style

@conference{icpram23,
author={Fabian Hinder and Valerie Vaquet and Johannes Brinkrolf and Barbara Hammer},
title={On the Hardness and Necessity of Supervised Concept Drift Detection},
booktitle={Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2023},
pages={164-175},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011797500003411},
isbn={978-989-758-626-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - On the Hardness and Necessity of Supervised Concept Drift Detection
SN - 978-989-758-626-2
AU - Hinder F.
AU - Vaquet V.
AU - Brinkrolf J.
AU - Hammer B.
PY - 2023
SP - 164
EP - 175
DO - 10.5220/0011797500003411