An Audit Framework for Technical Assessment of Binary Classifiers

Debarati Bhaumik, Diptish Dey

2023

Abstract

Multilevel models using logistic regression (MLogRM) and random forest models (RFM) are increasingly deployed in industry for the purpose of binary classification. The European Commission’s proposed Artificial Intelligence Act (AIA) necessitates, under certain conditions, that application of such models is fair, transparent, and ethical, which consequently implies technical assessment of these models. This paper proposes and demonstrates an audit framework for technical assessment of RFMs and MLogRMs by focussing on model-, discrimination-, and transparency & explainability-related aspects. To measure these aspects 20 KPIs are proposed, which are paired to a traffic light risk assessment method. An open-source dataset is used to train a RFM and a MLogRM model and these KPIs are computed and compared with the traffic lights. The performance of popular explainability methods such as kernel- and tree-SHAP are assessed. The framework is expected to assist regulatory bodies in performing conformity assessments of binary classifiers and also benefits providers and users deploying such AI-systems to comply with the AIA.

Paper Citation

in Harvard Style

Bhaumik D. and Dey D. (2023). An Audit Framework for Technical Assessment of Binary Classifiers. In Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-623-1, pages 312-324. DOI: 10.5220/0011744600003393

in Bibtex Style

@conference{icaart23,
author={Debarati Bhaumik and Diptish Dey},
title={An Audit Framework for Technical Assessment of Binary Classifiers},
booktitle={Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2023},
pages={312-324},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011744600003393},
isbn={978-989-758-623-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - An Audit Framework for Technical Assessment of Binary Classifiers
SN - 978-989-758-623-1
AU - Bhaumik D.
AU - Dey D.
PY - 2023
SP - 312
EP - 324
DO - 10.5220/0011744600003393