Answering the Call to Go Beyond Accuracy: An Online Tool for the Multidimensional Assessment of Decision Support Systems

Chiara Natali, Andrea Campagner, Federico Cabitza, Federico Cabitza

2024

Abstract

The research about, and use of, AI-based Decision Support Systems (DSS) has been steadily increasing in the recent years: however, tools and techniques to validate and evaluate these systems in an holistic manner are still largely lacking, especially in regard to their potential impact on actual human decision-making. This paper challenges the accuracy-centric paradigm in DSS evaluation by introducing the nuanced, multi-dimensional approach of the DSS Quality Assessment Tool. Developed at MUDI Lab (University of Milano-Bicocca), this free, open-source tool supports the quality assessment of AI-based decision support systems (DSS) along six different and complementary dimensions: model robustness, data similarity, calibration, utility, data reliability and impact on human decision making. Each dimension is analyzed for its relevance in the Medical AI domain, the metrics employed, and their visualizations, designed according to the principle of vague visualizations to promote cognitive engagement. Such a tool can be instrumental to foster a culture of continuous oversight, outcome monitoring, and reflective technology assessment.

Paper Citation

in Harvard Style

Natali C., Campagner A. and Cabitza F. (2024). Answering the Call to Go Beyond Accuracy: An Online Tool for the Multidimensional Assessment of Decision Support Systems. In Proceedings of the 17th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 2: HEALTHINF; ISBN 978-989-758-688-0, SciTePress, pages 219-229. DOI: 10.5220/0012471600003657

in Bibtex Style

@conference{healthinf24,
author={Chiara Natali and Andrea Campagner and Federico Cabitza},
title={Answering the Call to Go Beyond Accuracy: An Online Tool for the Multidimensional Assessment of Decision Support Systems},
booktitle={Proceedings of the 17th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 2: HEALTHINF},
year={2024},
pages={219-229},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012471600003657},
isbn={978-989-758-688-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 2: HEALTHINF
TI - Answering the Call to Go Beyond Accuracy: An Online Tool for the Multidimensional Assessment of Decision Support Systems
SN - 978-989-758-688-0
AU - Natali C.
AU - Campagner A.
AU - Cabitza F.
PY - 2024
SP - 219
EP - 229
DO - 10.5220/0012471600003657
PB - SciTePress