Designing Actionable and Interpretable Analytics Indicators for Improving Feedback in AI-Based Systems

Esther Félix, Elaine De Oliveira, Ilmara M. M. Ramos, Mar Perez-Sanagustin, Esteban Villalobos, Isabel Hilliger, Rafael Ferreira Mello, Julien Broisin

2025

Abstract

In AI-based educational systems, transparency and understandability are particularly important to ensure reliable human-AI interaction. This paper contributes to the ongoing research on developing analytics for AI-based educational systems by delivering feedback throughout indicators that learners can easily interpret and act upon during their studies. Specifically, this paper introduces a mixed methods study that examines the types of indicators that ought to be incorporated into the feedback offered by an AI-based system designed to help students develop competencies in programming. Building upon prior work in Human-Centered Design, the card sorting technique was used to collect both qualitative and quantitative data from 31 Computer Science students. We created 16 cards that presented students with different indicators to explain the reasoning behind the system’s decisions and feedback. These indicators were displayed in different formats (visual and textual representations; temporal vs. non-temporal and social vs. non-social reference frames). Our goal was to discover the most interpretable and actionable method for delivering feedback to learners. Our study found low consensus among students. Overall, students found indicators based on social comparison to be less actionable and interpretable compared to those without; and textual indicators were perceived as less actionable and interpretable than visual ones.

Paper Citation

in Harvard Style

Félix E., De Oliveira E., Ramos I., Perez-Sanagustin M., Villalobos E., Hilliger I., Mello R. and Broisin J. (2025). Designing Actionable and Interpretable Analytics Indicators for Improving Feedback in AI-Based Systems. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: CSEDU; ISBN 978-989-758-746-7, SciTePress, pages 428-435. DOI: 10.5220/0013294300003932

in Bibtex Style

@conference{csedu25,
author={Esther Félix and Elaine De Oliveira and Ilmara Ramos and Mar Perez-Sanagustin and Esteban Villalobos and Isabel Hilliger and Rafael Mello and Julien Broisin},
title={Designing Actionable and Interpretable Analytics Indicators for Improving Feedback in AI-Based Systems},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: CSEDU},
year={2025},
pages={428-435},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013294300003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: CSEDU
TI - Designing Actionable and Interpretable Analytics Indicators for Improving Feedback in AI-Based Systems
SN - 978-989-758-746-7
AU - Félix E.
AU - De Oliveira E.
AU - Ramos I.
AU - Perez-Sanagustin M.
AU - Villalobos E.
AU - Hilliger I.
AU - Mello R.
AU - Broisin J.
PY - 2025
SP - 428
EP - 435
DO - 10.5220/0013294300003932
PB - SciTePress