loading
Papers Papers/2022 Papers Papers/2022

Research.Publish.Connect.

Paper

Authors: Vanessa Bracamonte ; Seira Hidano and Shinsaku Kiyomoto

Affiliation: KDDI Research, Inc., Saitama, Japan

Keyword(s): User Perception, Errors, Machine Learning Model Evaluation, User Study.

Abstract: Information such as accuracy and outcome explanations can be useful for the evaluation of machine learning systems, but they can also lead to over-trust. This means that an evaluator may not have suspicion that a machine learning system could have errors, and that they may overlook problems in the explanation of those systems. Research has shown that errors not only decrease trust but can also promote curiosity about the performance of the system. Therefore, presenting errors to evaluators may be an option to induce suspicion in the context of the evaluation of a machine learning system. In this paper, we evaluate this possibility by conducting three experiments where we asked participants to evaluate text classification systems. We presented two types of errors: incorrect predictions and errors in the explanation. The results show that patterns of errors in explanation negatively influenced willingness to recommend a system, and that fewer participants chose a system with higher acc uracy when there was an error pattern, compared to when the errors were random. Moreover, more participants gave evidence from the explanations in their reason for their evaluation of the systems, suggesting that they were able to detect error patterns. (More)

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 3.237.232.196

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Bracamonte, V.; Hidano, S. and Kiyomoto, S. (2022). Effect of Errors on the Evaluation of Machine Learning Systems. In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - HUCAPP; ISBN 978-989-758-555-5; ISSN 2184-4321, SciTePress, pages 48-57. DOI: 10.5220/0010839300003124

@conference{hucapp22,
author={Vanessa Bracamonte. and Seira Hidano. and Shinsaku Kiyomoto.},
title={Effect of Errors on the Evaluation of Machine Learning Systems},
booktitle={Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - HUCAPP},
year={2022},
pages={48-57},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010839300003124},
isbn={978-989-758-555-5},
issn={2184-4321},
}

TY - CONF

JO - Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - HUCAPP
TI - Effect of Errors on the Evaluation of Machine Learning Systems
SN - 978-989-758-555-5
IS - 2184-4321
AU - Bracamonte, V.
AU - Hidano, S.
AU - Kiyomoto, S.
PY - 2022
SP - 48
EP - 57
DO - 10.5220/0010839300003124
PB - SciTePress