UX of Chatbots: An Exploratory Study on Acceptance of User Experience Evaluation Methods

Marcus Barbosa, Walter Nakamura, Pedro Valle, Guilherme Guerino, Alice Finger, Gabriel Lunardi, Williamson Silva

2022

Abstract

Companies increasingly invest in designing, developing, and evaluating conversational agents, mainly text-based chatbots. Chatbots have become the main and the fastest communication channel for providing customer service and helping users interact with systems and, consequently, obtain the requested information. Despite this, the potential market for chatbots,there is still too little known about evaluating the quality of chatbots from a User eXperience (UX) perspective, i.e., the emotions, feelings, and expectations of users towards this application. Besides, relatively little research addresses the feasibility and applicability of UX methods (generic or not) or how to adapt them (if necessary) to evaluate chatbots. The goal of this research is to investigate the adequacy, feasibility of use, and acceptance by users of UX methods when employed to evaluate a chatbot. To achieve this objective, we conducted an exploratory study comparing three UX methods: AttrakDiff, Think Aloud, and Method for the Assessment of eXperience. We compared these methods by assessing the degree of ease of use, usefulness, self-predicted future use from end-users. Also, we performed follow-up interviews to understand users’ perceptions about each method. The results show that users preferred to use the Think Aloud method due to the ease and freedom that the user has to express their positive and negative emotions/feelings while using the chatbot. Based on the results, we believe that combining the three methods was essential to capture the whole user experience when using the chatbot.

Paper Citation

in Harvard Style

Barbosa M., Nakamura W., Valle P., Guerino G., Finger A., Lunardi G. and Silva W. (2022). UX of Chatbots: An Exploratory Study on Acceptance of User Experience Evaluation Methods. In Proceedings of the 24th International Conference on Enterprise Information Systems - Volume 1: ICEIS, ISBN 978-989-758-569-2, pages 355-363. DOI: 10.5220/0011090100003179

in Bibtex Style

@conference{iceis22,
author={Marcus Barbosa and Walter Nakamura and Pedro Valle and Guilherme Guerino and Alice Finger and Gabriel Lunardi and Williamson Silva},
title={UX of Chatbots: An Exploratory Study on Acceptance of User Experience Evaluation Methods},
booktitle={Proceedings of the 24th International Conference on Enterprise Information Systems - Volume 1: ICEIS,},
year={2022},
pages={355-363},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011090100003179},
isbn={978-989-758-569-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 24th International Conference on Enterprise Information Systems - Volume 1: ICEIS,
TI - UX of Chatbots: An Exploratory Study on Acceptance of User Experience Evaluation Methods
SN - 978-989-758-569-2
AU - Barbosa M.
AU - Nakamura W.
AU - Valle P.
AU - Guerino G.
AU - Finger A.
AU - Lunardi G.
AU - Silva W.
PY - 2022
SP - 355
EP - 363
DO - 10.5220/0011090100003179