Exploring Customer Service Agent Preferences for Conversational and Keyword-Based Information Retrieval

Nektarios Machner, Yaren Mändle, Florian Matthes

2025

Abstract

Effective knowledge discovery and information retrieval drive organizational innovation and competitive advantage. To support this, organizations have long used knowledge management systems that historically have relied on keyword-based search. The rise of artificial intelligence (AI), most notably large language models (LLMs), has enabled conversational search (CS) interfaces that understand natural-language queries, synthesize information from multiple sources, and generate answers. This study investigates the factors that influence customer service agents’ preferences for conversational search versus traditional keyword-based search within an internal knowledge management system. Set in a large European insurance company, we employ a mixed-methods empirical approach, integrating semi-structured interviews (n = 13), a structured survey (n = 17), and log-file analysis of 508 real-world queries. Our research explores which factors drive agents’ choice between the two search approaches, and examines the practical strengths and limitations of each approach. Our findings reveal that agents choose keyword search when they are confident of where to look and conversational search when they need natural-language guidance, with trust and time constraints further tipping the balance. This complementarity suggests hybrid interfaces, blending ease of use, reliable results, and flexible query handling, best support agents’ workflows.

Paper Citation

in Harvard Style

Machner N., Mändle Y. and Matthes F. (2025). Exploring Customer Service Agent Preferences for Conversational and Keyword-Based Information Retrieval. In Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR; ISBN , SciTePress, pages 349-356. DOI: 10.5220/0013721400004000

in Bibtex Style

@conference{kdir25,
author={Nektarios Machner and Yaren Mändle and Florian Matthes},
title={Exploring Customer Service Agent Preferences for Conversational and Keyword-Based Information Retrieval},
booktitle={Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR},
year={2025},
pages={349-356},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013721400004000},
isbn={},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR
TI - Exploring Customer Service Agent Preferences for Conversational and Keyword-Based Information Retrieval
SN -
AU - Machner N.
AU - Mändle Y.
AU - Matthes F.
PY - 2025
SP - 349
EP - 356
DO - 10.5220/0013721400004000
PB - SciTePress