Authors:
Nektarios Machner
;
Yaren Mändle
and
Florian Matthes
Affiliation:
Department of Computer Science, Technical University of Munich, Munich, Germany
Keyword(s):
Conversational Search, Knowledge Discovery, Knowledge Management, Information Retrieval, Customer Service.
Abstract:
Effective knowledge discovery and information retrieval drive organizational innovation and competitive advantage. To support this, organizations have long used knowledge management systems that historically have relied on keyword-based search. The rise of artificial intelligence (AI), most notably large language models (LLMs), has enabled conversational search (CS) interfaces that understand natural-language queries, synthesize information from multiple sources, and generate answers. This study investigates the factors that influence customer service agents’ preferences for conversational search versus traditional keyword-based search within an internal knowledge management system. Set in a large European insurance company, we employ a mixed-methods empirical approach, integrating semi-structured interviews (n = 13), a structured survey (n = 17), and log-file analysis of 508 real-world queries. Our research explores which factors drive agents’ choice between the two search approach
es, and examines the practical strengths and limitations of each approach. Our findings reveal that agents choose keyword search when they are confident of where to look and conversational search when they need natural-language guidance, with trust and time constraints further tipping the balance. This complementarity suggests hybrid interfaces, blending ease of use, reliable results, and flexible query handling, best support agents’ workflows.
(More)