Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs

Phillip Schneider, Manuel Klettner, Kristiina Jokinen, Elena Simperl, Florian Matthes

2024

Abstract

Conversational question answering systems often rely on semantic parsing to enable interactive information retrieval, which involves the generation of structured database queries from a natural language input. For information-seeking conversations about facts stored within a knowledge graph, dialogue utterances are transformed into graph queries in a process that is called knowledge-based conversational question answering. This paper evaluates the performance of large language models that have not been explicitly pre-trained on this task. Through a series of experiments on an extensive benchmark dataset, we compare models of varying sizes with different prompting techniques and identify common issue types in the generated output. Our results demonstrate that large language models are capable of generating graph queries from dialogues, with significant improvements achievable through few-shot prompting and fine-tuning techniques, especially for smaller models that exhibit lower zero-shot performance.

Paper Citation

in Harvard Style

Schneider P., Klettner M., Jokinen K., Simperl E. and Matthes F. (2024). Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 807-814. DOI: 10.5220/0012394300003636

in Bibtex Style

@conference{icaart24,
author={Phillip Schneider and Manuel Klettner and Kristiina Jokinen and Elena Simperl and Florian Matthes},
title={Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2024},
pages={807-814},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012394300003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs
SN - 978-989-758-680-4
AU - Schneider P.
AU - Klettner M.
AU - Jokinen K.
AU - Simperl E.
AU - Matthes F.
PY - 2024
SP - 807
EP - 814
DO - 10.5220/0012394300003636
PB - SciTePress