Spread and (Mis)use of Evaluative Expressions in Human Written and LLM-Based Generated Text

Maurice Langner, Ralf Klabunde

2024

Abstract

We investigate the capacity of Large Language Models (LLMs) to generate evaluative expressions in a data-driven manner. The linguistic object of investigation is the production of justified and adequate evaluative language, such that the evaluative stance of the text is motivated by the underlying data. We use the SportSett corpus for generating summaries of basketball games. The input data is converted into RDF triples that are fed into GPT-4 and GPT-3.5, prompting the models to produce game summaries using evaluative adverbs and judgemental language. We annotated the generated texts and the original summaries for their propositional content contained in the line score and box score of each game, as well as for evaluative adverbs and their polarity. The results show that the models struggle to correctly interpret the numerical data and coherently assess the quality of team-wise and player-wise performances both within games and across games, often producing contradictory evaluations and displaying the lack of global evaluative scales.

Paper Citation

in Harvard Style

Langner M. and Klabunde R. (2024). Spread and (Mis)use of Evaluative Expressions in Human Written and LLM-Based Generated Text. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 646-654. DOI: 10.5220/0012590600003636

in Bibtex Style

@conference{icaart24,
author={Maurice Langner and Ralf Klabunde},
title={Spread and (Mis)use of Evaluative Expressions in Human Written and LLM-Based Generated Text},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2024},
pages={646-654},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012590600003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - Spread and (Mis)use of Evaluative Expressions in Human Written and LLM-Based Generated Text
SN - 978-989-758-680-4
AU - Langner M.
AU - Klabunde R.
PY - 2024
SP - 646
EP - 654
DO - 10.5220/0012590600003636
PB - SciTePress