Authors:
Laura Steffny
;
Nanna Dahlem
;
Robert Becker
and
Dirk Werth
Affiliation:
August-Wilhelm Scheer Institute for Digital Products and Processes gGmbH, Saarbrücken, Germany
Keyword(s):
Zero-Shot Prompting, Cross-Verification, Chain-of-Verification, CoV, Large Language Model, LLM, Data Extraction, Care, Documentation.
Abstract:
Automating care documentation through artificial intelligence (AI), particularly using large language models (LLMs), has great potential to improve workflow and efficiency in healthcare applications. However, in clinical or care environments where errors can have serious consequences, ensuring the reliability and accuracy of LLM output is essential. Zero-shot prompting, an advanced technique that does not require task-specific training data, shows promising results for data extraction in domains where large, well-structured datasets are scarce. This paper investigates how cross-verification affects zero-shot prompting performance in extracting relevant care indicators from unbalanced nursing documentation. The extraction was evaluated for three indicators on a dataset of care documentation from 38 participants across two facilities. The results show cross-verification significantly improves extraction accuracy, particularly by reducing false positives. While term extraction alone ach
ieved around 80% accuracy, at lower temperature settings (0.1) cross-verification increased accuracy to 96.74%. However, cross-verification also increased missed terms when no corresponding sentences were found, even though terms were in the ground truth. This study highlights the potential of cross-verification in care documentation and offers suggestions for further optimization, especially with unstructured text and unbalanced data.
(More)