Leveraging Large Language Models for Semantic Evaluation of RDF Triples

André Regino, Fernando Rezende Zagatti, Fernando Rezende Zagatti, Rodrigo Bonacin, Victor Jesus Sotelo Chico, Victor Hochgreb, Julio Reis

2025

Abstract

Knowledge Graphs (KGs) depend on accurate RDF triples, making the quality assurance of these triples a significant challenge. Large Language Models (LLMs) can serve as graders for RDF data, providing scalable alternatives to human validation. This study evaluates the feasibility of utilizing LLMs to assess the quality of RDF triples derived from natural language sentences in the e-commerce sector. We analyze 12 LLM configurations by comparing their Likert-scale ratings of triple quality with human evaluations, focusing on both complete triples and their individual components (subject, predicate, object). We employ statistical correlation measures (Spearman and Kendall Tau) to quantify the alignment between LLM and expert assessments. Our study examines whether justifications generated by LLMs can indicate higher-quality grading. Our findings reveal that some LLMs demonstrate moderate agreement with human annotators and none achieve full alignment. This study presents a replicable evaluation framework and emphasizes the current limitations and potential of LLMs as semantic validators. These results support efforts to incorporate LLM-based validation into KG construction processes and suggest avenues for prompt engineering and hybrid human-AI validation systems.

Paper Citation

in Harvard Style

Regino A., Zagatti F., Bonacin R., Chico V., Hochgreb V. and Reis J. (2025). Leveraging Large Language Models for Semantic Evaluation of RDF Triples. In Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD; ISBN 978-989-758-769-6, SciTePress, pages 74-85. DOI: 10.5220/0013837600004000

in Bibtex Style

@conference{keod25,
author={André Regino and Fernando Zagatti and Rodrigo Bonacin and Victor Chico and Victor Hochgreb and Julio Reis},
title={Leveraging Large Language Models for Semantic Evaluation of RDF Triples},
booktitle={Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD},
year={2025},
pages={74-85},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013837600004000},
isbn={978-989-758-769-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD
TI - Leveraging Large Language Models for Semantic Evaluation of RDF Triples
SN - 978-989-758-769-6
AU - Regino A.
AU - Zagatti F.
AU - Bonacin R.
AU - Chico V.
AU - Hochgreb V.
AU - Reis J.
PY - 2025
SP - 74
EP - 85
DO - 10.5220/0013837600004000
PB - SciTePress