Verifying LLM-Generated Code in the Context of Software Verification with Ada/SPARK

Marcos Cramer, Lucian McIntyre

2025

Abstract

Large language models (LLMs) have demonstrated remarkable code generation capabilities, but the correctness of the generated code cannot be inherently trusted. This paper explores the feasibility of using formal software verification, specifically the SPARK framework for Ada, to ensure the reliability of LLM-generated code. We present Marmaragan, a tool that leverages an LLM in order to generate SPARK annotations for existing programs, enabling formal verification of the code. The tool is benchmarked on a curated set of SPARK programs, with annotations selectively removed to test specific capabilities. The performance of Marmaragan with GPT-4o on the benchmark is promising, with correct annotations having been generated for 50.7% of the benchmark cases. The results establish a foundation for future work on combining the power of LLMs with the reliability of formal software verification.

Paper Citation

in Harvard Style

Cramer M. and McIntyre L. (2025). Verifying LLM-Generated Code in the Context of Software Verification with Ada/SPARK. In Proceedings of the 20th International Conference on Software Technologies - Volume 1: ICSOFT; ISBN 978-989-758-757-3, SciTePress, pages 39-50. DOI: 10.5220/0013461900003964

in Bibtex Style

@conference{icsoft25,
author={Marcos Cramer and Lucian McIntyre},
title={Verifying LLM-Generated Code in the Context of Software Verification with Ada/SPARK},
booktitle={Proceedings of the 20th International Conference on Software Technologies - Volume 1: ICSOFT},
year={2025},
pages={39-50},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013461900003964},
isbn={978-989-758-757-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 20th International Conference on Software Technologies - Volume 1: ICSOFT
TI - Verifying LLM-Generated Code in the Context of Software Verification with Ada/SPARK
SN - 978-989-758-757-3
AU - Cramer M.
AU - McIntyre L.
PY - 2025
SP - 39
EP - 50
DO - 10.5220/0013461900003964
PB - SciTePress