Application of Large Language Models and ReAct Prompting in Policy Evidence Collection

Yang Zhang, James Pope

2025

Abstract

Policy analysis or formulation often requires evidence-based support to ensure the scientific rigor and rationality of the policy, increase public trust, and reduce risks and uncertainties. However, manually collecting policy-related evidence is a time-consuming and tedious process, making some automated collection methods necessary. This paper presents a novel approach for automating policy evidence collection through large language models (LLMs) combined with Reasoning and Acting (ReAct) prompting. The advantages of our approach lie in its minimal data requirements, while ReAct prompting enables the LLM to call external tools, such as search engines, ensuring real-time evidence collection. Since this is a novel problem without existing methods for comparison, we relied on human experts for ground truth and baseline comparison. In 50 experiments, our method successfully collected correct policy evidence 36 times using GPT-3.5. Furthermore, with more advanced models such as GPT-4o, the improved understanding of prompts and context enhances our method’s efficiency. Finally, our method using GPT-4o successfully gathered correct evidence 45 times in 50 experiments. Our results demonstrate that, using our method, policy researchers can effectively gather evidence to support policy-making.

Paper Citation

in Harvard Style

Zhang Y. and Pope J. (2025). Application of Large Language Models and ReAct Prompting in Policy Evidence Collection. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 967-974. DOI: 10.5220/0013243000003890

in Bibtex Style

@conference{icaart25,
author={Yang Zhang and James Pope},
title={Application of Large Language Models and ReAct Prompting in Policy Evidence Collection},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2025},
pages={967-974},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013243000003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - Application of Large Language Models and ReAct Prompting in Policy Evidence Collection
SN - 978-989-758-737-5
AU - Zhang Y.
AU - Pope J.
PY - 2025
SP - 967
EP - 974
DO - 10.5220/0013243000003890
PB - SciTePress