Evaluating ChatGPT's Ability to Detect Naming Bugs in Java Methods

Kinari Nishiura, Atsuya Matsutomo, Akito Monden

2025

Abstract

In Java programming, large-scale and complex functions are realized by combining multiple methods. When method names do not match their functionality, readability decreases, making maintenance challenging. Although several machine learning models have been proposed to detect such naming bugs, they require extensive training data, limiting user accessibility. Recently, large language models (LLMs) like ChatGPT have gained popularity and show potential for code comprehension tasks. This study evaluates the performance of ChatGPT in detecting naming bugs using the same datasets as in previous machine learning studies. We evaluated detection accuracy through traditional methods, various prompt adjustments, and more direct approaches. The results indicate that, while ChatGPT does not surpass traditional models, it can match their accuracy with appropriately structured prompts, requiring no additional training.

Paper Citation

in Harvard Style

Nishiura K., Matsutomo A. and Monden A. (2025). Evaluating ChatGPT's Ability to Detect Naming Bugs in Java Methods. In Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE; ISBN 978-989-758-742-9, SciTePress, pages 128-136. DOI: 10.5220/0013354400003928

in Bibtex Style

@conference{enase25,
author={Kinari Nishiura and Atsuya Matsutomo and Akito Monden},
title={Evaluating ChatGPT's Ability to Detect Naming Bugs in Java Methods},
booktitle={Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE},
year={2025},
pages={128-136},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013354400003928},
isbn={978-989-758-742-9},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE
TI - Evaluating ChatGPT's Ability to Detect Naming Bugs in Java Methods
SN - 978-989-758-742-9
AU - Nishiura K.
AU - Matsutomo A.
AU - Monden A.
PY - 2025
SP - 128
EP - 136
DO - 10.5220/0013354400003928
PB - SciTePress