Authors:
Sayed Arefin
;
Tasnia Heya
;
Hasan Al-Qudah
;
Ynes Ineza
and
Abdul Serwadda
Affiliation:
Texas Tech University, Lubbock, Texas, U.S.A.
Keyword(s):
Large Language Models, ChatGPT, Code Smells, Algorithms.
Abstract:
We conduct an extensive analysis of ChatGPT, a standout Large Language Model (LLM), particularly in coding within the Python language, focusing on data structures and algorithms. We assess ChatGPT’s ability to accurately solve coding problems, its code quality, and the nature of run-time errors. Additionally, we examine how ChatGPT’s code performs when it executes but doesn’t solve the problem, identifying error patterns. We also explore whether ChatGPT has memorized training data through a structured experiment. Comparing with human performance where possible, our study encompasses both GPT-3.5 and GPT-4 models, various subtopics within the main areas, and problems of different complexities.