loading
Papers Papers/2022 Papers Papers/2022

Research.Publish.Connect.

Paper

Authors: Carlo Bellettini 1 ; Michael Lodi 1 ; 2 ; 3 ; Violetta Lonati 1 ; 3 ; Mattia Monga 1 ; 3 and Anna Morpurgo 1 ; 3

Affiliations: 1 Università degli Studi di Milano, Milan, Italy ; 2 Alma Mater Studiorum, Università di Bologna, Bologna, Italy ; 3 Laboratorio Nazionale CINI ‘Informatica e Scuola’, Rome, Italy

Keyword(s): Bebras, GPT-3, Large Language Models, Computer Science Education.

Abstract: In this paper we study the problem-solving ability of the Large Language Model known as GPT-3 (codename DaVinci), by considering its performance in solving tasks proposed in the “Bebras International Challenge on Informatics and Computational Thinking”. In our experiment, GPT-3 was able to answer with a majority of correct answers about one third of the Bebras tasks we submitted to it. The linguistic fluency of GPT-3 is impressive and, at a first reading, its explanations sound coherent, on-topic and authoritative; however the answers it produced are in fact erratic and the explanations often questionable or plainly wrong. The tasks in which the system performs better are those that describe a procedure, asking to execute it on a specific instance of the problem. Tasks solvable with simple, one-step deductive reasoning are more likely to obtain better answers and explanations. Synthesis tasks, or tasks that require a more complex logical consistency get the most incorrect answers.

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 44.200.86.95

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Bellettini, C.; Lodi, M.; Lonati, V.; Monga, M. and Morpurgo, A. (2023). Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3. In Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU; ISBN 978-989-758-641-5; ISSN 2184-5026, SciTePress, pages 59-69. DOI: 10.5220/0012007500003470

@conference{csedu23,
author={Carlo Bellettini. and Michael Lodi. and Violetta Lonati. and Mattia Monga. and Anna Morpurgo.},
title={Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3},
booktitle={Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU},
year={2023},
pages={59-69},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012007500003470},
isbn={978-989-758-641-5},
issn={2184-5026},
}

TY - CONF

JO - Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU
TI - Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3
SN - 978-989-758-641-5
IS - 2184-5026
AU - Bellettini, C.
AU - Lodi, M.
AU - Lonati, V.
AU - Monga, M.
AU - Morpurgo, A.
PY - 2023
SP - 59
EP - 69
DO - 10.5220/0012007500003470
PB - SciTePress