Authors:
André F. R. Cordeiro
and
Edson Oliveira Jr
Affiliation:
Informatics Department, State University of Maringá, Colombo Avenue - 5790, Maringá, Brazil
Keyword(s):
Reproducibility, Research Opportunities, Review, Software Engineering Research, Systematic Mapping, Controlled Experimentation.
Abstract:
Reproducibility can be described as a characteristic that contributes to expanding knowledge in science. This paper investigates the reproducibility of experiments in Software Engineering (SE) in a context where the literature points to challenges in verifying experimental results. The central problem addressed is the difficulty in reproducing experiments in SE due to the different factors, such as sharing and artifact management. We then aimed to identify the factors necessary to achieve reproducibility in SE experiments, characterizing these factors in terms of the reproducibility crisis, experimental workflows, research practices, FAIR principles application, and reproducibility improvements. We planned and conducted one survey with 16 participants who answered a questionnaire with 33 questions. The results show that most participants perceive a reproducibility crisis in the field and point to factors such as lack of public data and incomplete information on methods and experiment
al setups as the main causes. Furthermore, the results highlight the importance of sharing data, metadata, and information about research teams. We also provide points to possible actions to improve reproducibility in SE experiments. The contributions include a detailed analysis of the challenges to reproducibility in SE, as well as the identification of practices and measures that can improve reproducibility.
(More)