Authors:
André F. R. Cordeiro
and
Edson Oliveira Jr
Affiliation:
Informatics Department, State University of Maringá, Colombo Avenue - 5790, Maringá, Brazil
Keyword(s):
Reproducibility, Research Opportunities, Review, Software Engineering Research, Systematic Mapping.
Abstract:
This paper presents a Systematic Mapping Study (SMS) on reproducibility in Software Engineering (SE), analyzing definitions, procedures, investigations, solutions, artifacts, and evaluation assessments. The research explores how reproducibility is defined, applied, and investigated, identifying several approaches and solutions. The final set of studies considered 25 primary studies, grouping the definitions of reproducibility into categories such as method, repeatability, probability, and ability. The application procedures are categorized into method, architecture, container, technique, framework, environment, notebook, toolkit, and benchmark. The investigation of reproducibility is analyzed through workflows, approaches, prototypes, methods, measures, methodologies, technologies, case studies, and frameworks. Solutions to reproducibility problems include environments, tools, benchmarks, initiatives, methodologies, notebooks, and containers. The artifacts considered include tools, e
nvironments, scenarios, datasets, models, diagrams, notebooks, algorithms, codes, representation structures, methodologies, containers, repositories, sequences, and workflows. Reproducibility assessment is performed using methods, experiments, measurements, processes, and factors. We also discuss future research opportunities. The results aim to benefit SE researchers and practitioners by providing an overview of organizing and providing reproducible research projects and artifacts, in addition to pointing out research opportunities related to reproducibility in the area.
(More)