Authors:
Richard de Arruda Felix
;
Patricia Della Méa Plentz
and
Jean Hauck
Affiliation:
Federal University of Santa Catarina, Florianópolis, Brazil
Keyword(s):
Batch Processing, Big Data, Data Science, Agility.
Abstract:
Data science has become essential across industries such as government, healthcare, and finance, driving decision-making through large-scale data analysis. Deploying batch data products, like the periodic calculation of credit scores for millions, presents significant challenges, including integration with existing big data architectures and ensuring scalability and efficiency. This study proposes an optimized approach that leverages software engineering and agile methodologies to streamline the deployment of such products. Validated through action research conducted at a Brazilian credit bureau, the approach demonstrated a substantial reduction in deployment time by improving documentation, development, and testing processes, offering a scalable solution to modern batch data processing challenges.