Authors:
Gabriel Ioan Arcas
1
and
Tudor Cioara
2
Affiliations:
1
Engineering and Data Solutions Department, Bosch Engineering, Center, Cluj-Napoca, Romania
;
2
Computer Science Department, Technical University of Cluj Napoca, Romania
Keyword(s):
Data Orchestration, AI Workflow, Computing Continuum, Lambda Architecture, Edge-Fog-Cloud.
Abstract:
Cloud AI technologies have emerged to exploit the vast amount of data produced by digitized activities. However, despite these advancements, they still face challenges in several areas, including data processing, achieving fast response times, and reducing latency. This paper proposes a data orchestration platform for AI workflows, considering the computing continuum setup. The edge layer of the platform focuses on immediate data collection, the fog layer provides intermediate processing, and the cloud layer manages long-term storage and complex data analysis. The orchestration platform incorporates the Lambda Architecture principles for flexibility in managing batch processing and real-time data streams, enabling effective management of large data volumes for AI workflows. The platform was used to manage an AI workflow dealing with the prediction of household energy consumption, showcasing how each layer supports different stages of the machine learning pipeline. The results are pro
mising the models are being trained, validated, and deployed effectively, with reduced latency and use of computational resources.
(More)