Enhancing Workflow Efficiency Through DAG Merging and Parallel Execution in Apache Airflow

Anvith S G, Nithish Kushal Reddy, Sreebha Bhaskaran, Gurupriya M

2025

Abstract

Modern workflows are very complex, containing interdependencies and shared tasks that cause inefficiencies in the execution of tasks, utilization of resources, and dependency management. The structure of Directed Acyclic Graphs(DAGs) is very robust for modeling workflows but managing overlapping tasks across multiple DAGs results in redundancy, inconsistent dependencies, and long execution times. This work helps address the above issues with the development of a systematic way to merge multiple task-based DAGs into a single workflow that optimizes workflow execution. It verifies weak connectivity, cyclic properties, and consistency in dependencies between DAGs with algorithms such as Depth First Search(DFS)/Breadth First Search(BFS), Floyd-Warshall, Union-Find, In-Degree Similarity Check and Signature Hashing. Using the the merged DAG, Apache Airflow optimizes it for parallel execution, reducing the time taken to execute and maximizing resource usage. Each algorithm is assessed based on its time complexity, space complexity, and practical performance in order to determine the best solution for each stage. The final solution proves robust and scalable by deploying the integrated workflow in Apache Airflow, improving efficiency, removing redundancy, and optimizing the execution of tasks.

Paper Citation

in Harvard Style

S G A., Kushal Reddy N., Bhaskaran S. and M G. (2025). Enhancing Workflow Efficiency Through DAG Merging and Parallel Execution in Apache Airflow. In Proceedings of the 3rd International Conference on Futuristic Technology - Volume 2: INCOFT; ISBN 978-989-758-763-4, SciTePress, pages 309-319. DOI: 10.5220/0013591500004664

in Bibtex Style

@conference{incoft25,
author={Anvith S G and Nithish Kushal Reddy and Sreebha Bhaskaran and Gurupriya M},
title={Enhancing Workflow Efficiency Through DAG Merging and Parallel Execution in Apache Airflow},
booktitle={Proceedings of the 3rd International Conference on Futuristic Technology - Volume 2: INCOFT},
year={2025},
pages={309-319},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013591500004664},
isbn={978-989-758-763-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 3rd International Conference on Futuristic Technology - Volume 2: INCOFT
TI - Enhancing Workflow Efficiency Through DAG Merging and Parallel Execution in Apache Airflow
SN - 978-989-758-763-4
AU - S G A.
AU - Kushal Reddy N.
AU - Bhaskaran S.
AU - M G.
PY - 2025
SP - 309
EP - 319
DO - 10.5220/0013591500004664
PB - SciTePress