loading
Papers Papers/2020

Research.Publish.Connect.

Paper

Author: Michel Krämer

Affiliation: Fraunhofer Institute for Computer Graphics Research IGD, Fraunhoferstr. 5, 64283 Darmstadt, Germany, Technical University of Darmstadt, 64289 Darmstadt, Germany

ISBN: 978-989-758-440-4

ISSN: 2184-285X

Keyword(s): Scientific Workflow Management Systems, Cloud Computing, Distributed Systems, Task Scheduling.

Abstract: We present a distributed task scheduling algorithm and a software architecture for a system executing scientific workflows in the Cloud. The main challenges we address are (i) capability-based scheduling, which means that individual workflow tasks may require specific capabilities from highly heterogeneous compute machines in the Cloud, (ii) a dynamic environment where resources can be added and removed on demand, (iii) scalability in terms of scientific workflows consisting of hundreds of thousands of tasks, and (iv) fault tolerance because in the Cloud, faults can happen at any time. Our software architecture consists of loosely coupled components communicating with each other through an event bus and a shared database. Workflow graphs are converted to process chains that can be scheduled independently. Our scheduling algorithm collects distinct required capability sets for the process chains, asks the agents which of these sets they can manage, and then assigns process chains accor dingly. We present the results of four experiments we conducted to evaluate if our approach meets the aforementioned challenges. We finish the paper with a discussion, conclusions, and future research opportunities. An implementation of our algorithm and software architecture is publicly available with the open-source workflow management system “Steep”. (More)

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 3.235.184.215

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Krämer, M. (2020). Capability-based Scheduling of Scientific Workflows in the Cloud. In Proceedings of the 9th International Conference on Data Science, Technology and Applications - DATA, ISBN 978-989-758-440-4 ISSN 2184-285X, pages 43-54. DOI: 10.5220/0009805400430054

@conference{data20,
author={Michel Krämer.},
title={Capability-based Scheduling of Scientific Workflows in the Cloud},
booktitle={Proceedings of the 9th International Conference on Data Science, Technology and Applications - DATA,},
year={2020},
pages={43-54},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0009805400430054},
isbn={978-989-758-440-4},
issn={2184-285X},
}

TY - CONF

JO - Proceedings of the 9th International Conference on Data Science, Technology and Applications - DATA,
TI - Capability-based Scheduling of Scientific Workflows in the Cloud
SN - 978-989-758-440-4
IS - 2184-285X
AU - Krämer, M.
PY - 2020
SP - 43
EP - 54
DO - 10.5220/0009805400430054

Login or register to post comments.

Comments on this Paper: Be the first to review this paper.
0123movie.net