loading
Papers Papers/2022

Research.Publish.Connect.

Paper

Authors: Rafal Lokuciejewski 1 ; Dominik Schüssele 1 ; Florian Wilhelm 1 and Sven Groppe 2

Affiliations: 1 inovex GmbH, Germany ; 2 Institute of Information Systems (IFIS), University of Lübeck, Germany

Keyword(s): Jupyter, Spark, Kubernetes, YARN, Cluster, UEQ, Notebooks.

Abstract: Various cloud providers offer integrated platforms for interactive development in notebooks for processing and analysis of Big Data on large compute clusters. Such platforms enable users to easily leverage frameworks like Apache Spark as well as to manage cluster resources. However, Data Scientists and Engineers are facing the lack of a similar holistic solution when working with on-premises infrastructure. Especially a central point of administration to access a notebooks’ UI, manage notebook kernels, allocate resources for frameworks like Apache Spark or monitor cluster workloads, in general, is currently missing for on-premises infrastructure. To overcome these issues and provide on-premises users with a platform for interactive development, we propose a cross-cluster architecture resulting from an extensive requirements engineering process. Based on open-source components, the designed platform provides an intuitive Web-UI that enables users to easily access notebooks, manage cus tom kernel-environments as well as monitor cluster resources and current workloads. Besides an admin panel for user restrictions, the platform provides isolation of user workloads and scalability by design. The designed platform is evaluated against prior solutions for on-premises as well as from a user perspective by utilizing the User Experience Questionnaire, an independent benchmark tool for interactive products. (More)

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 18.204.56.185

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Lokuciejewski, R.; Schüssele, D.; Wilhelm, F. and Groppe, S. (2021). A Platform for Interactive Data Science with Apache Spark for On-premises Infrastructure. In Proceedings of the 11th International Conference on Cloud Computing and Services Science - CLOSER, ISBN 978-989-758-510-4; ISSN 2184-5042, pages 65-76. DOI: 10.5220/0010447500650076

@conference{closer21,
author={Rafal Lokuciejewski. and Dominik Schüssele. and Florian Wilhelm. and Sven Groppe.},
title={A Platform for Interactive Data Science with Apache Spark for On-premises Infrastructure},
booktitle={Proceedings of the 11th International Conference on Cloud Computing and Services Science - CLOSER,},
year={2021},
pages={65-76},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010447500650076},
isbn={978-989-758-510-4},
issn={2184-5042},
}

TY - CONF

JO - Proceedings of the 11th International Conference on Cloud Computing and Services Science - CLOSER,
TI - A Platform for Interactive Data Science with Apache Spark for On-premises Infrastructure
SN - 978-989-758-510-4
IS - 2184-5042
AU - Lokuciejewski, R.
AU - Schüssele, D.
AU - Wilhelm, F.
AU - Groppe, S.
PY - 2021
SP - 65
EP - 76
DO - 10.5220/0010447500650076