Authors:
Mayank Patel
1
and
Minal Bhise
2
Affiliations:
1
Department of Information & Communication Technology, Adani University, Shantigram, Gujarat, India
;
2
Distributed Databases Group, Dhirubhai Ambani Institute of Information and Communication Technology, India
Keyword(s):
Big Data Partitioning, Optimize Resource Utilization, Raw Data Query Processing, Real-Time Dynamic Resource Allocation, Task Scheduling.
Abstract:
Scientific experiments and contemporary applications generate substantial volumes of data daily, posing a challenge for traditional database management systems (DBMS) that expend considerable time and resources on data loading. In-situ engines offer a distinct advantage by enabling immediate querying on raw data. Re-searchers have observed that resources are often underutilized during data loading. In contrast, in-situ engines spend ample time and resources in reparsing required data multiple times. Allocating query specific resources is another challenging task that must be addressed to reduce overall workload execution time and resource utilization. This research paper introduces a novel approach called the Resource Availability & Workload-aware Hybrid Framework (RAW-HF), designed to enhance the efficiency of data querying by judiciously utilizing optimal resources in systems comprising an in-situ engine and DBMS. RAW-HF incorporates modules that facilitate the optimization of reso
urces necessary for executing a given workload, striving to maximize the utilization of available resources. The effectiveness of RAW-HF is demonstrated using the scientific dataset Sloan Digital Sky Survey (SDSS) and Linked Observation data (LOD). Comparative analysis with the state-of-the-art workload-aware partial loading technique (WA) reveals that RAW-HF excels in allocating query-specific resources and implementing resource-aware task scheduling. Results from the study indicate that RAW-HF outperforms WA, reducing workload execution time by 26%. It also reduces CPU and IO resource utilization by 26% and 25% compared to WA at a cost of 33% additional RAM.
(More)