IDAT: An Interactive Data Exploration Tool

Nir Regev, Asaf Shabtai, Lior Rokach

2025

Abstract

In the current landscape of data analytics, data scientists predominantly utilize in-memory processing tools such as Python’s pandas or big data frameworks like Spark to conduct exploratory data analysis (EDA). These methods, while powerful, often entail substantial trade-offs, including significant consumption of time, memory, and storage, alongside elevated data scanning costs. Considering these limitations, we developed iDAT, a cost-effective interactive data exploration method. Our method uses a deep neural network (NN) to learn the relationship between queries and their results to provide a rapid inference layer for the prediction of query results. To validate the method, we let 20 data scientists run EDA (exploratory data analysis) queries using the system underlying this method. We show that it reduces the need to scan data during inference (query calculation). We evaluated this method using 12 datasets and compared it to the latest query approximation engines (VerdictDB, BlinkDB) in terms of query latency, model weight, and accuracy. Our results indicate that the iDat predicted query results with a WMAPE (weighted mean absolute percentage error) ranging from approximately 1% to 4%, which, for most of our datasets, was better than the results of the compared benchmarks.

Paper Citation

in Harvard Style

Regev N., Shabtai A. and Rokach L. (2025). IDAT: An Interactive Data Exploration Tool. In Proceedings of the 14th International Conference on Data Science, Technology and Applications - Volume 1: DATA; ISBN 978-989-758-758-0, SciTePress, pages 603-613. DOI: 10.5220/0013597800003967

in Bibtex Style

@conference{data25,
author={Nir Regev and Asaf Shabtai and Lior Rokach},
title={IDAT: An Interactive Data Exploration Tool},
booktitle={Proceedings of the 14th International Conference on Data Science, Technology and Applications - Volume 1: DATA},
year={2025},
pages={603-613},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013597800003967},
isbn={978-989-758-758-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Data Science, Technology and Applications - Volume 1: DATA
TI - IDAT: An Interactive Data Exploration Tool
SN - 978-989-758-758-0
AU - Regev N.
AU - Shabtai A.
AU - Rokach L.
PY - 2025
SP - 603
EP - 613
DO - 10.5220/0013597800003967
PB - SciTePress