Approximate Query Processing for Lambda Architecture

Aleksey Burdakov, Uriy Grigorev, Andrey Ploutenko, Oleg Ermakov

2021

Abstract

The lambda architecture is widely used to implement streaming data processing systems. These systems create batch views (subsets of data) at the Serving Layer to speed up queries. This operation takes significant time. The article proposes a novel approach to lambda architecture implementation. A new method for Approximate Query Processing in a system with Lambda Architecture (LA-AQP) significantly reduces aggregate (sum, count, avg) calculation error. This is achieved by using a new way of calculating the reading segments probabilities. The developed method is compared with the modern Sapprox method for processing large distributed data. Experiments demonstrate that LA-AQP almost equals Sapprox in terms of volume and time characteristics. The introduced accuracy measures (δ-accuracy and ε-accuracy) are up to two times better than Sapprox for total aggregate calculation. Aggregate values can vary greatly from segment to segment. It is shown that in this case the LA-AQP method gives a small error in the total aggregate calculation in contrast to Sapprox.

Paper Citation

in Harvard Style

Burdakov A., Grigorev U., Ploutenko A. and Ermakov O. (2021). Approximate Query Processing for Lambda Architecture. In Proceedings of the 6th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS, ISBN 978-989-758-504-3, pages 253-261. DOI: 10.5220/0010465802530261

in Bibtex Style

@conference{iotbds21,
author={Aleksey Burdakov and Uriy Grigorev and Andrey Ploutenko and Oleg Ermakov},
title={Approximate Query Processing for Lambda Architecture},
booktitle={Proceedings of the 6th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS,},
year={2021},
pages={253-261},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010465802530261},
isbn={978-989-758-504-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 6th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS,
TI - Approximate Query Processing for Lambda Architecture
SN - 978-989-758-504-3
AU - Burdakov A.
AU - Grigorev U.
AU - Ploutenko A.
AU - Ermakov O.
PY - 2021
SP - 253
EP - 261
DO - 10.5220/0010465802530261