Authors:
Lorina Sinanaj
1
;
Hossein Haeri
2
;
Liming Gao
3
;
Satya Prasad Maddipatla
3
;
Cindy Chen
1
;
Kshitij Jerath
2
;
Craig Beal
4
and
Sean Brennan
3
Affiliations:
1
Computer Science Department, University of Massachusetts Lowell, 220 Pawtucket St., Lowell, U.S.A.
;
2
Mechanical Engineering Department, University of Massachusetts Lowell, Lowell, U.S.A.
;
3
Mechanical Engineering Department, The Pennsylvania State University, University Park, U.S.A.
;
4
Mechanical Engineering Department, Bucknell University, Lewisburg, U.S.A.
Keyword(s):
Big Data, Data Reduction, Temporal Granulation, Allan Variance.
Abstract:
In the era of Big Data, conducting complex data analysis tasks efficiently, becomes increasingly important and challenging due to large amounts of data available. In order to decrease query response time with limited main memory and storage space, data reduction techniques that preserve data quality are needed. Existing data reduction techniques, however, are often computationally expensive and rely on heuristics for deciding how to split or reduce the original dataset. In this paper, we propose an effective granular data reduction technique for temporal databases, based on Allan Variance (AVAR). AVAR is used to systematically determine the temporal window length over which data remains relevant. The entire dataset to be reduced is then separated into granules with size equal to the AVAR-determined window length. Data reduction is achieved by generating aggregated information for each such granule. The proposed method is tested using a large database that contains temporal informatio
n for vehicular data. Then comparison experiments are conducted and the outstanding runtime performance is illustrated by comparing with three clustering-based data reduction methods. The performance results demonstrate that the proposed Allan Variance-based technique can efficiently generate reduced representation of the original data without losing data quality, while significantly reducing computation time.
(More)