Authors:
Klev Diamanti
1
;
Andreas Kanavos
2
;
Christos Makris
2
and
Thodoris Tokis
2
Affiliations:
1
Uppsala University, Sweden
;
2
University of Patras, Greece
Keyword(s):
Searching and Browsing, Web Information Filtering and Retrieval, Text Mining, Indexing Structures, Inverted Files, n-gram Indexing, Sequence Analysis and Assembly, Weighted Sequences, Weighted Suffix Trees.
Abstract:
In this paper, we address the problem of handling weighted sequences. This is by taking advantage of the inverted files machinery and targeting text processing applications, where the involved documents cannot be separated into words (such as texts representing biological sequences) or word separation is difficult and involves extra linguistic knowledge (texts in Asian languages). Besides providing a handling of weighted sequences using n-grams, we also provide a study of constructing space efficient n-gram inverted indexes. The proposed techniques combine classic straightforward n-gram indexing, with the recently proposed two-level n-gram inverted file technique. The final outcomes are new data structures for n-gram indexing, which perform better in terms of space consumption than the existing ones. Our experimental results are encouraging and depict that these techniques can surely handle n-gram indexes more space efficiently than already existing methods.