Authors:
Luis F. S. Teixeira
1
;
Gabriel P. Lopes
1
and
Rita A. Ribeiro
2
Affiliations:
1
FCT/UNL, Portugal
;
2
CA3-Uninova, Portugal
Keyword(s):
Document keywords, Document topics, Words, Multi-words, Prefixes, Automatic extraction, Suffix arrays.
Related
Ontology
Subjects/Areas/Topics:
Applications
;
Artificial Intelligence
;
Knowledge Engineering and Ontology Development
;
Knowledge-Based Systems
;
Natural Language Processing
;
Pattern Recognition
;
Symbolic Systems
Abstract:
In this paper we compare twenty language independent statistical-based metrics for key term extraction from any document collection. While some of those metrics are widely used, others were recently created. Two different document representations are considered in our experiments. One is based on words and multi-words and the other is based on word prefixes of fixed length (5 characters for the experiments made) for handling morphologically rich languages, namely Portuguese and Czech. English is also experimented, as a non-morphologically rich language. Results are manually evaluated and agreement between evaluators is assessed using k-Statistics. The metrics based on Tf-Idf and Phi-square proved to have higher precision and recall. The use of prefix-based representation of documents enabled a significant improvement for documents written in Portuguese.