Authors:
Vojtěch Kovář
;
Miloš Jakubíček
and
Aleš Horák
Affiliation:
Masaryk University, Czech Republic
Keyword(s):
Natural Language Processing, Applications, Evaluation.
Related
Ontology
Subjects/Areas/Topics:
Applications
;
Artificial Intelligence
;
Industrial Applications of AI
;
Knowledge Engineering and Ontology Development
;
Knowledge-Based Systems
;
Natural Language Processing
;
Pattern Recognition
;
Soft Computing
;
Symbolic Systems
Abstract:
The paper discusses problems in state of the art evaluation methods used in natural language processing (NLP). Usually, some form of gold standard data is used for evaluation of various NLP tasks, ranging from morphological annotation to semantic analysis. We discuss problems and validity of this type of evaluation, for various tasks, and illustrate the problems on examples. Then we propose using application-driven evaluations, wherever it is possible. Although it is more expensive, more complicated and not so precise, it is the only way to find out if a particular tool is useful at all.