Target Evaluation for Neural Language Model using Japanese Case Frame

Kazuhito Tamura, Ikumi Suzuki, Kazuo Hara

Abstract

Automatic text generation are widely used in various type of natural language processing systems. It is crucial to capture correct grammar for these systems to work. According to the recent studies, neural language models successfully acquire English grammar. However, it’s not thoroughly investigated why the neural language models work. Therefore, fine-grained grammatical or syntactic analysis is important to assess neural language models. In this paper, we constructed grammatical evaluation methods to assess Japanese grammatical ability in neural language models by adopting a target evaluation approach. We especially focus on case marker and verb match in Japanese case grammar. In experiments, we report the grammatical ability of neural language model by comparing n-gram models. Neural language model performed better even some information lacks, while n-gram performs poorly. Also, Neural language model exhibited more robust performance for low frequency terms.

Download


Paper Citation