Informer, an Information Organization Transformer Architecture

Cristian Ojeda, Cayetano Artal, Francisco Tejera

2021

Abstract

The use of architectures based on transformers presents a state of the art revolution in natural language processing (NLP). The employment of these architectures with high computational costs has increased in the last few months, despite the existing use of parallelization techniques. This is due to the high performance that is obtained by increasing the size of the learnable parameters for these kinds of architectures, while maintaining the models’ predictability. This relates to the fact that it is difficult to do research with limited computational resources. A restrictive element is the memory usage, which seriously affects the replication of experiments. We are presenting a new architecture called Informer, which seeks to exploit the concept of information organization. For the sake of evaluation, we use a neural machine translation (NMT) dataset, the English-Vietnamese IWSLT15 dataset (Luong and Manning, 2015). In this paper, we also compare this proposal with architectures that reduce the computational cost to O(n · r), such as Linformer (Wang et al., 2020). In addition, we have managed to improve the SOTA of the BLEU score from 33.27 to 35.11.

Paper Citation

in Harvard Style

Ojeda C., Artal C. and Tejera F. (2021). Informer, an Information Organization Transformer Architecture.In Proceedings of the 13th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-484-8, pages 381-389. DOI: 10.5220/0010372703810389

in Bibtex Style

@conference{icaart21,
author={Cristian Ojeda and Cayetano Artal and Francisco Tejera},
title={Informer, an Information Organization Transformer Architecture},
booktitle={Proceedings of the 13th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2021},
pages={381-389},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010372703810389},
isbn={978-989-758-484-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Informer, an Information Organization Transformer Architecture
SN - 978-989-758-484-8
AU - Ojeda C.
AU - Artal C.
AU - Tejera F.
PY - 2021
SP - 381
EP - 389
DO - 10.5220/0010372703810389