Comparative Analysis of Encoder-Only, Decoder-Only, and Encoder- Decoder Language Models

Boyu Liu

2024

Abstract

Abstract： With the surge in Artificial Intelligence (AI) popularity sparked by ChatGPT, a plethora of Transformer-based models have emerged, and the decoder-only architecture has become the mainstream development direction of large language models (LLMs) in most big-tech companies. In the rapidly advancing field of Natural Language Processing (NLP), understanding the capabilities and limitations of different language model architectures is critical for pushing the boundaries of AI. This paper delves into the comparative analysis of encoder-only, decoder-only, and encoder-decoder models, illuminating their strengths, weaknesses, and optimal use cases within the landscape of NLP. Encoder-only models are highlighted for their efficiency and deep understanding, decoder-only models for their generative capabilities and adaptability, and encoder-decoder hybrids for their versatile application across a broad spectrum of NLP tasks. This comparative analysis provides valuable insights into the strategic deployment of these models in real-world applications and underscores the ongoing need for innovation in model architecture to optimize performance and computational efficiency.

Paper Citation

in Harvard Style

Liu B. (2024). Comparative Analysis of Encoder-Only, Decoder-Only, and Encoder- Decoder Language Models. In Proceedings of the 1st International Conference on Data Science and Engineering - Volume 1: ICDSE; ISBN 978-989-758-690-3, SciTePress, pages 524-530. DOI: 10.5220/0012829800004547

in Bibtex Style

@conference{icdse24,
author={Boyu Liu},
title={Comparative Analysis of Encoder-Only, Decoder-Only, and Encoder- Decoder Language Models},
booktitle={Proceedings of the 1st International Conference on Data Science and Engineering - Volume 1: ICDSE},
year={2024},
pages={524-530},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012829800004547},
isbn={978-989-758-690-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 1st International Conference on Data Science and Engineering - Volume 1: ICDSE
TI - Comparative Analysis of Encoder-Only, Decoder-Only, and Encoder- Decoder Language Models
SN - 978-989-758-690-3
AU - Liu B.
PY - 2024
SP - 524
EP - 530
DO - 10.5220/0012829800004547
PB - SciTePress