loading
Papers Papers/2022 Papers Papers/2022

Research.Publish.Connect.

Paper

Authors: Eyyüb Sari ; Vanessa Courville and Vahid Partovi Nia

Affiliation: Huawei Noah’s Ark Lab, Canada

Keyword(s): Recurrent Neural Network, LSTM, Model Compression, Quantization, NLP, ASR.

Abstract: Recurrent neural networks (RNN) are used in many real-world text and speech applications. They include complex modules such as recurrence, exponential-based activation, gate interaction, unfoldable normalization, bi-directional dependence, and attention. The interaction between these elements prevents running them on integer-only operations without a significant performance drop. Deploying RNNs that include layer normalization and attention on integer-only arithmetic is still an open problem. We present a quantization-aware training method for obtaining a highly accurate integer-only recurrent neural network (iRNN). Our approach supports layer normalization, attention, and an adaptive piecewise linear approximation of activations (PWL), to serve a wide range of RNNs on various applications. The proposed method is proven to work on RNNbased language models and challenging automatic speech recognition, enabling AI applications on the edge. Our iRNN maintains similar performance as its full-precision counterpart, their deployment on smartphones improves the runtime performance by 2×, and reduces the model size by 4×. (More)

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 18.206.12.31

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Sari, E.; Courville, V. and Partovi Nia, V. (2022). iRNN: Integer-only Recurrent Neural Network. In Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - ICPRAM; ISBN 978-989-758-549-4; ISSN 2184-4313, SciTePress, pages 110-121. DOI: 10.5220/0010975700003122

@conference{icpram22,
author={Eyyüb Sari. and Vanessa Courville. and Vahid {Partovi Nia}.},
title={iRNN: Integer-only Recurrent Neural Network},
booktitle={Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - ICPRAM},
year={2022},
pages={110-121},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010975700003122},
isbn={978-989-758-549-4},
issn={2184-4313},
}

TY - CONF

JO - Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - ICPRAM
TI - iRNN: Integer-only Recurrent Neural Network
SN - 978-989-758-549-4
IS - 2184-4313
AU - Sari, E.
AU - Courville, V.
AU - Partovi Nia, V.
PY - 2022
SP - 110
EP - 121
DO - 10.5220/0010975700003122
PB - SciTePress