Exploring Alternatives to Softmax Function

Kunal Banerjee, Vishak C., Rishi Gupta, Kartik Vyas, Anushree H., Biswajit Mishra

2021

Abstract

Softmax function is widely used in artificial neural networks for multiclass classification, multilabel classification, attention mechanisms, etc. However, its efficacy is often questioned in literature. The log-softmax loss has been shown to belong to a more generic class of loss functions, called spherical family, and its member log-Taylor softmax loss is arguably the best alternative in this class. In another approach which tries to enhance the discriminative nature of the softmax function, soft-margin softmax (SM-softmax) has been proposed to be the most suitable alternative. In this work, we investigate Taylor softmax, SM-softmax and our proposed SM-Taylor softmax, an amalgamation of the earlier two functions, as alternatives to softmax function. Furthermore, we explore the effect of expanding Taylor softmax up to ten terms (original work proposed expanding only to two terms) along with the ramifications of considering Taylor softmax to be a finite or infinite series during backpropagation. Our experiments for the image classification task on different datasets reveal that there is always a configuration of the SM-Taylor softmax function that outperforms the normal softmax function and its other alternatives.

Paper Citation

in Harvard Style

Banerjee K., C. V., Gupta R., Vyas K., H. A. and Mishra B. (2021). Exploring Alternatives to Softmax Function. In Proceedings of the 2nd International Conference on Deep Learning Theory and Applications - Volume 1: DeLTA, ISBN 978-989-758-526-5, pages 81-86. DOI: 10.5220/0010502000810086

in Bibtex Style

@conference{delta21,
author={Kunal Banerjee and Vishak C. and Rishi Gupta and Kartik Vyas and Anushree H. and Biswajit Mishra},
title={Exploring Alternatives to Softmax Function},
booktitle={Proceedings of the 2nd International Conference on Deep Learning Theory and Applications - Volume 1: DeLTA,},
year={2021},
pages={81-86},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010502000810086},
isbn={978-989-758-526-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 2nd International Conference on Deep Learning Theory and Applications - Volume 1: DeLTA,
TI - Exploring Alternatives to Softmax Function
SN - 978-989-758-526-5
AU - Banerjee K.
AU - C. V.
AU - Gupta R.
AU - Vyas K.
AU - H. A.
AU - Mishra B.
PY - 2021
SP - 81
EP - 86
DO - 10.5220/0010502000810086