Authors:
Mina Basirat
1
and
Peter M. Roth
2
;
1
Affiliations:
1
Institute of Computer Graphics and Vision, Graz University of Technology, Austria
;
2
Data Science in Earth Observation, Technical University of Munich, Germany
Keyword(s):
Visual Categorization, Activation Functions, Particle Swarm Optimization.
Abstract:
Recently, it has been shown that properly parametrized Leaky ReLU (LReLU) as an activation function yields significantly better results for a variety of image classification tasks. However, such methods are not feasible in practice. Either the only parameter (i.e., the slope of the negative part) needs to be set manually (L*ReLU), or the approach is vulnerable due to the gradient-based optimization and, thus, highly dependent on a proper initialization (PReLU). In this paper, we would like to exploit the benefits of piecewise linear functions, avoiding these problems. To this end, we propose a fully automatic approach to estimate the slope parameter for LReLU from the data. We realize this via Stochastic Optimization, namely Particle Swarm Optimization (PSO): S*ReLU. In this way, we can show that, compared to widely-used activation functions (including PReLU), better results can be obtained on seven different benchmark datasets. Moreover, the results even match those of L*ReLU, where
the optimal parameter is estimated in a brute-force manner. In this way, our fully-automatic approach allows for drastically reducing the computational effort.
(More)