Conic Linear Units: Improved Model Fusion and Rotational-Symmetric Generative Model

Changqing Fu, Laurent Cohen

2024

Abstract

We introduce Conic Linear Unit (CoLU), a natural generalization of commonly used activation functions in neural networks. The common pointwise ReLU activation is a projection onto the positive cone and is permutation symmetric. We propose a nonlinearity that goes beyond this symmetry: CoLU is a skew projection onto a hypercone towards the cone’s axis. Due to the nature of this projection, CoLU enforces symmetry in a neural network with width C from the finite-order permutation group S(C) to the infinite-order rotation/reflection group O(C− 1), thus producing deep features that are motivated by the HSV color representation. Recent results on merging independent neural networks via permutation modulus can be relaxed and generalized to soft alignment modulo an optimal transport plan (Singh and Jaggi, 2020), which is useful in aligning models of different widths. CoLU aims to further alleviate the apparent deficiency of soft alignment. Our simulation indicates that CoLU outperforms existing generative models including Autoencoder and Latent Diffusion Model on small or large-scale image datasets. Additionally, CoLU does not increase the number of parameters and requires negligible additional computation overhead. The CoLU concept is quite general and can be plugged into various neural network architectures. Ablation studies on extensions to soft projections, general L p cones, and the non-convex double-cone cases are briefly discussed.

Paper Citation

in Harvard Style

Fu C. and Cohen L. (2024). Conic Linear Units: Improved Model Fusion and Rotational-Symmetric Generative Model. In Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP; ISBN 978-989-758-679-8, SciTePress, pages 686-693. DOI: 10.5220/0012406500003660

in Bibtex Style

@conference{visapp24,
author={Changqing Fu and Laurent Cohen},
title={Conic Linear Units: Improved Model Fusion and Rotational-Symmetric Generative Model},
booktitle={Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP},
year={2024},
pages={686-693},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012406500003660},
isbn={978-989-758-679-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP
TI - Conic Linear Units: Improved Model Fusion and Rotational-Symmetric Generative Model
SN - 978-989-758-679-8
AU - Fu C.
AU - Cohen L.
PY - 2024
SP - 686
EP - 693
DO - 10.5220/0012406500003660
PB - SciTePress