Batch Contrastive Regularization for Deep Neural Network

Muhammad Tanveer, Hung Tan, Hui Ng, Maylor Leung, Joon Chuah

Abstract

As neural network becomes deeper, it becomes more capable of generating more powerful representation for a wide variety of tasks. However, deep neural network has a large number of parameters and easy to overfit the training samples. In this paper, we present a new regularization technique, called batch contrastive regularization. Regularization is performed by comparing samples collectively via contrastive loss which encourages intra-class compactness and inter-class separability in an embedded Euclidean space. To facilitate learning of embedding features for contrastive loss, a two-headed neural network architecture is used to decouple regularization classification. During inference, the regularization head is discarded and the network operates like any conventional classification network. We also introduce bag sampling to ensure sufficient positive samples for the classes in each batch. The performance of the proposed architecture is evaluated on CIFAR-10 and CIFAR-100 databases. Our experiments show that features regularized by contrastive loss has strong generalization performance, yielding over 8% improvement on ResNet50 for CIFAR-100 when trained from scratch.

Download


Paper Citation