Dynamically Modular and Sparse General Continual Learning

Arnav Varma, Elahe Arani, Elahe Arani, Bahram Zonooz, Bahram Zonooz

2023

Abstract

Real-world applications often require learning continuously from a stream of data under ever-changing conditions. When trying to learn from such non-stationary data, deep neural networks (DNNs) undergo catastrophic forgetting of previously learned information. Among the common approaches to avoid catastrophic forgetting, rehearsal-based methods have proven effective. However, they are still prone to forgetting due to task-interference as all parameters respond to all tasks. To counter this, we take inspiration from sparse coding in the brain and introduce dynamic modularity and sparsity (Dynamos) for rehearsal-based general continual learning. In this setup, the DNN learns to respond to stimuli by activating relevant subsets of neurons. We demonstrate the effectiveness of Dynamos on multiple datasets under challenging continual learning evaluation protocols. Finally, we show that our method learns representations that are modular and specialized, while maintaining reusability by activating subsets of neurons with overlaps corresponding to the similarity of stimuli. The code is available at https://github.com/NeurAI-Lab/DynamicContinualLearning.

Paper Citation

in Harvard Style

Varma A., Arani E. and Zonooz B. (2023). Dynamically Modular and Sparse General Continual Learning. In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 5: VISAPP; ISBN 978-989-758-634-7, SciTePress, pages 262-273. DOI: 10.5220/0011790200003417

in Bibtex Style

@conference{visapp23,
author={Arnav Varma and Elahe Arani and Bahram Zonooz},
title={Dynamically Modular and Sparse General Continual Learning},
booktitle={Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 5: VISAPP},
year={2023},
pages={262-273},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011790200003417},
isbn={978-989-758-634-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 5: VISAPP
TI - Dynamically Modular and Sparse General Continual Learning
SN - 978-989-758-634-7
AU - Varma A.
AU - Arani E.
AU - Zonooz B.
PY - 2023
SP - 262
EP - 273
DO - 10.5220/0011790200003417
PB - SciTePress