CNNs Sparsification and Expansion for Continual Learning

Basile Tousside, Jörg Frochte, Tobias Meisen

2024

Abstract

Learning multiple sequentially arriving tasks without forgetting previous knowledge, known as Continual Learning (CL), remains a long-standing challenge for neural networks. Most existing CL methods rely on data replay. However, they are not applicable when past data is unavailable or is not allowed to be synthetically generated. To address this challenge, we propose Sparification and Expansion-based Continual Learning (SECL). SECL avoids forgetting of previous tasks by ensuring the stability of the CNN via a stability regularization term, which prevents filters detected as important for past tasks to deviate too much when learning a new task. On top of that, SECL makes the network plastic via a plasticity regularization term that leverage the over-parameterization of CNNs to efficiently sparsify the network and tunes unimportant filters making them relevant for future tasks. Also, SECL enhances the plasticity of the network through a simple but effective heuristic mechanism that automatically decides when and where (at which layers) to expand the network. Experiments on popular CL vision benchmarks show that SECL leads to significant improvements over state-of-the-art method in terms of overall CL performance, as measured by classification accuracy as well as in terms of avoiding catastrophic forgetting.

Paper Citation

in Harvard Style

Tousside B., Frochte J. and Meisen T. (2024). CNNs Sparsification and Expansion for Continual Learning. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 110-120. DOI: 10.5220/0012314000003636

in Bibtex Style

@conference{icaart24,
author={Basile Tousside and Jörg Frochte and Tobias Meisen},
title={CNNs Sparsification and Expansion for Continual Learning},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2024},
pages={110-120},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012314000003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - CNNs Sparsification and Expansion for Continual Learning
SN - 978-989-758-680-4
AU - Tousside B.
AU - Frochte J.
AU - Meisen T.
PY - 2024
SP - 110
EP - 120
DO - 10.5220/0012314000003636
PB - SciTePress