Randout-KD: Finetuning Foundation Models for Text Classification via Random Noise and Knowledge Distillation

Pervaiz Khan, Pervaiz Khan, Andreas Dengel, Andreas Dengel, Sheraz Ahmed

2023

Abstract

Finetuning foundation models effectively on downstream tasks is ongoing research. In this paper, we present a finetuning method “Randout-KD” that enhances the performance of a student model for text classification. We specifically propose a noise-injecting method in the representations of the transformer model during its finetuning that works as regularization. Moreover, we integrate the knowledge distillation and noise injection methods and show that combining these approaches boosts the baseline model performance. We evaluate the proposed method on two datasets namely “CODA-19” and “RHMD” using PubMedBERT and RoBERTa Large as teacher models, and data2vec as a student model. Results show that the proposed approach improves the accuracy up to 1.2% compared to the baseline methods.

Paper Citation

in Harvard Style

Khan P., Dengel A. and Ahmed S. (2023). Randout-KD: Finetuning Foundation Models for Text Classification via Random Noise and Knowledge Distillation. In Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART, ISBN 978-989-758-623-1, pages 457-465. DOI: 10.5220/0011687800003393

in Bibtex Style

@conference{icaart23,
author={Pervaiz Khan and Andreas Dengel and Sheraz Ahmed},
title={Randout-KD: Finetuning Foundation Models for Text Classification via Random Noise and Knowledge Distillation},
booktitle={Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,},
year={2023},
pages={457-465},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011687800003393},
isbn={978-989-758-623-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,
TI - Randout-KD: Finetuning Foundation Models for Text Classification via Random Noise and Knowledge Distillation
SN - 978-989-758-623-1
AU - Khan P.
AU - Dengel A.
AU - Ahmed S.
PY - 2023
SP - 457
EP - 465
DO - 10.5220/0011687800003393