Classification of Complaints Text Data by Ensembling Large Language Models

Pruthweesha Airani, Neha Pipada, Pratik Shah

2025

Abstract

Effective and efficient management of consumer complaints requires segregation of complaints based on products, services, etc. categories. In this work, we propose an ensemble classification approach based on statistical class incidence frequencies from softmax confidence scores of ensemble of classifiers. The classifiers process the complaint text through Large Language Models (LLMs) followed by discriminating networks. LLMs along with discriminators are fine-tuned on a large, publicly available dataset of over 162,000 annotated consumer complaint records pertaining to banking services. The proposed ensemble approach utilizes confidence scores from individual classifiers (LLM embeddings + discriminator network) achieving better accuracy. It is based on statistical analysis of class-wise precision as a function of confidence score. The individual classifiers built on various SMLMs & LLMs are experimented with, and the results are tabulated for the complaints classification task.

Paper Citation

in Harvard Style

Airani P., Pipada N. and Shah P. (2025). Classification of Complaints Text Data by Ensembling Large Language Models. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 679-686. DOI: 10.5220/0013173900003890

in Bibtex Style

@conference{icaart25,
author={Pruthweesha Airani and Neha Pipada and Pratik Shah},
title={Classification of Complaints Text Data by Ensembling Large Language Models},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2025},
pages={679-686},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013173900003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - Classification of Complaints Text Data by Ensembling Large Language Models
SN - 978-989-758-737-5
AU - Airani P.
AU - Pipada N.
AU - Shah P.
PY - 2025
SP - 679
EP - 686
DO - 10.5220/0013173900003890
PB - SciTePress