Efficient Use of Large Language Models for Analysis of Text Corpora

David Adamczyk, Jan Hůla, Jan Hůla

2024

Abstract

In this paper, we propose an efficient approach for tracking a given phenomenon in a corpus using natural language processing (NLP) methods. The topic of tracking phenomena in a corpus is important, especially in the fields of sociology, psychology, and economics, which study human behavior in society. Unlike existing approaches that rely on universal large language models (LLMs), which are computationally expensive, we focus on using computationally less expensive methods. These methods allow for high data processing speed while maintaining high accuracy. Our approach is inspired by the cascade approach to optimization, where we first roughly filter out unwanted information and then gradually use more accurate models, which are computationally more expensive. In this way, we are able to process large amounts of data with high accuracy using different models, while also reducing the overall cost of computations. To demonstrate the proposed method, we chose a task that consists of finding the frequency of occurrence of a certain phenomenon in a large text corpus, which is divided into individual months of the year. In practice, this means that we can, for example, use Internet discussions to find out how much people are discussing a particular topic. The entire solution is presented as a pipeline, which consists of individual phases that successively process text data using methods selected to minimize the overall cost of processing all data.

Paper Citation

in Harvard Style

Adamczyk D. and Hůla J. (2024). Efficient Use of Large Language Models for Analysis of Text Corpora. In Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM; ISBN 978-989-758-684-2, SciTePress, pages 695-705. DOI: 10.5220/0012349800003654

in Bibtex Style

@conference{icpram24,
author={David Adamczyk and Jan Hůla},
title={Efficient Use of Large Language Models for Analysis of Text Corpora},
booktitle={Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM},
year={2024},
pages={695-705},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012349800003654},
isbn={978-989-758-684-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM
TI - Efficient Use of Large Language Models for Analysis of Text Corpora
SN - 978-989-758-684-2
AU - Adamczyk D.
AU - Hůla J.
PY - 2024
SP - 695
EP - 705
DO - 10.5220/0012349800003654
PB - SciTePress