Heimatkunde: Dataset for Multi-Modal Historical Document Analysis

Josef Baloun, Josef Baloun, Václav Honzík, Ladislav Lenc, Ladislav Lenc, Jiří Martínek, Jiří Martínek, Pavel Král, Pavel Král

2024

Abstract

This paper introduces a novel Heimatkunde dat aset comprising printed documents in German, specifically designed for evaluating layout analysis methods with a focus on multi-modality. The dataset is openly accessible for research purposes. The study further presents baseline results for instance segmentation and multi-modal element classification. Three advanced models, Mask R-CNN, YOLOv8, and LayoutLMv3, are employed for instance segmentation, while a fusion-based model integrating BERT and various vision Transformers are proposed for multi-modal classification. Experimental findings reveal that optimal bounding box segmentation is achieved with YOLOv8 using an input image size of 1280 pixels, and the best segmentation mask is produced by LayoutLMv3 with PubLayNet weights. Moreover, the research demonstrates superior multi-modal classification results using BERT for textual and Vision Transformer for image modalities. The study concludes by suggesting the integration of the proposed models into the historical Porta fontium portal to enhance the information retrieval from historical data.

Paper Citation

in Harvard Style

Baloun J., Honzík V., Lenc L., Martínek J. and Král P. (2024). Heimatkunde: Dataset for Multi-Modal Historical Document Analysis. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 995-1001. DOI: 10.5220/0012428500003636

in Bibtex Style

@conference{icaart24,
author={Josef Baloun and Václav Honzík and Ladislav Lenc and Jiří Martínek and Pavel Král},
title={Heimatkunde: Dataset for Multi-Modal Historical Document Analysis},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2024},
pages={995-1001},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012428500003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - Heimatkunde: Dataset for Multi-Modal Historical Document Analysis
SN - 978-989-758-680-4
AU - Baloun J.
AU - Honzík V.
AU - Lenc L.
AU - Martínek J.
AU - Král P.
PY - 2024
SP - 995
EP - 1001
DO - 10.5220/0012428500003636
PB - SciTePress