Authors:
Josef Baloun
1
;
2
;
Václav Honzík
1
;
Ladislav Lenc
1
;
2
;
Jiří Martínek
1
;
2
and
Pavel Král
1
;
2
Affiliations:
1
Department of Computer Science and Engineering, University of West Bohemia, Univerzitní, Pilsen, Czech Republic
;
2
NTIS - New Technologies for the Information Society, University of West Bohemia, Univerzitní, Pilsen, Czech Republic
Keyword(s):
BERT, Deep Learning, Layout Analysis, Multi-Modality, Transformer.
Abstract:
This paper introduces a novel Heimatkunde dat aset comprising printed documents in German, specifically designed for evaluating layout analysis methods with a focus on multi-modality. The dataset is openly accessible for research purposes. The study further presents baseline results for instance segmentation and multi-modal element classification. Three advanced models, Mask R-CNN, YOLOv8, and LayoutLMv3, are employed for instance segmentation, while a fusion-based model integrating BERT and various vision Transformers are proposed for multi-modal classification. Experimental findings reveal that optimal bounding box segmentation is achieved with YOLOv8 using an input image size of 1280 pixels, and the best segmentation mask is produced by LayoutLMv3 with PubLayNet weights. Moreover, the research demonstrates superior multi-modal classification results using BERT for textual and Vision Transformer for image modalities. The study concludes by suggesting the integration of the proposed
models into the historical Porta fontium portal to enhance the information retrieval from historical data.
(More)