Extracting Body Text from Academic PDF Documents for Text Mining

Changfeng Yu, Cheng Zhang, Jie Wang

2020

Abstract

Accurate extraction of body text from PDF-formatted academic documents is essential in text-mining applications for deeper semantic understandings. The objective is to extract complete sentences in the body text into a txt file with the original sentence flow and paragraph boundaries. Existing tools for extracting text from PDF documents would often mix body and nonbody texts. We devise and implement a system called PDFBoT to detect multiple-column layouts using a line-sweeping technique, remove nonbody text using computed text features and syntactic tagging in backward traversal, and align the remaining text back to sentences and paragraphs. We show that PDFBoT is highly accurate with average F1 scores of, respectively, 0.99 on extracting sentences, 0.96 on extracting paragraphs, and 0.98 on removing text on tables, figures, and charts over a corpus of PDF documents randomly selected from arXiv.org across multiple academic disciplines.

Paper Citation

in Harvard Style

Yu C., Zhang C. and Wang J. (2020). Extracting Body Text from Academic PDF Documents for Text Mining. In Proceedings of the 12th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2020) - Volume 1: KDIR; ISBN 978-989-758-474-9, SciTePress, pages 235-242. DOI: 10.5220/0010131402350242

in Bibtex Style

@conference{kdir20,
author={Changfeng Yu and Cheng Zhang and Jie Wang},
title={Extracting Body Text from Academic PDF Documents for Text Mining},
booktitle={Proceedings of the 12th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2020) - Volume 1: KDIR},
year={2020},
pages={235-242},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010131402350242},
isbn={978-989-758-474-9},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2020) - Volume 1: KDIR
TI - Extracting Body Text from Academic PDF Documents for Text Mining
SN - 978-989-758-474-9
AU - Yu C.
AU - Zhang C.
AU - Wang J.
PY - 2020
SP - 235
EP - 242
DO - 10.5220/0010131402350242
PB - SciTePress