On the Path to Buffer Overflow Detection by Model Checking the Stack of Binary Programs

Luís Ferreirinha, Ibéria Medeiros

2024

Abstract

The C programming language, prevalent in Cyber-Physical Systems, is crucial for system control where reliability is critical. However, it is notably susceptible to vulnerabilities, particularly buffer overflows that are ranked among the most dangerous due to their potential for catastrophic consequences. Traditional techniques, such as static analysis, often struggle with scalability and precision when detecting these vulnerabilities in the binary code of compiled C programs. This paper introduces a novel approach designed to overcome these limitations by leveraging model checking techniques to verify security properties within a program’s stack memory. To verify these properties, we propose the construction of a state space of the stack memory from a binary program’s control flow graph. Security properties, modelled for stack buffer overflow vulnerabilities and defined in Linear Temporal Logic, are verified against this state space. When violations are detected, counter-example traces are generated to undergo a reverse-flow analysis process to identify specific instances of stack buffer overflow vulnerabilities. This research aims to provide a scalable and precise approach to vulnerability detection in C binaries.

Paper Citation

in Harvard Style

Ferreirinha L. and Medeiros I. (2024). On the Path to Buffer Overflow Detection by Model Checking the Stack of Binary Programs. In Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE; ISBN 978-989-758-696-5, SciTePress, pages 719-726. DOI: 10.5220/0012732700003687

in Bibtex Style

@conference{enase24,
author={Luís Ferreirinha and Ibéria Medeiros},
title={On the Path to Buffer Overflow Detection by Model Checking the Stack of Binary Programs},
booktitle={Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE},
year={2024},
pages={719-726},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012732700003687},
isbn={978-989-758-696-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE
TI - On the Path to Buffer Overflow Detection by Model Checking the Stack of Binary Programs
SN - 978-989-758-696-5
AU - Ferreirinha L.
AU - Medeiros I.
PY - 2024
SP - 719
EP - 726
DO - 10.5220/0012732700003687
PB - SciTePress