Multi-Objective Policy Optimization for Effective and Cost-Conscious Penetration Testing

Xiaojuan Cai, Lulu Zhu, Zhuo Li, Hiroshi Koide

2025

Abstract

Penetration testing, which identifies security vulnerabilities before malicious actors can exploit them, is essential for strengthening cybersecurity defenses. Effective testing helps discover deep, high-impact vulnerabilities across complex networks, while efficient testing ensures fast execution, low resource utilization, and reduced risk of detection in constrained or sensitive environments. However, achieving both effectiveness and efficiency in real-world network environments presents a core challenge: deeper compromises often require more actions and time. At the same time, excessively conservative strategies may miss critical vulnerabilities. This work addresses the trade-off between maximizing attack performance and minimizing operational costs. We propose a multi-objective reinforcement learning framework that minimizes costs while maximizing rewards. Our approach introduces a Lagrangian-based policy optimization scheme in which a dynamically adjusted multiplier balances the relative importance of rewards and costs during learning. We evaluate our method on benchmark environments with varied network topologies and service configurations. Experimental results demonstrate that our method achieves successful penetration performance and significantly reduces time costs compared to the baselines, thereby improving the adaptability and practicality of automated penetration testing in real-world scenarios.

Paper Citation

in Harvard Style

Cai X., Zhu L., Li Z. and Koide H. (2025). Multi-Objective Policy Optimization for Effective and Cost-Conscious Penetration Testing. In Proceedings of the 21st International Conference on Web Information Systems and Technologies - Volume 1: WEBIST; ISBN 978-989-758-772-6, SciTePress, pages 374-385. DOI: 10.5220/0013713400003985

in Bibtex Style

@conference{webist25,
author={Xiaojuan Cai and Lulu Zhu and Zhuo Li and Hiroshi Koide},
title={Multi-Objective Policy Optimization for Effective and Cost-Conscious Penetration Testing},
booktitle={Proceedings of the 21st International Conference on Web Information Systems and Technologies - Volume 1: WEBIST},
year={2025},
pages={374-385},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013713400003985},
isbn={978-989-758-772-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 21st International Conference on Web Information Systems and Technologies - Volume 1: WEBIST
TI - Multi-Objective Policy Optimization for Effective and Cost-Conscious Penetration Testing
SN - 978-989-758-772-6
AU - Cai X.
AU - Zhu L.
AU - Li Z.
AU - Koide H.
PY - 2025
SP - 374
EP - 385
DO - 10.5220/0013713400003985
PB - SciTePress