The Proposal of Double Agent Architecture using Actor-critic Algorithm for Penetration Testing

Hoang Nguyen, Songpon Teerakanok, Atsuo Inomata, Tetsutaro Uehara

2021

Abstract

Reinforcement learning (RL) is a widely used machine learning method for optimal decision-making compared to rule-based methods. Because of that advantage, RL has also recently been used a lot in penetration testing (PT) problems to assist in planning and deploying cyber attacks. Although the complexity and size of networks keep increasing vastly every day, RL is currently applied only for small scale networks. This paper proposes a double agent architecture (DAA) approach that is able to drastically increase the size of the network which can be solved with RL. This work also examines the effectiveness of using current popular deep reinforcement learning algorithms including DQN, DDQN, Dueling DQN and D3QN algorithms for PT. The A2C algorithm using Wolpertinger architecture is also adopted as a baseline for comparing the results of the methods. All algorithms are evaluated using a proposed network simulator which is constructed as a Markov decision process (MDP). Our results demonstrate that DAA with A2C algorithm far outweighs other approaches when dealing with large network environments reaching up to 1000 hosts.

Paper Citation

in Harvard Style

Nguyen H., Teerakanok S., Inomata A. and Uehara T. (2021). The Proposal of Double Agent Architecture using Actor-critic Algorithm for Penetration Testing.In Proceedings of the 7th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, ISBN 978-989-758-491-6, pages 440-449. DOI: 10.5220/0010232504400449

in Bibtex Style

@conference{icissp21,
author={Hoang Nguyen and Songpon Teerakanok and Atsuo Inomata and Tetsutaro Uehara},
title={The Proposal of Double Agent Architecture using Actor-critic Algorithm for Penetration Testing},
booktitle={Proceedings of the 7th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP,},
year={2021},
pages={440-449},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010232504400449},
isbn={978-989-758-491-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 7th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP,
TI - The Proposal of Double Agent Architecture using Actor-critic Algorithm for Penetration Testing
SN - 978-989-758-491-6
AU - Nguyen H.
AU - Teerakanok S.
AU - Inomata A.
AU - Uehara T.
PY - 2021
SP - 440
EP - 449
DO - 10.5220/0010232504400449