Applying reinforcement learning in automated penetration testing


  • Nguyen Viet Hung
  • Nguyen Thanh Cong



Penetration testing, reinforcement learning, information secutity

Tóm tắt

Abstract— Facing increasingly diverse and frequent information security threats today, penetration testing is a security assessment method for information systems that organizations prioritize. Pentesters usually perform penetration testing manually and can detect critical bugs and information security issues. However, this method requires much work and requires pentesters to have high levels of practical experience and qualifications. One of the current research directions that has been interested recently is methods to support automated penetration testing. Several research groups have used attack graph analysis techniques and reinforcement learning algorithms worldwide to make automated pentesting tools. This paper proposes a model based on a reinforcement learning algorithm and parameter optimization method for this model in automated pentesting problems. To evaluate the proposed model, we utilize the data set based on the method used by other research groups. We also assess the self-built dataset on real environments with vulnerabilities. The experimental results show that the proposed method gives better assessments than other methods.


Download data is not yet available.


X. Y. B. T. B. C. M. J. Aileen G. Bacudio, “An Overview of Penetration Testing,” International Journal of Network Security And Its Applications (IJNSA), pp. Vol.3, No.6, November 2011.

M. E. Farmeena Khan, “A Comparative Study of White Box, Black Box and Grey Box Testing Techniques,” in International Journal of Advanced Computer Science and Applications, June 2012.

C. B. S. a. S. S. Vivek Shandilya, “Use of Attack Graphs in Security Systems,” Journal of Computer Networks and Communications, 2014.

R. L. a. K. Ingols, “An Annotated Review of Past Papers on Attack Graphs,” in Lincoln Laboratory MASSACHUSETTS INSTITUTE OF TECHNOLOGY, 2005.

L. P. S. Cynthia Phillips, “A Graph-Based System for Network Vulnerability Analysis,” in Proceedings of the 1998 workshop on New security paradigms, 1998.

J. H. S. J. e. a. O. Sheyner, “Automated generation and analysis of attack graphs,” in (IJACSA) Proceedings of the IEEE Symposium on Security and Privacy, Berkeley, CA USA, May 2022.

N. M. Y. Z. a. H. T. Mehdi Yousefi, “A Reinforcement Learning Approach for Attack Graph Analysis,” in IEEE International Conference on Trust Security and Privacy in Computing and Communications (TrustCom), 2018.

P. Đ. K. N. Đ. V. N. V. H. Nguyễn Mạnh Thiên, “Phát triển Framework ứng dụng AI hỗ trợ tự động khai thác lỗ hổng bảo mật,” Journal of Science and Technology on Information Security, vol. 1, no. 13, pp. 80-92, 2022.

“OpenVas,” [Online]. Available: [Accessed 20 04 2022].

S. G. A. W. A. Xinming Ou, “MulVAL: A Logic-based Network Security Analyzer,” in USENIX Security Sympo, 31 July 2005.

“Datalog,” [Online]. Available: 2022]. Available: [Accessed 20 6 2022].

S. G. A. A. Xinming Ou, “MulVAL: A Logic-based Network Security Analyzer,” in USENIX Security Symposium, 31 July 2005.

B. J. K. W. a. F. D. Marcin Szpyrka, “Telecommunications Networks Risk Assessment with Bayesian Networks,” in Computer Information Systems and Industrial Management Proceedings of the 12th IFIP TC8 International Conference CISIM, 2013.

R. B. Y. T. Zhenguo Hu, “Automated Penetration Testing Using Deep Reinforcement Learning,” in 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS and PW), 2020.

“GNS3,” [Online]. Available: [Accessed 20 06 2022].

“Metasploit,” [Online]. Available: https://www.offensive [Accessed 20 06 2022].

“Metasploit Pivoting,” [Online]. Available: [Accessed 22 06 2022].

“Pymetasploit,” [Online]. Available:[Accessed 20 06 2022].

S. W. Y. C. R. Z. a. C. W. Ianping Zeng, “Survey of Attack Graph Analysis Methods from the Perspective of Data and Knowledge Processing,” Security and Communication Networks, vol. 2019, no. 2031063, 2019.


Abstract views: 252 / PDF downloads: 141



How to Cite

Hung, N. V., & Cong, N. T. (2023). Applying reinforcement learning in automated penetration testing. Journal of Science and Technology on Information Security, 3(17), 61-77.