Generating evasive payloads for assessing Web Application Firewalls with Reinforcement Learning and Pre-trained Language Models
DOI:
https://doi.org/10.54654/isj.v2i25.1128Keywords:
Web Application Firewall, reinforcement learning, large language model, payload generation, grammar attacksTóm tắt
Web Application Firewalls (WAFs) serve as a critical defense mechanism against various web-based attacks such as SQL Injection (SQLi), Cross-Site Scripting (XSS), Server-Side Request Forgery (SSRF), Remote Code Execution (RCE), and NoSQL Injection. However, modern adversaries often craft evasive and obfuscated payloads capable of bypassing traditional WAF rules. To effectively assess and challenge the robustness of WAFs, we propose DEG-WAF, a Deep Evasion Generation framework that leverages Large Language Models (LLM) in conjunction with Reinforcement Learning (RL) to generate evasive payloads against WAFs. The system consists of four core components: a payload generation agent based on a pre-trained LLM (OPT-125M), a reward model that approximates WAF behavior, a grammar-based sampling agent that ensures syntactic validity, and an RL agent trained with either Proximal Policy Optimization (PPO) or Advantage Actor-Critic (A2C) to fine-tune generation strategies. Experimental evaluations on real-world WAFs, including ModSecurity and SafeLine, demonstrate that the A2C-based model significantly outperforms baseline LLMs—achieving a bypass success rate of 80.16% on SQLi and 74.70% on NoSQLi for ModSecurity, and 97.8% on RCE for SafeLine. These results underscore the potential of our LLM-RL framework to serve as a robust foundation for evaluating and enhancing the resilience of WAF systems under adversarial conditions.
Downloads
References
O. Fredj, O. Cheikhrouhou, M. Krichen, H. Hamam, and A. Derhab, “An owasp top ten driven survey on web application protection methods,” 11 2020.
V. Clincy and H. Shahriar, “Web application firewall: Network security models and configuration,” in 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), vol. 01, 2018, pp. 835–836. DOI: 10.1109/COMPSAC.2018.00144
A. Coscia, V. Dentamaro, S. Galantucci, A. Maci, and G. Pirlo, “Progesi: a proxy grammar to enhance web application firewall for sql injection prevention,” IEEE Access, vol. 12, pp. 107 689–107 703, 08 2024. DOI:
1109/ACCESS.2024.3438092
N. N. Thanh, V.-G. Ung, P. T. Duy, and V.-H. Pham, “A study on adversarial attacks for benchmarking deep learning-based
web application firewalls,” in 2024 RIVF International Conference on Computing and Communication Technologies (RIVF). IEEE, 2024, pp. 151–155.
A. Valenza, L. Demetrio, G. Costa, and G. Lagorio, “Waf-a-mole: An adversarial tool for assessing ml-based wafs,” SoftwareX, vol. 11, p. 100367, 2020. DOI: https://doi.org/10.1016/j.softx.2019.100367
H. Liang, X. Li, D. Xiao, J. Liu, Y. Zhou, A. Wang, and J. Li, “Generative pre-trained transformer-based reinforcement learning for testing web application firewalls,” IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 1, pp. 309–324, 2024. DOI: 10.1109/TDSC.2023.3252523
D. Leung, O. Tsai, K. Hashemi, B. Tayebi, and M. A. Tayebi, “Xploitsql: Advancing adversarial sql injection attack generation with language models and reinforcement learning,” in Proceedings of the 33rd ACM
International Conference on Information and Knowledge Management, ser. CIKM ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 4653–4660. DOI: 10.1145/3627673.3680102
S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” . , https://arxiv.org/abs/2402.06196
H. Xu, S. Wang, N. Li, K. Wang, Y. Zhao, K. Chen, T. Yu, Y. Liu, and H. Wang, “Large language models for cyber security: A
systematic literature review,” arXiv preprint arXiv:2405.04760, 2024.
V. Babaey and A. Ravindran, “Gensqli: A generative artificial intelligence framework for automatically securing web application firewalls against structured query language injection attacks,” Future Internet, vol. 17, no. 1, 2025. DOI: 10.3390/fi17010008
D. Miczek, D. Gabbireddy, and S. Saha, “Leveraging llm to strengthen ml-based cross-site scripting detection,” . , https: //arxiv.org/abs/2504.21045
Z. Gui, E. Wang, B. Deng, M. Zhang, Y. Chen, S. Wei, W. Xie, and B. Wang, “Sqligpt: Evaluating and utilizing large language models for automated sql injection black-box detection,” Applied Sciences, vol. 14, no. 16, 2024. DOI: 10.3390/app14166929
V. Babaey and A. Ravindran, “Genxss: an ai-driven framework for automated detection of xss attacks in wafs,” . , https://arxiv.org/ abs/2504.08176
H. Kheddar, D. W. Dawoud, A. I. Awad, Y. Himeur, and M. K. Khan, “Reinforcementlearning-based intrusion detection in communication networks: A review,” IEEE Communications Surveys & Tutorials, 2024.
M. Ghasemi and D. Ebrahimi, “Introduction to reinforcement learning,” . , https://arxiv.
org/abs/2408.07712
S. Finistrella, S. Mariani, and F. Zambonelli, “Multi-agent reinforcement learning for cybersecurity: Classification and survey,” Intelligent Systems with Applications, p. 200495, 2025.
C. Folini and I. Ristic, ModSecurity Handbook, Second Edition, 2nd ed. London, GBR: Feisty Duck, 2017.
Chaitin Technology, “Safeline web application firewall documentation,” 2023. , Access date: 10/6/2025, https://docs.waf .chaitin.com
Wargio, “Naxsi: Nginx anti xss and sql injection,” 2023. , Access date: 10/6/2025, https://github.com/wargio/naxsi
OWASP ModSecurity Core Rule Set Team, “Owasp core rule set (crs),” 2023. , Access date: 9/6/2025, https://github.com/corerules et/coreruleset
S. Dhote, A. Magdum, S. Singh, and D. Raigar, “Ml based web application firewall for signature and anomaly detection using feature extraction,” in 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), 2024, pp. 1–6. DOI: 10.1109/ICCCNT61001.2024.10725511
K. S. Kalyan, “A survey of gpt-3 family large language models including chatgpt and gpt-4,” Natural Language Processing Journal, vol. 6, p. 100048, 2024. DOI: https://doi.org/10.1016/j.nlp.2023.100048
G. Yenduri, R. M, C. S. G, S. Y, G. Srivastava, P. K. R. Maddikunta, D. R. G, R. H. Jhaveri, P. B, W. Wang, A. V. Vasilakos,
and T. R. Gadekallu, “Generative pre-trained transformer: A comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions” , https://arxiv.org/abs/2305.10435
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models”, https://arxiv.org/abs/2205.01068
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar,
A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models”, https://arxiv.org/abs/2302.13971
H. Zhou, C. Hu, Y. Yuan, Y. Cui, Y. Jin, C. Chen, H. Wu, D. Yuan, L. Jiang, D. Wu, X. Liu, J. Zhang, X. Wang, and J. Liu, “Large language model (llm) for telecommunications: A comprehensive survey on principles, key techniques, and opportunities,” IEEE Communications Surveys Tutorials, vol. 27, no. 3, pp. 1955–2005, 2025. DOI: 10.1109/COMST.2024.3465447
Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” High-Confidence Computing, vol. 4, no. 2, p. 100211, Jun. 2024. DOI: 10.1016/j.hcc.2024.100211
J. Zhang, H. Bu, H. Wen, Y. Liu, H. Fei, R. Xi, L. Li, Y. Yang, H. Zhu, and D. Meng, “When llms meet cybersecurity: A systematic literature review,” Cybersecurity, vol. 8, no. 1, p. 55, 2025.
A. Ramé, N. Vieillard, L. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret, “Warm: On the benefits
of weight averaged reward models,” . , https://arxiv.org/abs/2401.12187
V. Atlidakis, R. Geambasu, P. Godefroid, M. Polishchuk, and B. Ray, “Pythia: Grammar-based fuzzing of rest apis with coverage-guided feedback and learning-based mutations,” arXiv preprint arXiv:2005.11498, 2020.
P. Godefroid, H. Peleg, and R. Singh, “Learn&fuzz: Machine learning for
input fuzzing,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2017, pp. 50–59.
P. Srivastava and M. Payer, “Gramatron: effective grammar-aware fuzzing,” in Proceedings of the 30th ACM SIGSOFT
International Symposium on Software Testing and Analysis, ser. ISSTA 2021. New York, NY, USA: Association for Computing Machinery, 2021, p. 244–256. DOI: 10.1145/3460319.3464814
Z. Qu, X. Ling, T. Wang, X. Chen, S. Ji, and C. Wu, “Advsqli: Generating adversarial sql injections against realworld waf-as-a-service,” IEEE Transactions on Information Forensics and Security, vol. 19, p. 2623–2638, 2024. DOI: 10.1109/tifs.2024.3350911
K. Li, H. Yang, and W. Visser, “Evolutionary multi-task injection testing on web application firewalls,” . , https://arxiv.org/abs/2206.05743
C. Wu, J. Chen, S. Zhu, W. Feng, R. Du, and Y. Xiang, “Wafbooster: Automatic boosting of waf security against mutated malicious payloads,” . , https://arxiv.org/abs/2501.140 08
F. Yang, W. Zhou, Z. Liu, D. Zhao, and D. Held, “Reinforcement learning in a safetyembedded mdp with trajectory optimizatioz , https://arxiv.org/abs/2310.06903.
M.-A. Chadi and H. Mousannif, “Understanding reinforcement learning algorithms: The progress from basic qlearning to proximal policy optimization”, https://arxiv.org/abs/2304.00026
OpenAI, “Spinning up - proximal policy optimization (ppo),” 2024. , Access date: 10/6/2025, https://spinningup.openai.com/en /latest/algorithms/ppo.html
GeeksforGeeks Contributors, “Actor-critic algorithm in reinforcement learning,” 2023. , Access date: 10/6/2025, https://www.geeksf orgeeks.org/machine-learning/actor-critic-a lgorithm-in-reinf orcement-learning/
Swisskyrepo, “Payloads all the things,” 2023, Access date: 10/6/2025, https://github.com/swisskyrepo/PayloadsAllTheThings.
Downloads
Published
How to Cite
Issue
Section
License
Proposed Policy for Journals That Offer Open Access
Authors who publish with this journal agree to the following terms:
1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
Proposed Policy for Journals That Offer Delayed Open Access
Authors who publish with this journal agree to the following terms:
1. Authors retain copyright and grant the journal right of first publication, with the work [SPECIFY PERIOD OF TIME] after publication simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
