Modeling Penetration Testing with Reinforcement Learning Using Capture-the-Flag Challenges: Trade-offs between Model-free Learning and A Priori Knowledge Modeling Penetration Testing with Reinforcement Learning Using Capture-the-Flag Challenges: Trade-offs between Model-free Learning and A Priori Knowledge

Problem & Motivation 问题与动机

Automating penetration testing is difficult because the range of actions and knowledge an expert relies on are hard to capture computationally. This paper investigates to what extent model-free reinforcement learning algorithms can be used to solve CTF challenges, and identifies the specific challenge of discovering the hidden structure of a target system.

自动化渗透测试非常困难，因为专家所依赖的操作范围和知识很难通过计算方式捕捉。本文研究了无模型强化学习（model-free RL）算法在多大程度上可以用于解决 CTF 挑战，并识别了发现目标系统隐藏结构的特定挑战。

While RL has shown success in games and complex decision-making, applying it to penetration testing introduces unique challenges not present in traditional games: the structure of the problem is obscured, the environment may be non-stationary (defenders adapt), and the state/action spaces can be enormous. The paper fills a gap by critically assessing when model-free RL works for pentest-like problems and when a priori knowledge is necessary.

虽然强化学习在游戏和复杂决策中取得了成功，但将其应用于渗透测试会引入传统游戏中不存在的独特挑战：问题的结构被遮蔽、环境可能是非平稳的（防御者会自适应）、且状态/动作空间可能极其庞大。本文通过批判性地评估无模型强化学习何时适用于类渗透测试问题，以及何时需要先验知识，填补了这一空白。

Threat Model 威胁模型

Red team agent interacting with a vulnerable target system modeled as a fully observable Markov decision process. The blue team (defender) may be passive or may react to attacker actions by relocating services. The agent receives dense negative rewards per step and a large positive reward for capturing a flag.

红队智能体与被建模为完全可观测马尔可夫决策过程的脆弱目标系统进行交互向。蓝队（防御者）可能是被动的，也可能通过迁移服务来对攻击者的操作做出反应。智能体每步都会收到密集的负奖励，并在捕获 flag 时获得巨大的正奖励。

Methodology 核心方法

The authors formalize three classes of CTF problems (port scanning, server hacking, website hacking) as reinforcement learning problems using the MDP framework. They implement tabular Q-learning agents to solve increasingly complex CTF scenarios across five simulations. They systematically evaluate the impact of non-stationarity and high-entropy environments on RL performance, and then demonstrate how three forms of a priori knowledge -- lazy loading, state aggregation, and imitation learning -- can mitigate these challenges.

作者将三类 CTF 问题（端口扫描、服务器黑客攻击、网站黑客攻击）形式化为基于 MDP 框架的强化学习问题。他们实现了表格 Q 学习（tabular Q-learning）智能体，通过五次模拟来解决日益复杂的 CTF 场景。他们系统地评估了非平稳性和高熵环境对强化学习性能的影响，然后演示了三种形式的先验知识——延迟加载（lazy loading）、状态聚合（state aggregation）和模仿学习（imitation learning）——如何减轻这些挑战。

Architecture 架构设计

Simulated CTF environments built using the OpenAI Gym interface, with tabular Q-learning agents interacting with custom environments modeling port scanning, server hacking, and website hacking scenarios.

使用 OpenAI Gym 接口构建的模拟 CTF 环境，表格 Q 学习智能体与模拟端口扫描、服务器黑客攻击和网站黑客攻击场景的自定义环境进行交互。

Memory Mechanism 记忆机制

none

Attack Phases Covered 覆盖的攻击阶段

Evaluation 评估结果

In simple stationary port scanning (Simulation 1, N=64 ports), Q-learning converges to the optimal policy within ~200 episodes. Non-stationarity (Simulation 2) degrades performance proportionally to the detection probability p, with p=1 reducing the agent to random guessing. For complex server hacking (Simulation 3), lazy loading makes tabular Q-learning feasible by reducing memory from O(2*10^6) states to O(3.5*10^3) visited states. State aggregation (Simulation 4) enables learning a generalizable policy across similar files in ~10^5 episodes. Imitation learning (Simulation 5) with 100 demonstrations matches the reward level that standard RL requires ~2000 episodes to achieve, providing an order-of-magnitude speedup.

在简单的平稳端口扫描中（模拟 1，N=64 个端口），Q 学习在大约 200 个回合内收敛到最优策略。非平稳性（模拟 2）会按探测概率 p 比例降低性能，当 p=1 时，智能体退化为随机猜测。对于复杂的服务器黑客攻击（模拟 3），延迟加载通过将内存从 O(2*10^6) 个状态减少到 O(3.5*10^3) 个访问状态，使得表格 Q 学习变得可行。状态聚合（模拟 4）使得在大约 10^5 个回合内学习到跨相似文件的通用策略成为可能。带有 100 个演示的模仿学习（模拟 5）达到了标准强化学习需要大约 2000 个回合才能达到的奖励水平，提供了数量级的加速。

Environment 评估环境

Metrics 评估指标

Baseline Comparisons 基准对比

• Standard tabular Q-learning (no prior knowledge)
• Q-learning with lazy loading
• Q-learning with state aggregation
• Q-learning with imitation learning (100, 200, 500 demonstrations)

Scale 评估规模

5 simulated CTF scenarios of increasing complexity

Contributions 核心贡献

• Formal modeling of three classes of CTF problems (port scanning, server hacking, website hacking) as RL problems using the MDP framework
• Identification and analysis of the fundamental challenge of structure discovery in applying RL to penetration testing, distinguishing CTF systems from max-entropy systems
• Systematic experimental evaluation of tabular Q-learning across five simulations with increasing complexity and non-stationarity
• Demonstration that three forms of a priori knowledge (lazy loading, state aggregation, imitation learning) can effectively address scalability and exploration challenges in RL-based penetration testing
• Open-source implementation of all simulations using OpenAI Gym (github.com/FMZennaro/CTF-RL)

• 使用 MDP 框架将三类 CTF 问题（端口扫描、服务器黑客攻击、网站黑客攻击）正式建模为强化学习问题
• 识别并分析了在渗透测试中应用强化学习时“结构发现”这一根本挑战，将 CTF 系统与最大熵系统区分开来
• 通过五次复杂度和非平稳性递增的模拟，对表格 Q 学习进行了系统的实验评估
• 证明了三种形式的先验知识（延迟加载、状态聚合、模仿学习）可以有效解决基于强化学习的渗透测试中的可扩展性和探索挑战
• 使用 OpenAI Gym 开源了所有模拟的实现 (github.com/FMZennaro/CTF-RL)

Limitations 局限性

• Uses only tabular Q-learning, which inherently limits the complexity and scale of problems that can be addressed; the authors acknowledge this restricts generalizability to larger real-world scenarios
• All simulations are highly simplified abstractions of real penetration testing; real systems have far larger state and action spaces, richer observations, and more complex dynamics
• Does not use deep RL (neural network function approximation) despite the title suggesting it; the paper explicitly trades off scalability for interpretability
• Non-stationarity is only briefly explored in one simulation (Simulation 2) with a simple model of defender behavior
• No evaluation on real systems, real CTF platforms, or established benchmarks; all environments are custom-built toy problems
• The paper does not address partial observability, which is a more realistic model for penetration testing than the fully observable MDP assumed
• State aggregation and imitation learning still require human expertise to define equivalence classes or provide demonstrations, undermining the fully autonomous goal

• 仅使用了表格 Q 学习，这本质上限制了可以处理的问题的复杂性和规模；作者承认这限制了对更大规模现实场景的推广
• 所有模拟都是对真实渗透测试的高度简化抽象；真实系统拥有大得多的状态和动作空间、更丰富的观察结果和更复杂的动力学
• 尽管标题提到了深度强化学习，但并未实际使用（如神经网络函数近似）；论文明确表示为了可解释性而权衡了可扩展性
• 仅在一次模拟（模拟 2）中通过简单的防御者行为模型初步探讨了非平稳性
• 未在真实系统、真实 CTF 平台或既定基准上进行评估；所有环境都是自定义构建的玩具问题
• 论文未解决部分可观测性问题，而这比假设的完全可观测 MDP 更符合渗透测试的现实模型
• 状态聚合和模仿学习仍需要人类专家来定义等价类或提供演示，这削弱了全自主的目标

Research Gaps 研究空白

• Scaling RL-based penetration testing to realistic state and action spaces remains unsolved; the transition from tabular to deep RL for pentest is largely unexplored
• The integration of model-based and model-free approaches for penetration testing lacks a principled framework
• Transfer learning between different CTF problems or vulnerability classes has not been studied
• Non-stationary and adversarial defender dynamics are insufficiently modeled in current RL formulations for penetration testing
• No established benchmarks exist for evaluating RL agents on penetration testing tasks in a standardized way
• The relationship between exploration strategies in RL and information gathering in penetration testing needs deeper investigation

• 将基于强化学习的渗透测试扩展到现实的状态和动作空间仍未解决；从表格到深度强化学习在渗透测试中的转变在很大程度上仍未被探索
• 渗透测试中模型驱动（model-based）和无模型（model-free）方法的结合缺乏原则性的框架
• 尚未研究不同 CTF 问题或漏洞类别之间的迁移学习
• 在当前的渗透测试强化学习表述中，对非平稳和对抗性的防御者动力学建模不足
• 尚不存在以标准化方式评估强化学习智能体执行渗透测试任务的既定基准
• 强化学习中的探索策略与渗透测试中的信息收集之间的关系需要更深入的研究

Novel Techniques 新颖技术

• Formalization of the 'structure discovery' problem as the central challenge distinguishing RL for pentest from RL for traditional games
• Concept of max-entropy systems as a theoretical limit where RL reduces to random guessing, providing a framework for reasoning about when RL can and cannot help
• Application of lazy loading to dynamically build Q-tables only for visited state-action pairs, making tabular RL feasible for large pentest state spaces
• Use of state aggregation to generalize across structurally similar components (e.g., treating all web files uniformly) in penetration testing contexts

• 将“结构发现”问题正式化为区分渗透测试强化学习与传统游戏强化学习的核心挑战
• 提出了最大熵系统的概念，作为强化学习退化为随机猜测的理论极限，为推理强化学习何时能（或不能）提供帮助提供了框架
• 应用延迟加载来仅针对访问过的状态-动作对动态构建 Q 表，使得表格强化学习在大型渗透测试状态空间中变得可行
• 在渗透测试语境下使用状态聚合来对结构相似的组件（例如统一处理所有 Web 文件）进行泛化

Open Questions 开放问题

• Can deep RL (with function approximation) scale to realistic penetration testing scenarios while maintaining the convergence guarantees of tabular methods?
• How should prior knowledge be optimally injected into RL agents for pentest -- through reward shaping, curriculum learning, hierarchical decomposition, or demonstrations?
• Can RL agents learn to transfer penetration testing strategies across fundamentally different target systems or vulnerability classes?
• How do modern LLM-based agents compare to RL agents for penetration testing, given that LLMs inherently encode vast prior knowledge about system structures?
• What is the right level of abstraction for modeling penetration testing as an RL problem -- too abstract loses realism, too detailed becomes intractable?

• 深度强化学习（带有函数近似）能否扩展到现实的渗透测试场景，同时保持表格方法的收敛保证？
• 应如何以最佳方式为渗透测试强化学习智能体注入先验知识——是通过奖励塑形、课程学习、分层分解还是演示？
• 强化学习智能体能否学会跨根本不同的目标系统或漏洞类别迁移渗透测试策略？
• 鉴于 LLM 天生编码了关于系统结构的海量先验知识，现代基于 LLM 的智能体与渗透测试强化学习智能体相比表现如何？
• 将渗透测试建模为强化学习问题的合适抽象级别是什么——太抽象会失去现实感，太详细则变得难以处理？

Builds On 基于前人工作

• DARPA Cyber Grand Challenge (2016)
• Hoffmann - Simulated penetration testing using AI planning
• Ghanem and Chen - RL for network penetration testing
• Pozdniakov et al. - Smart computer security audit with deep RL
• Sarraute et al. - Penetration testing as POMDP solving
• Sutton and Barto - Reinforcement Learning: An Introduction

Open Source 开源信息

Yes - https://github.com/FMZennaro/CTF-RL

Tags