Executive Summary

This report synthesizes 68 academic papers on LLM-based penetration testing, published between 2005 and 2026, with 63% concentrated in 2025–2026. The field has undergone explosive growth: from early chat-based assistants (#01 PentestGPT, 2024) to sophisticated multi-agent systems with hierarchical planning (#04 xOffense, #19 Cochise, 2025) and RL-tuned models (#10 Pentest-R1, #54 Cyber-Zero, 2025) in just two years.

Key Findings

1. Rapid but immature growth. 68% of papers are preprints and only 13% appear at top conferences. The field lacks theoretical foundations — no paper offers formal complexity analysis or provable guarantees. Most systems rely on GPT-4/4o despite evidence that fine-tuned smaller models (#04 Qwen-32B, #42 Hackphyr-7B) achieve competitive or superior results.

2. Severe phase coverage imbalance. 94% of papers address exploitation, but only 25% tackle lateral movement, 21% cover reporting, and a mere 1% address persistence. Current LLM agents are effectively "single-host exploiters" rather than end-to-end penetration testers. Only 4 systems (#06, #19, #42, #60) attempt post-exploitation automation.

3. Fragmented and unrealistic evaluation. No benchmark is used by more than 7 papers. 40% rely on custom lab environments, and only 6% evaluate on real networks. CTF challenges dominate evaluation, but they poorly approximate actual enterprise penetration tests involving multi-host networks, Active Directory, and time-constrained engagements.

Implications for Research

The field presents significant opportunities for doctoral research. We identify 17 concrete research gaps across technical (T1–T7), evaluation (E1–E3), application domain (D1–D3), safety (S1–S2), and theoretical (Th1–Th2) dimensions, and propose 8 PhD thesis directions with feasibility assessments. The most promising unexplored areas are: multi-host network penetration (addressed by only 3 papers), cross-session persistent learning (only 1 paper), and the completely uncharted DRL+LLM integration space.

Agent Architectures & System Design

Overview

The field of LLM-based penetration testing has evolved rapidly from 2023 to 2026, moving from simple chat-based assistants to sophisticated multi-agent systems with structured planning, memory, and tool integration. This analysis covers 65 papers, of which approximately 40 propose concrete system architectures.

Architecture Taxonomy

Single-Agent Systems

Papers where a single LLM handles all reasoning and execution:

• #02 PentestGPT V2 (EGATS-based tree search)
• #03 CHECKMATE (Classical Planning+)
• #10 Pentest-R1 (RL-tuned single agent)
• #13 RapidPen (layered ReAct)
• #15 PwnGPT (prompt-chaining for exploit generation)
• #18 Pentest Copilot (step-chaining)
• #20 HackSynth (Planner-Summarizer dual module)
• #33 Mantis (defensive prompt injection)
• #38 Getting Pwn'd by AI (SSH feedback loop)
• #42 Hackphyr (fine-tuned 7B model)
• #44 CTFAgent (RAG-augmented)
• #45 Improving LLM RL CTF (RL-trained)
• #48 CHAP (context relay across sessions)
• #52 RedTeamLLM (ReAct-based)
• #54 Cyber-Zero (trajectory-trained)
• #55 LLMs as Hackers (hackingBuddyGPT)
• #56 EnIGMA (SWE-agent with IATs)
• #62 LLM Agents Exploit One-day (minimal ReAct)

Multi-Agent Systems

Papers with multiple specialized LLM agents:

• #01 PentestGPT (Reasoning/Generation/Parsing)
• #04 xOffense (5 components with MemAgent)
• #05 AutoPenGPT (Decision/Expert/Analyser/Summarizer/Util)
• #06 AutoAttacker (Summarizer/Planner/Navigator/Experience)
• #07 AutoPT (FSM with Agent+Rule states)
• #08 AutoPentester (5 agents + Repetition Identifier)
• #09 VulnBot (phase-specialized agents)
• #11 PentestAgent (Recon/Search/Planning/Execution)
• #14 ARACNE (multi-LLM modules)
• #21 Teams of LLM (HPTSA manager + experts)
• #22 CAI (role-based swarm)
• #41 BreachSeek (LangGraph supervisor)
• #43 CRAKEN (ReWOO with Graph-RAG)
• #46 CyberExplorer (reactive multi-agent)
• #49 G-CTR (game-theoretic)
• #57 MAPTA (Coordinator/Sandbox/Validation)
• #61 AutoPentest (Planner/Supervisor/Workers)
• #66 PenHeal (Planner/Executor/Instructor/Summarizer/Extractor)

Hierarchical (Planner-Executor) Systems

A specific multi-agent pattern that separates strategic planning from tactical execution:

• #19 Cochise (Planner + Executor for AD)
• #21 Teams of LLM (Manager + task-specific experts)
• #60 Incalmo (LLM planner + domain expert agents)

Key Design Dimensions

Planning Strategy

Strategy	Papers
ReAct	#05, #06, #13, #14, #19, #22, #28, #29, #39, #42, #44, #47, #52, #55, #56, #57, #60, #62, #66
Tree-based (PTT, attack tree, MCTS)	#01, #02, #08, #19
Chain-of-thought	#08, #11, #18
Classical planning	#03, #31
Finite state machine	#07, #59
RL-based	#10, #24, #26, #27, #45
Hierarchical planning	#21, #41, #61
Task graph (DAG)	#04, #09
Game-theoretic	#25, #49
None/minimal	#15, #16, #20, #32, #33, #50, #53

Observation: ReAct dominates (~30% of systems), but more structured approaches (classical planning, FSM, task graphs) show promising results when applied.

Memory Mechanism

Mechanism	Papers
Conversation history only	#01, #10, #14, #19, #20, #42, #45, #47, #48, #50, #52, #54, #55, #62, #67
RAG (vector store)	#02, #04, #05, #06, #08, #09, #11, #13, #18, #29, #34, #43, #44, #60, #61, #66
Knowledge graph	#03, #31, #43
Dual memory (success/failure logs)	#59
Structured handoff (cross-session)	#48
Belief state (POMDP)	#24
None	#32, #33, #50

Observation: RAG is the most popular memory mechanism, but its effectiveness is debated. #43 CRAKEN found only 43.8% of retrieved documents meet relevance standards. Cross-session memory (#48 CHAP) is underexplored.

Automation Level

Level	Papers
Fully autonomous	#02, #03, #04, #06, #07, #08, #09, #10, #13, #14, #15, #19, #20, #21, #22, #41, #42, #45, #46, #49, #52, #54, #57, #60, #62, #66
Semi-autonomous	#05, #11, #61
Human-in-the-loop	#01, #16, #39, #40, #59, #67
Copilot	#18

Observation: The field is aggressively pursuing full autonomy, but the most successful real-world results (#02 PentestGPT V2) still use human-in-the-loop elements for difficult decisions.

Tool Integration Approaches

1. Shell command generation (most common): Agent generates bash/shell commands, system executes them (#01-#09, most systems)
2. Predefined action library: Curated tool wrappers with preconditions (#03 CHECKMATE, #36 ADAPT)
3. MCP/function calling: Structured tool interfaces (#57 MAPTA)
4. Interactive tools (IATs): Non-blocking parallel sessions for debuggers and network tools (#56 EnIGMA)
5. Expert agents: Domain-specific tool handlers (#60 Incalmo)

Architectural Patterns

Pattern 1: Pentesting Task Tree (PTT)

A tree-structured representation of testing state maintained across steps.

• Used by: #01, #02, #08, #13, #19
• Strength: Provides structured context for LLM reasoning
• Weakness: Can become stale, vulnerable to LLM hallucination corrupting tree state

Pattern 2: Phase Decomposition

Splitting pentesting into distinct phases (recon, scan, exploit, etc.) with specialized agents per phase.

• Used by: #04, #05, #07, #09, #11, #58
• Strength: Reduces complexity per agent, enables specialization
• Weakness: Information loss at phase boundaries (#09 found 42.36% of failures due to context loss)

Pattern 3: RAG-Augmented Exploitation

Using retrieval to inject relevant exploit knowledge during testing.

• Used by: #04, #05, #08, #09, #11, #39, #43, #44, #61
• Strength: Supplements LLM knowledge gaps for specific CVEs/tools
• Weakness: Quality issues (#43: 43.8% relevance), can misguide agents (#44)

Pattern 4: Planning-Execution Separation

Decoupling high-level strategy from low-level command generation.

• Used by: #03, #19, #21, #60
• Strength: #60 Incalmo showed this enables small LLMs to outperform large ones
• Weakness: Information transfer between planner and executor is error-prone (#19)

Pattern 5: RL-Trained Agents

Using reinforcement learning to train or fine-tune agents for pentesting.

• Used by: #10, #45, #54
• Strength: Can learn from experience, improve over time
• Weakness: Reward sparsity, limited transfer across domains

Evolution Timeline

2023: Human-Guided Assistants

• PentestGPT (#01): First major LLM pentesting system, human-in-the-loop
• Getting Pwn'd (#38): Early exploration of LLM for privilege escalation
• LLM CTF Challenges (#67): Basic evaluation of ChatGPT on CTF tasks

2024: Autonomous Agent Explosion

• AutoAttacker (#06), AutoPT (#07): Push toward full autonomy
• Teams of LLM (#21): Multi-agent for zero-day exploitation
• NYU CTF Bench (#28), AutoPenBench (#29): Benchmark establishment
• LLM Exploit One-day (#62): Demonstrated GPT-4 can exploit real CVEs

2025: Architecture Sophistication

• CHECKMATE (#03): Classical planning integration
• xOffense (#04): Fine-tuned mid-scale model beats larger ones
• VulnBot (#09): Open-source multi-agent with PTG
• Pentest-R1 (#10): First RL-trained pentesting LLM
• PentestEval (#58): Stage-level benchmark revealing exploitation is the bottleneck
• Incalmo (#60): Proved abstractions matter more than model size
• EnIGMA (#56): Interactive tools for CTF solving

2026: Maturation and Scaling

• PentestGPT V2 (#02): Top-100 live HTB ranking, most comprehensive real-world validation
• HackWorld (#53): GUI/visual agent evaluation
• CHAP (#48): Long-running context management across sessions
• Cyber-Zero (#54): Trajectory synthesis for training without runtime

Gaps & Opportunities

1. Post-Exploitation and Lateral Movement — Critical

Most systems focus exclusively on initial access (getting a shell or capturing a flag). Of the approximately 40 systems surveyed, only 3 address post-exploitation or lateral movement in multi-host networks: #06 AutoAttacker (post-breach automation), #19 Cochise (Active Directory lateral movement), and #60 Incalmo (multi-host pivoting via abstractions). Even among these, results are limited: #19 achieved only ~1.83 compromised accounts per run, and #60 succeeded by using high-level action abstractions rather than LLM-driven lateral movement. The Phase Decomposition pattern (Pattern 2 above) typically terminates after exploitation, with no agents specialized for post-exploitation phases. Real penetration tests spend the majority of time in post-exploitation, making this gap a fundamental barrier to practical deployment. See also gap T1 in Research Gaps document.

2. Multimodal Capabilities — Critical

No system in the survey effectively handles GUI-based testing, image analysis, or visual interaction with web applications. The sole evaluation of visual agents (#53 HackWorld) showed that Computer-Using Agents achieve below 12% success on web vulnerabilities through visual interaction, highlighting a massive capability gap. This limitation is explicitly noted as a gap by at least 6 papers (#01, #03, #04, #09, #10, #44), yet none attempt to address it architecturally. All 40+ systems in the Architecture Taxonomy above operate exclusively through text-based shell commands (Tool Integration approach 1), with zero systems incorporating visual perception into their tool integration stack. Given that web application testing represents the largest segment of the commercial pentesting market, this gap severely limits real-world applicability. See also gap T2 in Research Gaps document.

3. Cross-Session Memory — Critical

Of the 40 systems surveyed, only #48 CHAP explicitly addresses maintaining context across multiple testing sessions. The Memory Mechanism table above shows that 15 systems rely on conversation history alone, which is lost between sessions. Multiple papers report severe context degradation: #05 showed retention drops to 57.1% on complex tasks, #09 found 42.36% of failures attributable to session context loss, and #01 noted context degradation as a primary limitation. Real penetration tests span days or weeks, requiring persistent state management that no current architecture provides at scale. The only system operating across multi-day engagements (#02 PentestGPT V2) achieves this through human-maintained state rather than automated memory. This gap is particularly acute for the Hierarchical and Multi-Agent patterns, where coordination state across sessions adds further complexity. See also gap T3 in Research Gaps document.

4. Defense-Aware Agents — Important

No offensive system in the entire survey accounts for active defenders such as IDS/IPS, honeypots, or SOC analysts. This is starkly illustrated by two defensive papers: #32 CHeaT achieved 100% defense success against PentestGPT using simple countermeasures, and #33 Mantis demonstrated that prompt injection can halt or even reverse LLM-based attacks. At least 4 offensive papers (#19, #06, #07, #08) explicitly acknowledge "evaluation limited to systems without active defenses" as a limitation. Among all architecture patterns identified above, none incorporate evasion, stealth, or defender-awareness as a design consideration. This creates a fundamental disconnect between benchmark performance and real-world deployability, where defenders are always present. See also gap T5 in Research Gaps document.

5. Formal Verification of Agent Plans — Important

No system provides formal guarantees about agent behavior, scope adherence, or safety constraints. Classical planning (#03 CHECKMATE) represents the closest approach, demonstrating that formal action models improve consistency (100% vs. 75% for ReAct baselines), but it does not verify that generated plans stay within authorized scope. The FSM-based approaches (#07 AutoPT, #59 RefPentester) provide some structural guarantees about state transitions but not about the safety of individual actions. The POMDP formalization in #24 and the labeled transition system in #36 offer theoretical foundations, but neither has been applied to verify real agent behavior. Given that autonomous agents executing commands on live systems risk unintended damage (#20, #45), formal verification is a prerequisite for real-world deployment and regulatory acceptance. See also gaps S2 and Th1 in Research Gaps document.

6. Cost-Efficient Architectures — Important

Of the 26 fully autonomous systems listed in the Automation Level table, the vast majority depend on GPT-4 or GPT-4o, creating significant cost and latency barriers. However, two key results suggest this dependency is not fundamental: #04 xOffense demonstrated that a fine-tuned 32B model outperforms 405B models on pentesting tasks, and #60 Incalmo showed that good planning-execution abstractions allow Claude Haiku (a small model) to match or exceed larger models. Pattern 4 (Planning-Execution Separation) appears particularly promising for cost efficiency, as it allows the expensive reasoning to be concentrated in a small planner while cheaper models handle execution. Additionally, #42 Hackphyr showed that a fine-tuned 7B model can perform competitively on privilege escalation tasks, and #54 Cyber-Zero demonstrated trajectory synthesis can train small models without expensive runtime inference. Despite these encouraging results, no systematic study compares architecture-cost tradeoffs across the design space. See also PhD Topic 5 in Research Gaps document.

7. Learning from Failure — Emerging

The overwhelming majority of systems start each engagement from scratch with no accumulated experience. Of 40+ systems, only 3 implement any form of experience retention: #59 RefPentester maintains dual success/failure logs, #06 AutoAttacker includes an experience manager component, and #13 RapidPen uses success-case RAG. #48 CHAP's handoff protocols preserve context between sessions but do not generalize lessons learned to new targets. No system demonstrates cross-engagement transfer learning, where lessons from testing one network improve performance on another. This contrasts sharply with human pentesters, who accumulate expertise over years. The RAG-Augmented Exploitation pattern (Pattern 3) could theoretically support experience storage, but current implementations retrieve static knowledge rather than learned strategies. See also gap T4 in Research Gaps document.

8. Reliable Tool Usage and Error Recovery — Important

LLM agents exhibit alarmingly high rates of invalid command generation, directly undermining the practical value of autonomous pentesting. Quantitative evidence spans multiple systems: #19 Cochise reported 35.9% invalid commands overall with a 94% failure rate for hashcat specifically, #09 VulnBot recorded 19.70% failed tool errors, and #42 Hackphyr noted persistently high invalid action rates. Despite these consistent findings, no architecture pattern explicitly addresses command validation or error recovery as a first-class design concern. The Predefined Action Library approach (Tool Integration approach 2, used by #03 and #36) partially mitigates this by constraining the action space, but sacrifices flexibility. Interactive tools (#56 EnIGMA's IATs) offer a different mitigation by allowing agents to observe tool output in real time, but this has only been applied to CTF contexts. Systematic architectural solutions for command validation, graceful error handling, and adaptive retry strategies remain an open problem. See also gap T7 in Research Gaps document.

9. Standardized Inter-Agent Communication — Emerging

Among the 18 multi-agent systems listed in the Architecture Taxonomy, there is no standardized protocol for inter-agent communication or information handoff. Each system invents its own coordination mechanism: #04 xOffense uses a MemAgent, #05 AutoPenGPT chains agents sequentially, #21 HPTSA uses a manager-expert hierarchy, #22 CAI uses role-based swarms, and #41 BreachSeek uses LangGraph supervision. Pattern 2 (Phase Decomposition) is particularly affected, as #09 VulnBot found that 42.36% of failures stem from information loss at phase boundaries between agents. Without standardized interfaces, multi-agent systems cannot benefit from modular agent reuse or community-developed specialized agents. This mirrors the evolution of microservice architectures in software engineering, where standardized APIs enabled ecosystem growth.

Evaluation & Benchmarking Landscape

Overview

Evaluation in LLM-based pentesting is fragmented. Most papers use different environments, metrics, and baselines, making cross-paper comparison nearly impossible. This analysis synthesizes evaluation practices across 65 papers.

Benchmark Inventory

CTF-Based Benchmarks

Benchmark	Type	Scale	Used By
NYU CTF Bench	CTF (6 categories)	200 challenges	#28, #54, #56, #65
AutoPenBench	Docker pentest tasks	33 tasks	#04, #10, #29
Cybench	Professional CTF	40 tasks (4 competitions)	#10, #54, #56, #63
InterCode-CTF	CTF framework	Variable	#10, #54, #56
XBOW	Web CTF	104 challenges	#02, #57
PentestEval	Stage-level	346 tasks, 12 scenarios	#58
picoCTF	Educational CTF	Variable	#45, #52
HackWorld	GUI-based web CTF	36 challenges	#53

Real-World / Simulated Environments

Environment	Type	Used By
HackTheBox	Real machines	#01, #02, #08, #13, #22, #59, #61
VulnHub	VM images	#05, #07, #29, #39, #40, #52
Vulhub (Docker)	Docker CVEs	#03, #04
Metasploitable	Intentionally vulnerable	#06, #41, #66
GOAD	Active Directory lab	#02, #19
MHBench	Multi-host OpenStack	#60
NetSecGame	Network simulation	#42
CyBORG	Cyber gym	#27

Observations

• No single benchmark dominates: The most reused are NYU CTF Bench (4 papers), AutoPenBench (3), Cybench (3), and HackTheBox (7)
• CTF vs. real pentest disconnect: CTF benchmarks dominate, but CTF ≠ real penetration testing. Only #02, #19, and #60 evaluate on realistic multi-host networks
• Benchmark contamination risk: #20 HackSynth identified that static CTF flags can be memorized from training data; #39 noted VulnHub machines may be in LLM training data

Metrics Used

Metric	Papers Using It	Notes
Task/challenge completion rate	Nearly all	Most common but definition varies
Success rate (binary)	#01, #06, #07, #13, #21	Coarse-grained
Sub-task completion	#08, #29, #58	More fine-grained
Milestone-based progress	#03, #29	Best practice for partial credit
Cost ($)	#02, #03, #19, #57, #60	Critical but often unreported
Time/steps	#03, #08, #55	Important for practical deployment
Human interactions	#08	Measures autonomy

Critical Metric Gaps

1. Stealth/detection avoidance: No paper measures whether the agent avoids triggering IDS/IPS
2. False positive rate: Only #57 MAPTA requires PoC validation
3. Reproducibility: Most papers report single-run results; #29 AutoPenBench found high variance across runs
4. Coverage breadth: Few papers measure what percentage of the attack surface was explored

Attack Phase Coverage

Phase	Papers Covering It	Gap Assessment
Reconnaissance	#01, #04, #05, #07, #09, #11, #16, #48, #57, #61	Well covered
Scanning	#01, #04, #05, #07, #09, #10, #13, #16, #48	Well covered
Enumeration	#01, #04, #05, #07, #09, #10, #11, #13, #16, #48, #55	Well covered
Exploitation	Nearly all	Best covered, yet hardest to succeed at
Post-exploitation	#06, #42, #48, #58	Severely underexplored
Privilege escalation	#05, #10, #16, #48, #52, #55	Moderate, #55 is dedicated to this
Lateral movement	#06, #19, #42, #60	Severely underexplored
Reporting	#05, #16, #57, #61	Minimal coverage

Key finding: The later stages of penetration testing (post-exploitation, lateral movement) are dramatically underrepresented. Most systems stop after getting a shell or capturing a flag.

Model Comparison

Models Most Frequently Tested

Model	Papers	Typical Result
GPT-4 / GPT-4o	~30 papers	Consistently best performer
GPT-3.5	~15 papers	Significantly worse than GPT-4
Claude 3.5 Sonnet	#22, #53, #56, #58	Competitive with GPT-4, best on some benchmarks
Llama family	#04, #09, #10, #42, #45, #54, #55, #62	Generally much weaker than GPT-4
DeepSeek	#10, #54, #58	Competitive when fine-tuned
Qwen	#04, #19, #54, #58	Promising with fine-tuning
o1/o3 (reasoning)	#19, #23, #45	Better for complex reasoning tasks

Key Trends

1. GPT-4 dominance: In #62, only GPT-4 solved any one-day exploits (87%); all other models scored 0%
2. Fine-tuning closes the gap: #04 xOffense's fine-tuned Qwen3-32B beat GPT-4o and Llama-405B
3. Reasoning models help: #19 Cochise found o1+GPT-4o compromised ~5x more accounts than non-reasoning models
4. Model size isn't everything: #60 Incalmo showed Haiku 3.5 with good abstractions beats Sonnet 4 without them

Critical Gaps in Evaluation

1. No Standardized Benchmark Suite

Each paper uses different environments, metrics, and baselines. #37 Benchmarking Practices systematically documented this fragmentation.

2. CTF ≠ Real Pentesting

CTF challenges are isolated, have known solutions, and lack active defenders. Real penetration tests involve multi-host networks, ambiguous goals, and time pressure. Only #02, #19, #60 approach real-world conditions.

3. No Active Defender Evaluation

Zero papers evaluate agents against actively defended systems. #32 CHeaT and #33 Mantis show trivial defenses defeat all current agents, yet no offensive paper accounts for this.

4. Missing Reproducibility

• Few papers release code (#04, #09, #22, #42, #48, #55, #56, #60, #61 are open-source)
• Run-to-run variance is rarely reported
• Environment setup instructions often insufficient

5. No Cost-Normalized Comparison

API costs vary dramatically ($0.68 per target in #03 to $96.20 total in #61). Without cost normalization, comparing system effectiveness is misleading.

6. Lack of Attack/Defense Benchmarks

#37 noted all benchmarks are Jeopardy-style (solve isolated challenges). No benchmark models attack vs. defense dynamics, multi-step attack chains, or realistic network topologies.

Recommendations for Ideal Evaluation

1. Adopt AutoPenBench + PentestEval as minimum baselines for system papers
2. Report cost, time, and success rate together - all three matter
3. Measure partial progress using milestones, not just binary success
4. Include active defense scenarios - at minimum, evaluate against CHeaT/Mantis-style defenses
5. Test on multi-host networks like GOAD or MHBench, not just isolated CTF challenges
6. Report variance across multiple runs (at least 3-5)
7. Use held-out test sets to avoid benchmark contamination from training data

Research Gaps & PhD Opportunities

Overview

After analyzing 65 papers on LLM-based penetration testing (2012-2026), this document synthesizes the major research gaps and proposes concrete PhD thesis directions. The field is in early adolescence: core capabilities are demonstrated but reliability, scalability, and real-world applicability remain far from solved.

Consolidated Research Gaps

Technical Gaps

T1. Post-Exploitation and Lateral Movement (Critical)

Gap: The overwhelming majority of systems stop after initial exploitation (getting a shell/flag). Only #06 AutoAttacker, #19 Cochise, #42 Hackphyr, and #60 Incalmo address post-exploitation or lateral movement. Evidence: #06 noted "no prior comprehensive study on LLM-driven post-breach attack automation"; #19 tested AD networks but achieved only ~1.83 accounts per run; #60 Incalmo succeeded in 37/40 multi-host scenarios but used high-level abstractions rather than LLM-driven lateral movement. Significance: Real penetration tests focus heavily on post-exploitation. Without this, LLM pentesting remains a toy problem.

T2. Multimodal Understanding (Critical)

Gap: No system effectively handles visual/GUI-based information. #53 HackWorld showed CUAs achieve below 12% on web vulnerabilities through visual interaction. Evidence: Noted as a gap by #01, #03, #04, #09, #10, #44. Modern web applications require browser interaction, CAPTCHA solving, and visual analysis that text-only agents cannot perform. Significance: Web application testing is the largest real-world pentesting market segment.

T3. Context Management Across Long Engagements (Critical)

Gap: Real penetration tests span days/weeks. Current systems operate within single sessions of minutes/hours. #48 CHAP is the only paper addressing cross-session context relay. Evidence: #01 noted context degradation; #05 showed retention drops to 57.1% on complex tasks; #09 found 42.36% of failures due to session context loss; #02 is the only system that maintains state across multi-day engagements. Significance: Without solving this, LLM agents are limited to "sprints" rather than realistic engagement timelines.

T4. Learning from Experience (Important)

Gap: Most systems don't accumulate knowledge across engagements. Each test starts from scratch. Evidence: #59 RefPentester's dual success/failure logs, #06 AutoAttacker's experience manager, and #13 RapidPen's success-case RAG are initial attempts. No system demonstrates cross-engagement transfer learning. Significance: Human pentesters improve over years of experience. Current LLM agents cannot.

T5. Defense-Aware Offensive Agents (Important)

Gap: No offensive system accounts for active defenders. #32 CHeaT achieved 100% defense success against PentestGPT; #33 Mantis demonstrated prompt injection can halt/reverse LLM attacks. Evidence: #19, #06, #07, #08 all note "evaluation limited to systems without active defenses" as a limitation. Significance: Any real deployment faces defenders. Current agents are trivially defeated by basic countermeasures.

T6. Binary Exploitation and Low-Level Attacks (Important)

Gap: LLMs fundamentally struggle with binary exploitation, heap manipulation, and low-level memory attacks. Evidence: #15 PwnGPT found LLMs cannot understand runtime memory state; #20 HackSynth found all LLMs completely fail at binary exploitation; #22 CAI showed weak performance in pwn (0.77x) and crypto (0.47x). Significance: Binary exploitation is a core pentesting skill that current LLM architectures may be fundamentally unsuited for.

T7. Reliable Tool Usage (Moderate)

Gap: LLMs frequently generate invalid commands with wrong parameters. Evidence: #19 Cochise found 35.9% invalid commands, with 94% hashcat failure rate; #09 VulnBot reported 19.70% failed tool errors; #42 Hackphyr noted high invalid action rates. Significance: Unreliable tool usage wastes time and can alert defenders.

Evaluation Gaps

E1. No Standardized Benchmark Suite (Critical)

Gap: Each paper uses different environments, metrics, and baselines. Cross-paper comparison is nearly impossible. Evidence: #37 Benchmarking Practices systematically documented this fragmentation. #29 AutoPenBench and #58 PentestEval are steps forward but not universally adopted. Significance: Without standardized evaluation, the field cannot measure progress.

E2. CTF ≠ Real Pentesting (Critical)

Gap: Most benchmarks use isolated CTF challenges or single-vulnerability Docker containers. These don't represent real network penetration tests. Evidence: Only #02, #19, and #60 evaluate on multi-host networks. #37 noted "multi-step attack chains in real-world networks are rarely modeled." Significance: Good CTF performance doesn't predict real-world pentesting ability.

E3. No Attack/Defense Evaluation (Important)

Gap: #37 noted all benchmarks are Jeopardy-style. No benchmark models adversarial dynamics. Evidence: #32 CHeaT and #33 Mantis are defensive papers that revealed offensive agent fragility, but no offensive paper tests against defenders. Significance: Adversarial evaluation is essential for understanding real-world deployability.

Application Domain Gaps

D1. Active Directory / Enterprise Network Testing (Critical)

Gap: Only #19 Cochise tests on AD networks. Enterprise networks with domain controllers, trust relationships, and GPOs are the primary target of real penetration tests. Evidence: #19 achieved limited results (1.83 accounts/run); #60 Incalmo tested multi-host but not AD-specific attacks. Significance: Enterprise AD pentesting is the highest-value commercial pentesting activity.

D2. Cloud and Container Security (Important)

Gap: No paper specifically addresses cloud-native pentesting (AWS/Azure/GCP misconfigurations, container escapes, serverless exploitation). Evidence: Noted as a gap by #10, #34. Cloud infrastructure is increasingly the attack surface. Significance: Growing market segment with unique challenges.

D3. API and Mobile Security Testing (Moderate)

Gap: No paper addresses API security testing or mobile application pentesting. Evidence: #02 notes "benchmark scope omits mobile security." Significance: API-first architectures and mobile apps represent a significant attack surface.

Safety & Ethics Gaps

S1. Dual-Use Risk Management (Critical)

Gap: The same tools used for defensive testing can be weaponized. No paper provides a satisfactory framework for managing this risk. Evidence: #23 noted "missing safeguards for distinguishing ethical from unethical use"; #34 raised "governance and guardrails" as a primary gap; #06 demonstrated 100% jailbreak success rate. Significance: This is both a technical and policy problem that affects publishability and deployment.

S2. Agent Containment and Safety (Important)

Gap: Autonomous agents executing commands on real systems can cause unintended damage. Evidence: #20 HackSynth identified that "current containment mechanisms are insufficient"; #45 noted agents with networking capabilities could cause unintentional DoS. Significance: Safety guarantees are prerequisite for real-world deployment.

Theoretical Gaps

Th1. Formal Model of Pentesting as an AI Problem (Important)

Gap: No consensus on the formal problem definition. Papers variously model pentesting as MDP, POMDP, Mealy machine, DAG, or ad-hoc. Evidence: #24 uses POMDP, #03 uses classical planning, #07 uses FSM, #36 uses labeled transition system. These formalizations are incompatible. Significance: A unified theoretical framework would enable principled comparison and combination of approaches.

Th2. When and Why LLMs Work for Pentesting (Moderate)

Gap: #23 hypothesizes pentesting is fundamentally pattern-matching, explaining LLM suitability, but this is untested. Evidence: #23 is the only paper that attempts to explain why LLMs work for pentesting. All others are empirical. Significance: Understanding the theoretical basis would guide architecture design.

---

Potential PhD Thesis Topics

⭐ Topic 1: Autonomous Multi-Host Network Penetration Testing with Persistent Memory

Research questions:

• How can LLM agents maintain and transfer knowledge across multi-day, multi-host penetration testing engagements?
• What abstractions enable efficient planning over large network topologies?
• How should agents balance exploration vs. exploitation in unfamiliar networks?

Why it matters: Addresses gaps T1, T3, T4, D1. Real pentests involve multi-host networks and span days. Only #48 CHAP and #60 Incalmo touch this.

Why existing work is insufficient: The 60+ papers in this survey overwhelmingly target single-host, single-session scenarios that bear little resemblance to real penetration testing engagements. #60 Incalmo demonstrates multi-host orchestration but relies on high-level plan abstractions rather than genuine LLM-driven lateral movement, and it lacks any persistent memory across sessions. #48 CHAP is the sole paper addressing cross-session context relay, but it focuses narrowly on context summarization without tackling multi-host coordination or knowledge transfer across engagements.

Builds on: #02 (EGATS), #19 (AD testing), #48 (context relay), #60 (planning-execution decoupling)

Differentiation: No existing work combines persistent cross-session memory with multi-host lateral movement and AD-specific attack knowledge. Specifically: #60 Incalmo handles multi-host but not persistent memory; #48 CHAP handles context relay but only for single-host CTF tasks; #19 Cochise tests on AD networks but achieves only 1.83 accounts per run with no cross-session learning; #02 EGATS maintains multi-day state but does not address lateral movement or network-wide planning. This topic uniquely integrates all four dimensions.

Feasibility: High. GOAD provides free AD labs; context relay techniques exist; multi-agent frameworks are mature.

---

Topic 2: Defense-Aware Offensive AI — Adversarial Pentesting Against Active Defenders

Research questions:

• How can LLM agents detect and evade honeypots, IDS/IPS, and deception-based defenses?
• What is the equilibrium between LLM-based attackers and LLM-based defenders?
• Can agents learn stealth strategies through adversarial training?

Why it matters: Addresses gaps T5, E3. No offensive paper accounts for defenders. #32 CHeaT showed current agents are trivially defeated.

Why existing work is insufficient: The entire offensive literature operates in a defender-free vacuum. #32 CHeaT demonstrated that a simple LLM-based defensive honeypot achieves 100% success at stopping PentestGPT, and #33 Mantis showed that prompt injection can completely halt or even reverse LLM-based attacks. Despite these alarming findings, no subsequent offensive paper has incorporated adversarial robustness into its design or evaluation.

Builds on: #32 (CHeaT defenses), #33 (Mantis), #50 (Honeypot detection), #02 (adversarial robustness gap)

Differentiation: First work to study the attacker-defender co-evolution in LLM-based pentesting. Unlike #32 CHeaT (defense-only perspective) and #33 Mantis (prompt injection defense), this topic would develop offensive agents that actively model and adapt to defensive countermeasures, creating a game-theoretic framework for studying equilibrium rather than one-sided evaluation.

Feasibility: Medium. Requires building both offensive and defensive agents. CHeaT provides a starting defensive framework.

---

Topic 3: Multimodal Penetration Testing Agents for Web Applications

Research questions:

• Can vision-language models perform GUI-based web application testing (clicking, form filling, visual analysis)?
• How should agents combine visual understanding with traditional tool-based testing?
• What benchmark adequately evaluates multimodal web pentesting?

Why it matters: Addresses gaps T2, D3. #53 HackWorld showed CUAs achieve <12%. Web apps are the largest pentesting market.

Why existing work is insufficient: #53 HackWorld is the only paper evaluating vision-based interaction for web pentesting, and its results are discouraging (below 12% success). All other web pentesting systems (#07, #56) rely entirely on text-based tool output, ignoring the visual complexity of modern web applications. No paper has attempted a hybrid approach that combines visual understanding with traditional command-line tools, which is how human pentesters actually work.

Builds on: #53 (HackWorld benchmark), #07 (web pentesting), #56 (interactive tools)

Differentiation: No existing system combines vision-language models with traditional pentesting tools for web application security. #53 HackWorld evaluates pure CUA approaches but does not propose a hybrid architecture; #07 tests web applications but only through text-based interactions; #56 provides interactive tool environments but no visual capabilities. This topic would pioneer the multimodal fusion approach.

Feasibility: Medium. VLMs are rapidly improving. HackWorld provides an initial benchmark.

---

⭐ Topic 4: Deep Reinforcement Learning Meets LLMs for Autonomous Penetration Testing

Research questions:

• Can DRL policies trained on network simulation transfer to real-world pentesting scenarios when combined with LLM reasoning?
• What reward functions and curriculum designs are effective for multi-stage pentesting?
• How can DRL address the reward sparsity problem in complex network environments while LLMs handle natural-language tool interaction?

Why it matters: Addresses gaps T4, Th1. #10 Pentest-R1 and #45 showed RL potential but only on CTF tasks. No work integrates DRL strategic planning with LLM tactical execution.

Why existing work is insufficient: The DRL-for-pentesting line of work (#45, #27 CyBORG) and the LLM-for-pentesting line of work (#01, #02, #60) have developed in complete isolation from each other. DRL approaches excel at strategic planning over network topologies but cannot handle natural-language tool interaction or adapt to novel vulnerability descriptions. LLM approaches excel at understanding and executing individual attack steps but lack the ability to learn optimal strategies through experience. No paper has attempted to bridge these two paradigms, leaving a completely unexplored design space.

Builds on: #10 (two-stage RL), #45 (curriculum RL), #54 (trajectory synthesis), #24 (POMDP formalization), #27 (CyBORG gym)

Differentiation: First to integrate DRL strategic planning with LLM tactical execution for pentesting. Unlike #10 Pentest-R1 (RL fine-tuning of LLM weights on CTF tasks), this topic uses DRL as a separate strategic planning layer that guides LLM-based execution agents, enabling the system to learn optimal network traversal policies while leveraging LLM flexibility for individual attack steps. Unlike #45 (pure RL in simulation), this bridges simulation-trained policies to real-world environments via LLM grounding. Unlike #24 (POMDP formalization without learning), this provides a concrete learning algorithm.

Feasibility: Medium-High. CyBORG and GOAD provide training environments. Pentest-R1 provides RL methodology. The DRL and LLM communities have separately matured the building blocks.

---

Topic 5: Efficient Small-Model Architectures for Pentesting Agents

Research questions:

• What architectural abstractions allow small LLMs (7B-32B) to match or exceed large models for pentesting?
• How should tool interfaces, planning modules, and memory be designed to compensate for model capacity?
• What fine-tuning strategies are most effective for pentesting specialization?

Why it matters: Addresses gap T6 (indirectly), builds on the finding from #04 and #60 that abstractions > model size. Enables private, offline deployment.

Why existing work is insufficient: Individual papers have demonstrated promising results with smaller models (#04 fine-tuned 32B beating 405B, #60 Incalmo achieving strong results with Haiku, #42 Hackphyr fine-tuning), but each operates as an isolated experiment on a different benchmark with different architecture choices. There is no systematic study of which architectural components (tool abstraction layers, planning modules, memory systems, fine-tuning data composition) contribute most to closing the gap between small and large models.

Builds on: #04 (fine-tuned 32B beats 405B), #42 (Hackphyr fine-tuning), #54 (trajectory synthesis), #60 (Incalmo with Haiku)

Differentiation: Systematic study of the architecture-model size tradeoff, going beyond single fine-tuning experiments. Unlike #04 (single model, single benchmark), this would conduct controlled ablation studies across multiple architectures, model sizes, and benchmarks. Unlike #42 Hackphyr (fine-tuning only), this includes non-parametric components (planning modules, tool abstraction, external memory). Unlike #60 Incalmo (fixed architecture, model comparison), this optimizes the full architecture for small-model constraints.

Feasibility: High. Open-source models, existing fine-tuning pipelines, existing benchmarks.

---

Topic 6: Unified Benchmark Suite for LLM-Based Penetration Testing

Research questions:

• What evaluation dimensions are necessary and sufficient for comparing pentesting agents?
• How should benchmarks model multi-step attack chains, active defenders, and realistic network topologies?
• Can automated difficulty calibration replace expert manual grading?

Why it matters: Addresses gaps E1, E2, E3. The field desperately needs standardized evaluation.

Why existing work is insufficient: #37 Benchmarking Practices documented severe fragmentation: no two papers use the same benchmark, metrics, or evaluation methodology. #29 AutoPenBench and #58 PentestEval are steps forward but each covers only a narrow slice (AutoPenBench focuses on single-host Docker tasks, PentestEval on knowledge assessment). No benchmark includes multi-host network scenarios, active defenders, or difficulty-calibrated progressive challenge sets that would enable meaningful cross-system comparison.

Builds on: #29 (AutoPenBench), #58 (PentestEval), #37 (benchmarking practices), #63 (Cybench), #53 (HackWorld)

Differentiation: First unified benchmark combining CTF challenges, multi-host networks, active defenses, and standardized metrics. Unlike #29 AutoPenBench (single-host Docker only), this includes network-level scenarios. Unlike #58 PentestEval (knowledge-focused), this measures end-to-end operational capability. Unlike #37 (meta-analysis only), this delivers a concrete benchmark artifact. Unlike #63 Cybench (CTF-only), this models realistic enterprise environments.

Feasibility: High. Docker-based environments exist. Challenge is coordination and community adoption.

---

Topic 7: Neurosymbolic Pentesting — Combining LLMs with Formal Planning

Research questions:

• How can classical planning, knowledge graphs, and LLMs be optimally combined for pentesting?
• Can formal methods provide safety guarantees for autonomous pentesting agents?
• How should the action/state space be defined for pentesting as a planning problem?

Why it matters: Addresses gaps Th1, T7. #03 CHECKMATE showed classical planning improves consistency (100% vs 75%); #31 MulVAL showed Datalog-based reasoning works for attack graphs.

Why existing work is insufficient: Papers have explored individual formal approaches in isolation: #03 uses classical PDDL planning, #24 models pentesting as a POMDP, #07 uses finite state machines, #36 uses labeled transition systems, and #31 uses Datalog-based attack graphs. These formalizations are mutually incompatible and none has been systematically combined with LLM capabilities. #03 CHECKMATE comes closest but uses a rigid planning layer that cannot adapt to unexpected findings during execution.

Builds on: #03 (Classical Planning+), #24 (POMDP), #31 (MulVAL Datalog), #36 (formal transition system), #49 (game-theoretic)

Differentiation: Systematic framework combining symbolic reasoning for planning with LLMs for execution, providing formal guarantees. Unlike #03 CHECKMATE (fixed PDDL domain), this dynamically constructs and updates the planning domain from LLM observations. Unlike #24 (POMDP formalization without implementation), this provides a working system. Unlike #31 MulVAL (static attack graph analysis), this supports online replanning during active engagements.

Feasibility: Medium. Classical planning and knowledge graphs are well-understood. Integration with LLMs is the research challenge.

---

⭐ Topic 8: Cross-Engagement Learning and Knowledge Transfer for Pentesting Agents

Research questions:

• How can pentesting agents accumulate and transfer tactical knowledge across independent engagements?
• What knowledge representation enables generalization from past exploits to novel but structurally similar targets?
• How should success and failure experiences be encoded, retrieved, and applied in new contexts?

Why it matters: Addresses gaps T3, T4. Human pentesters improve dramatically over years of experience, but current LLM agents start every engagement from scratch. Only #48 CHAP addresses cross-session context relay, and only 3 papers (#06, #13, #59) attempt any form of experience accumulation.

Why existing work is insufficient: #59 RefPentester maintains dual success/failure logs but only within a single engagement without cross-engagement transfer. #06 AutoAttacker's experience manager stores past actions but lacks structured generalization. #13 RapidPen's success-case RAG retrieves past solutions but cannot learn from failures or adapt strategies over time. #48 CHAP relays context across sessions but focuses on summarization rather than learning. None of these approaches demonstrates genuine transfer learning where performance on engagement N+1 measurably improves from experience on engagements 1 through N.

Builds on: #48 (CHAP context relay), #59 (RefPentester dual logs), #06 (experience manager), #13 (success-case RAG), #54 (trajectory synthesis)

Differentiation: First systematic study of cross-engagement transfer learning for pentesting agents. Unlike #59 RefPentester (within-engagement logs only), this builds a persistent knowledge base that grows across engagements. Unlike #06 AutoAttacker (raw action replay), this uses structured knowledge representations that support analogical reasoning. Unlike #13 RapidPen (success-only retrieval), this learns equally from failures. Unlike #48 CHAP (context summarization), this extracts generalizable tactical patterns rather than session-specific state.

Feasibility: High. RAG infrastructure is mature; trajectory datasets exist (#54); evaluation can be designed around measuring improvement curves across sequential engagements on diverse targets.

---

Recommended Research Agenda

The three starred topics above form a coherent, mutually reinforcing research program suitable for a PhD proposal. They address the three largest gaps in the field and build naturally on each other.

Foundation: Cross-Engagement Learning (Topic 8) serves as the methodological foundation. A pentesting agent that cannot learn from experience will always be limited to the knowledge frozen in its training data. By developing structured knowledge representations and transfer learning mechanisms, this work creates the substrate on which the other two directions depend. The key deliverable is an experience accumulation framework that demonstrably improves agent performance across sequential engagements.

Application: Multi-Host Network Pentesting (Topic 1) provides the real-world problem setting where cross-engagement learning becomes essential. Single-host CTF challenges are too simple to require persistent memory or accumulated experience, but multi-host network engagements spanning multiple sessions demand both. Topic 1 builds directly on Topic 8's knowledge transfer framework, applying it to the most commercially relevant and technically challenging pentesting scenario. The persistent memory architecture developed in Topic 8 becomes the backbone for maintaining state across multi-day, multi-host campaigns.

Scaling: DRL+LLM Integration (Topic 4) addresses the strategic planning layer that neither pure LLM approaches nor pure experience replay can provide. Once an agent has accumulated experience (Topic 8) across network-scale engagements (Topic 1), DRL provides the mechanism to distill that experience into optimal network traversal and exploitation policies. The DRL component learns macro-level strategy (which hosts to target, in what order, with what techniques) while the LLM component handles micro-level execution (interpreting tool output, crafting payloads, adapting to unexpected responses). Together, these three topics trace a path from "LLM agents that solve isolated CTF challenges" to "learning systems that conduct realistic, multi-session network penetration tests with improving performance over time."

Suggested sequencing: Begin with Topic 8 (Year 1-2), which produces immediately publishable results on experience accumulation and has the highest feasibility. Extend to Topic 1 (Year 2-3), applying the learning framework to multi-host scenarios. Integrate Topic 4 (Year 3-4) as the strategic planning layer that ties everything together. Each stage produces independent publications while building toward the unified thesis.

---

Field Maturity Assessment

Dimension	Maturity	Evidence
Initial access / single-host exploitation	Growing	Multiple systems achieve 40-90% on CTF/VulnHub
Multi-host / real network testing	Nascent	Only 3-4 papers attempt this
Post-exploitation / lateral movement	Embryonic	Almost no work beyond #06, #19, #60
Evaluation standardization	Nascent	Benchmarks emerging but not adopted
Theoretical foundations	Embryonic	Multiple incompatible formalizations
Defense interaction	Embryonic	Defensive papers exist but no offensive paper accounts for them
Real-world deployment	Pre-nascent	#02 is the closest with live HTB ranking

Overall: The field is in early growth phase. Core capabilities are demonstrated but the gap between CTF-solving and real-world pentesting remains vast. This makes it an excellent time for PhD research — the fundamental problems are identified but unsolved.

Cross-Cutting Themes

1. Abstractions beat model size: Multiple papers (#04, #60) show that good architectural design matters more than using the largest model.
2. Exploitation is the bottleneck: Across all papers, exploitation phase has the lowest success rates. Recon and scanning are largely solved.
3. Context management is universal: Every system struggles with maintaining context over long interactions.
4. RAG is necessary but insufficient: RAG helps but retrieval quality is poor and can mislead agents.
5. Open-source models are catching up: Fine-tuned open-source models (#04, #10, #54) increasingly competitive with GPT-4.

阶段一：统一基准套件（Year 1-2）    → 评测基础设施
       ↓ 为阶段二提供标准化评测平台
阶段二：理想环境多主机渗透（Year 2-3） → 系统能力验证
       ↓ 为阶段三提供攻击方基线
阶段三：对抗环境渗透测试（Year 3-4）  → 真实场景研究