基准测试全景 Benchmark Landscape

The capacity of LLMs to solve Capture-the-Flag (CTF) challenges in cybersecurity has not been thoroughly evaluated. Existing studies are limited in scope, use closed benchmarks, lack automated frameworks, and test only a small number of challenges or models.

LLM 在解决网络安全中的夺旗赛（CTF）挑战方面的能力尚未得到透彻评估。现有研究范围有限、使用闭源基准、缺乏自动化框架，且仅测试了少量的挑战或模型。

Despite growing interest in automating penetration testing with LLM-based generative agents, there is no comprehensive and standardized benchmark framework for evaluating, comparing, and developing such agents.

尽管人们对使用基于LLM的生成式智能体自动化渗透测试的兴趣日益浓厚，但目前还缺乏一个用于评估、比较和开发此类智能体的全面且标准化的基准测试框架。

There is a scarcity of comprehensive benchmarks for evaluating LLMs in the Vulnerability Assessment and Penetration Testing (VAPT) domain, especially for small, open-source models suitable for local deployment. Existing general LLM benchmarks and even emerging cybersecurity-specific benchmarks do not adequately address the specific requirements of VAPT or the particular considerations for smaller, locally deployable models.

在漏洞评估和渗透测试（VAPT）领域，缺乏全面的基准来评估大语言模型（LLM），尤其是适用于本地部署的小型开源模型。现有的通用 LLM 基准和新兴的网络安全基准都未能充分满足 VAPT 的特定需求，也未考虑小型本地可部署模型的特殊要求。

There is no comprehensive, open, end-to-end penetration testing benchmark to evaluate and drive progress of LLM-based automated penetration testing. Existing tools like PentestGPT rely heavily on human participation, and the degree of human involvement and the specific challenges LLMs face at each pentest stage are not well understood.

目前缺乏一个全面的、开放的、端到端的渗透测试基准测试，用于评估和推动基于LLM的自动化渗透测试的进展。现有的工具（如PentestGPT）严重依赖人工参与，而LLM在渗透测试各阶段的人工参与程度以及面临的具体挑战尚不明确。

Existing LLM-based offensive agent evaluations operate in closed-world settings with predefined goals and binary success criteria, using isolated single-service environments that fail to capture realistic multi-target attack scenarios involving reconnaissance, target selection, and exploitation under uncertainty.

现有的基于大语言模型（LLM）的攻防智能体评估通常在闭循环设置中进行，具有预定义的目标和二进制成功标准，且使用隔离的单一服务环境。这些方法未能捕捉到真实世界中涉及侦察、目标选择以及在不确定性下进行利用的多目标攻击场景。

While computer-use agents (CUAs) have shown strong capabilities in web browsing and visual task automation, their potential to discover and exploit web application vulnerabilities through graphical interfaces remains unknown. Existing benchmarks for CUAs assume sanitized environments and overlook security aspects.

虽然计算机使用智能体（CUA）在网页浏览和视觉任务自动化方面表现出强大能力，但它们通过图形界面发现和利用 Web 应用程序漏洞的潜力尚不清楚。现有的 CUA 基准测试通常假设环境是经过清理的，忽略了安全方面的问题。

Existing LLM-based penetration testing approaches rely on simplistic prompting without task decomposition or domain adaptation, resulting in unreliable black-box behavior and limited insight into model capabilities across individual penetration testing stages.

现有的基于 LLM 的渗透测试方法依赖于简单的提示词，缺乏任务分解或领域自适应，导致黑盒行为不可靠，且无法深入了解模型在各个渗透测试阶段的能力。

There is a need to quantify the cybersecurity capabilities and risks of language model agents, but existing benchmarks use easy, non-professional-level tasks, are not open-source, or lack objective difficulty grounding.

需要量化语言模型代理的网络安全能力和风险，但现有基准测试使用的是简单的、非专业级别的任务，且不是开源的，或者缺乏客观的难度依据。