An Empirical Survey of Functions and Configurations of Open-Source Capture the Flag (CTF) Environments An Empirical Survey of Functions and Configurations of Open-Source Capture the Flag (CTF) Environments

Problem & Motivation 问题与动机

There is no systematic assessment of the configuration and setup of virtual environments for live security competitions such as CTFs. While many CTF competitions are assessed regarding learning obstacles, participant views, and evaluation theory, only single instances of CTF software environments are described in prior work.

目前缺乏对 CTF 等现场安全竞赛虚拟环境的配置和设置的系统性评估。虽然许多 CTF 竞赛已在学习障碍、参与者观点和评估理论方面进行了评估，但在之前的工作中，仅描述了 CTF 软件环境的单个实例。

CTF competitions have been growing in popularity as a tool for cybersecurity education and training, yet no survey on CTF environments, their features, and challenge configuration options had been published. This gap limits CTF organizers, developers, and participants who need to compare and select platforms.

CTF 竞赛作为网络安全教育和培训工具正变得越来越流行，但尚未发表过关于 CTF 环境、其特征和挑战配置选项的调查。这一空白限制了需要比较和选择平台的 CTF 组织者、开发人员和参与者。

Methodology 核心方法

The authors conducted a 5-step empirical review: (1) review of existing CTF environments and software, identifying 28 platforms; (2) selection of 12 open-source platforms; (3) installation and testing of 8 successfully installable platforms; (4) development and integration of 16 challenge examples (7 representative ones across quiz, jeopardy, and king-of-the-hill types) into each platform; (5) systematic analysis and evaluation of supported features, challenge types, and game configuration options.

作者进行了五个步骤的实证审查：(1) 审查现有的 CTF 环境和软件，识别出 28 个平台；(2) 选择 12 个开源平台；(3) 安装并测试 8 个成功安装的平台；(4) 开发 16 个挑战示例（涵盖测验、解谜和攻防等类型的 7 个代表性示例）并集成到每个平台中；(5) 系统地分析和评估支持的功能、挑战类型和游戏配置选项。

Architecture 架构设计

The survey identifies a common architectural abstraction for CTF platforms consisting of layers: hardware, operating system, software container (e.g., Docker, Vagrant) and/or virtualization provider (e.g., VirtualBox, VMware), with the CTF platform and its challenges on top.

该调查识别了 CTF 平台的通用架构抽象，由以下各层组成：硬件、操作系统、软件容器（如 Docker、Vagrant）和/或虚拟化提供商（如 VirtualBox、VMware），最顶层是 CTF 平台及其挑战。

Memory Mechanism 记忆机制

none

Evaluation 评估结果

All 8 platforms support quiz and jeopardy challenge types, but only FBCTF and RootTheBox support king-of-the-hill challenges. The platforms share 9 common configuration options (description, visibility, flag, hints, points, title, upload, upload-size-limit) but vary significantly in 7 additional options (case sensitivity, dependencies between challenges, hint penalty, limited attempts, multiple flags, time frame for solving, waiting time between re-submissions). Game configurations such as hint penalties, time frames, attempt limits, and challenge dependencies vary strongly between platforms.

所有 8 个平台都支持测验（quiz）和解谜（jeopardy）类型的挑战，但只有 FBCTF 和 RootTheBox 支持攻防（king-of-the-hill）类型的挑战。这些平台共有 9 个通用的配置选项（描述、可见性、flag、提示、分数、标题、上传、上传大小限制），但在 7 个附加选项（大小写敏感性、挑战间的依赖关系、提示惩罚、尝试次数限制、多个 flag、解决时限、重新提交间的等待时间）上存在显著差异。提示惩罚、时限、尝试限制和挑战依赖等游戏配置在不同平台之间差异很大。

Environment 评估环境

Metrics 评估指标

Scale 评估规模

8 open-source CTF platforms evaluated with 7 representative challenge examples (16 total developed)

Contributions 核心贡献

• First systematic survey comparing features, challenge types, and game configuration options across open-source CTF platforms
• Identification and categorization of 28 CTF platforms, with in-depth analysis of 8 open-source ones (PicoCTF, FacebookCTF, HackTheArch, WrathCTF, Pedagogic-CTF, RootTheBox, CTFd, Mellivora)
• Development of 16 challenge examples across quiz, jeopardy, and king-of-the-hill types to systematically test platform capabilities
• Identification of 9 common and 7 additional game configuration options across CTF platforms
• Classification of CTF delivery models (hosting service, live competition, online competition, online training, local installation)

• 首个系统调查，比较了开源 CTF 平台的功能、挑战类型和游戏配置选项
• 识别并分类了 28 个 CTF 平台，并对其中 8 个开源平台（PicoCTF, FacebookCTF, HackTheArch, WrathCTF, Pedagogic-CTF, RootTheBox, CTFd, Mellivora）进行了深入分析
• 开发了 16 个涵盖测验、解谜和攻防类型的挑战示例，以系统地测试平台能力
• 识别了 CTF 平台的 9 个通用和 7 个附加游戏配置选项
• 对 CTF 交付模型进行了分类（托管服务、现场竞赛、在线竞赛、在线培训、本地安装）

Limitations 局限性

• Only open-source CTF platforms were analyzed; closed-source and commercial platforms were excluded
• Out of 12 open-source platforms found, 4 could not be installed due to maintenance issues or insufficient documentation (iCTF, NightShade, OCCP, OpenCTF)
• The survey does not compare open-source versus closed-source platforms
• Challenge examples developed were simple and limited to beginner/intermediate level cybersecurity
• Research was conducted between August and December 2017, so platforms may have evolved since then
• The survey does not assess the semantic aspects (contents and relations) of CTF competitions

• 仅分析了开源 CTF 平台；排除了闭源和商业平台
• 在发现的 12 个开源平台中，有 4 个因维护问题或文档不足而无法安装（iCTF, NightShade, OCCP, OpenCTF）
• 该调查未比较开源与闭源平台
• 开发的挑战示例比较简单，仅限于初级/中级水平的网络安全
• 研究于 2017 年 8 月至 12 月期间进行，因此平台自那时以来可能已经发生了演变
• 该调查未评估 CTF 竞赛的语义方面（内容和关系）

Research Gaps 研究空白

• No standard interface or format exists for export/import of CTF data between platforms, hindering interoperability
• Only FBCTF supports multiple languages; internationalization is largely missing from CTF platforms
• King-of-the-hill challenge support is limited to only 2 out of 8 platforms, indicating a gap in attack-defense CTF infrastructure
• A golden standard for CTF environments incorporating both open-source and commercial platforms is needed
• Dependencies between challenges are supported by only 3 platforms, limiting structured difficulty progression

• 不存在用于在平台之间导出/导入 CTF 数据的标准接口或格式，阻碍了互操作性
• 只有 FBCTF 支持多种语言；CTF 平台在很大程度上缺乏国际化
• 攻防类型挑战的支持仅限于 8 个平台中的 2 个，表明攻防 CTF 基础设施存在空白
• 需要一个结合了开源和商业平台的 CTF 环境黄金标准
• 只有 3 个平台支持挑战之间的依赖关系，限制了结构化的难度进阶

Novel Techniques 新颖技术

• Systematic categorization of CTF platform features into pre-CTF, in-CTF, and post-CTF activity phases
• Layered architectural abstraction model for CTF environments (hardware, OS, container/virtualization, platform, challenges)
• Taxonomy of CTF game types: Quiz, Jeopardy, Attack-Defense, Mixtures, King of the Hill

• 将 CTF 平台功能系统地分类为 CTF 前、CTF 中和 CTF 后 activity 阶段
• CTF 环境的分层架构抽象模型（硬件、操作系统、容器/虚拟化、平台、挑战）
• CTF 游戏类型的分类：测验（Quiz）、解谜（Jeopardy）、攻防（Attack-Defense）、混合型、攻防（King of the Hill）

Open Questions 开放问题

• How can CTF platforms be standardized to enable interoperability and data migration?
• What additional features would be needed to support more complex attack-defense and king-of-the-hill scenarios?
• How do commercial CTF platforms compare to open-source ones in terms of features and configurations?

• 如何标准化 CTF 平台以实现互操作性和数据迁移？
• 支持更复杂的攻防和 king-of-the-hill 场景需要哪些额外功能？
• 商业 CTF 平台在功能和配置方面与开源平台相比如何？

Builds On 基于前人工作

• iCTF
• DEFCON CTF
• NYU-CSAW
• MITRE CTF

Open Source 开源信息

No

Tags