AwesomeREA.org

2026 · Conference paper

Cyber-Zero: Training Cybersecurity Agents without Runtime

Terry Yue Zhuo; Dingmin Wang; Hantian Ding; Varun Kumar; Zijian Wang

Cyber-Zero synthesizes training trajectories for capture-the-flag (CTF) agents from public writeups instead of running the challenges. Writeups collected from CTFtime through the CTF Archives repository are cleaned, filtered and completed with DeepSeek-V3-0324, which fills in missing metadata and extracts flags; after removing 47 challenges that overlap the evaluation benchmarks, the corpus holds 6,188 writeups for 4,610 challenges from 543 competitions in six categories, 2017–2025. Two DeepSeek-V3-0324 personas then simulate each solve: a CTF player that sees only the challenge content, and a Bash terminal that sees the writeup and flag, invents plausible command output and may inject hints, which are removed before training. Trajectories use the EnIGMA scaffold format and are kept only if they reach the exact flag, pass format checks and are judged by an LLM filter to follow the writeup's approach. Qwen3, Qwen2.5-Instruct and SWE-agent-LM models are fine-tuned on 9,464 of these trajectories and evaluated with EnIGMA+, the authors' parallel version of EnIGMA, on InterCode-CTF, NYU CTF Bench and Cybench. The fine-tuned Qwen3-32B (Cyber-Zero-32B) rises from 20.3% to 33.4% average Pass@1, the 13.1-point gain reported in the abstract.

Scope and limitations

No training trajectory was executed: command outputs are simulated by an LLM that knows the writeup and flag, and trajectories are filtered by exact flag match, format checks and an LLM judge rather than by running commands. The authors acknowledge that the corpus may contain incorrect command traces and fabricated flags, that they report no quantitative error analysis, and that the simulated traces exclude interactive tools such as gdb, Python shells and browsers. The writeups and the evaluation tasks span six general CTF categories, not reverse engineering alone; the three test benchmarks hold 323 runnable tasks, 84 of them reverse engineering, and aggregate Pass@1 is not reverse-engineering accuracy. Results are single greedy Pass@1 runs in the authors' EnIGMA+ scaffold with a 40-turn cap, on benchmark copies the authors repaired; 17 tasks could not be set up. Contamination control removed 47 overlapping challenges by manual comparison. Capturing a flag does not show recovered program structure. The repository is archived and licensed CC BY-NC 4.0, and its benchmark README states that the bundled benchmark content remains the property of its original creators.

Why it fits this collection

Adds runtime-free trajectory synthesis for training open CTF agents whose tasks include reverse engineering; it formats its data for EnIGMA, also in this section, and evaluates on InterCode-CTF, NYU CTF Bench and Cybench.

Publication and review details

Venue
ICLR 2026
Status
Conference paper
Publication / posting date
2026 (exact date not verified)
Date source
Publication year stated by the official ICLR proceedings; no finer publication date is asserted. The proceedings page's citation date of 2026-04-20 is not adopted, and the arXiv dates are not substituted.
First arXiv submission
July 29, 2025
arXiv
2508.00910
Reviewed arXiv version
v2

The proceedings page, its PDF and arXiv list the same five authors in order. The proceedings page lists pages 154054–154184 and no DOI. Its 131-page PDF is headed 'Published as a conference paper at ICLR 2026'; compared with arXiv v2 it adds an appendix of limitations, describes the 47-challenge contamination removal and moves Zijian Wang's affiliation to Meta Superintelligence Labs. Its abstract says Cyber-Zero-32B is comparable to frontier open-weight models, while arXiv v2 and the proceedings page abstract claim state-of-the-art results among open-weight models. arXiv v1 is dated 2025-07-29 and v2 2025-08-25; v2's comment links the repository. The OpenReview forum and API could not be retrieved past a browser check, so no presentation type is recorded. At the reviewed 2025-09-02 commit the GitHub repository is archived; its LICENSE is the CC BY-NC 4.0 legal code (GitHub reports NOASSERTION) and its NOTICE gives SPDX CC-BY-NC-4.0. It contains the generation, quality-filter and reformatting scripts (Python 3.8 or later, LiteLLM, with DeepSeek-V3-0324 as the default model, so runs need a model provider), the EnIGMA+ scaffold and repaired copies of the three benchmarks. The README links no released trajectory corpus or model weights. No code or experiments were run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Report an outdated entry Suggest a translation fix

Primary sources

  1. https://proceedings.iclr.cc/paper_files/paper/2026/hash/f9f54762cbb4fe4dbffdd4f792c31221-Abstract-Conference.html
  2. https://proceedings.iclr.cc/paper_files/paper/2026/file/f9f54762cbb4fe4dbffdd4f792c31221-Paper-Conference.pdf
  3. https://arxiv.org/abs/2508.00910
  4. https://arxiv.org/pdf/2508.00910v2
  5. README.md GitHub · amazon-science/Cyber-Zero · Revision e0c4493
  6. LICENSE GitHub · amazon-science/Cyber-Zero · Revision e0c4493
  7. NOTICE GitHub · amazon-science/Cyber-Zero · Revision e0c4493
  8. requirements.txt GitHub · amazon-science/Cyber-Zero · Revision e0c4493
  9. benchmarks/README.md GitHub · amazon-science/Cyber-Zero · Revision e0c4493