AwesomeREA.org

2025 · Conference paper

Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

Andy K. Zhang; Neil Perry; Riya Dulepet; Joey Ji; Celeste Menders; Justin W. Lin; Eliot Jones; Gashon Hussein; Samantha Liu; Donovan Jasper; Pura Peetathawatchai; Ari Glenn; Vikram Sivashankar; Daniel Zamoshchin; Leo Glikbarg; Derek Askaryar; Mike Yang; Teddy Zhang; Rishi Alluri; Nathan Tran; Rinnara Sangpisit; Polycarpos Yiorkadjis; Kenny Osele; Gautham Raghupathi; Dan Boneh; Daniel E. Ho; Percy Liang

Cybench turns 40 professional-level capture-the-flag tasks from four competitions (HackTheBox Cyber Apocalypse 2024, SekaiCTF 2022–23, GlacierCTF and HKCert) into runnable agent tasks. Each task has a description, starter files, a Kali Linux container for the agent with task servers in Docker, and the time the first human team needed to solve it, from 2 minutes to 24 hours 54 minutes. Each task is also split into subtasks that grade intermediate steps. The authors evaluate eight models with a structured bash agent and compare four scaffolds for GPT-4o and Claude 3.5 Sonnet. Without subtask guidance, the best agents solved only tasks whose first human solve took at most 11 minutes, and Claude 3.5 Sonnet reached 17.5% unguided success.

Scope and limitations

Only 6 of the 40 tasks are reverse engineering; the rest are cryptography (16), web (8), forensics (4), miscellaneous (4) and pwn (2), so aggregate rates are not reverse-engineering accuracy, and Cybench is not a reverse-engineering-only benchmark. In the unguided structured-bash runs, six or seven of the eight models solved the two easiest reversing tasks (first human solves in 2 and 6 minutes), and none solved the other four. Results cover 2024 models (GPT-4o, o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3 and 3.1, Mixtral 8x22b) with the authors' scaffolds: 15 iterations unguided or 5 per subtask, short memory, no decompiler tools, one attempt in the main comparison and the best of three in the scaffold comparison. First-solve times come from competition records and are not directly comparable across competitions. The authors judge train–test overlap by model knowledge cutoffs and cannot rule it out for Claude 3.5 Sonnet. CTF tasks are short, small-codebase competition problems, and capturing a flag does not show recovered program structure.

Why it fits this collection

Adds a CTF agent benchmark with reverse-engineering tasks and subtask-level grading; EnIGMA in this section reports results on it, and it complements the reverse-engineering-specific benchmarks here.

Publication and review details

Venue
ICLR 2025
Status
Conference paper
Publication / posting date
2025 (exact date not verified)
Date source
Publication year stated by the official ICLR proceedings; no finer publication date is asserted. The proceedings page's citation date of 2025-05-01 is not adopted, and the arXiv dates are not substituted.
First arXiv submission
August 15, 2024
arXiv
2408.08926
Reviewed arXiv version
v4

Authors follow the byline of the proceedings PDF, which matches arXiv v4. The proceedings page metadata instead lists 25 names: it omits Polycarpos Yiorkadjis and Gautham Raghupathi, gives Haoxiang Yang, Aolin Zhang and Kenny Oseleononmen where the PDF prints Mike Yang, Teddy Zhang and Kenny Osele, and drops middle initials. The proceedings list pages 25094–25243 and no DOI. arXiv v1 is dated 2024-08-15, v2 2024-10-06, v3 2024-12-05 and v4 2025-04-12; v4 carries the comment 'ICLR 2025 Oral'. The OpenReview forum could not be retrieved past a browser check, so the oral designation rests on that comment. The proceedings abstract links anonymized Google Drive archives, while the PDF points to cybench.github.io, which links andyzorigin/cybench. At the reviewed 2026-09-24 commit the repository is Apache-2.0; runs need model API keys, Python 3.9–3.10 and Docker with the cybench/cybench image. Its task_list.txt names all 40 tasks, but the nine GlacierCTF tasks, including the reversing tasks RPGO and SOP, are not in the repository tree. No code or experiments were run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Report an outdated entry Suggest a translation fix

Primary sources

  1. https://proceedings.iclr.cc/paper_files/paper/2025/hash/3e9412a9c1d93810ef3ef7825115016b-Abstract-Conference.html
  2. https://proceedings.iclr.cc/paper_files/paper/2025/file/3e9412a9c1d93810ef3ef7825115016b-Paper-Conference.pdf
  3. https://arxiv.org/abs/2408.08926
  4. https://arxiv.org/pdf/2408.08926v4
  5. https://cybench.github.io/
  6. README.md GitHub · andyzorigin/cybench · Revision ca483c6
  7. LICENSE GitHub · andyzorigin/cybench · Revision ca483c6
  8. task_list.txt GitHub · andyzorigin/cybench · Revision ca483c6
  9. benchmark GitHub · andyzorigin/cybench · Revision ca483c6
  10. run_task.sh GitHub · andyzorigin/cybench · Revision ca483c6