2024 · Conference paper
NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
NYU CTF Bench packages 200 manually validated challenges from CSAW CTF qualifying and final rounds held between 2017 and 2023, spanning cryptography, forensics, pwn, reverse engineering, web and miscellaneous tasks, with challenge.json metadata and Docker images for server-side challenges. The paper also presents an automated framework that gives LLMs function-calling tools for shell commands, file creation, Ghidra disassembly and decompilation, flag checking and giving up. It evaluates five LLMs and compares their scores with human teams at CSAW 2022 and 2023.
Scope and limitations
Only 51 of the 200 challenges are reverse engineering (31 qualifying and 20 final), so overall solve rates are not reverse-engineering accuracy. All challenges come from one competition series, categories are imbalanced, and the remaining 367 of the initial 567 challenges, including incident-response tasks, are excluded pending validation. The baseline counts a challenge as solved if any of five attempts succeeds, within a 48-hour limit per model; no statistical tests were run. Among GPT-3.5, GPT-4, Claude 3, Mixtral and LLaMA, the best reverse-engineering rate was 9.80% (GPT-4), and the open models solved nothing. The human comparison uses competition scores from four CSAW events, and the public repository has continued to change after publication, so later runs may not match the paper’s challenge set or framework.
Why it fits this collection
Supplies a public, dockerized CTF challenge set with a reverse-engineering subset; later agent work such as EnIGMA evaluates on it.
Publication and review details
- Venue
- NeurIPS 2024 Datasets and Benchmarks
- Status
- Conference paper
- Publication / posting date
- 2024 (exact date not verified)
- Date source
- The official NeurIPS proceedings and publisher-deposited Crossref metadata identify publication year 2024; the proceedings page’s citation_publication_date metadata reads 2024-12-16, but neither Crossref nor a publisher statement confirms that day, so only the year is recorded. The first arXiv submission date is recorded separately.
- First arXiv submission
- June 8, 2024
- arXiv
- 2406.05590
- Reviewed arXiv version
- v3
- DOI
- 10.52202/079017-1832
The 13-author order follows the NeurIPS proceedings page, the PDF, arXiv and Crossref. NeurIPS volume 37 lists pages 57472–57498; arXiv v3 (2025-02-18) matches the reviewed proceedings text. The NeurIPS checklist gives the Zenodo software DOI 10.5281/zenodo.13930622 for the v20241008 benchmark release; it is an artifact identifier, not the paper DOI. The reviewed NYU_CTF_Bench snapshot (2025-09-22) is GPL-2.0, holds the 200-challenge test split and a 55-challenge development split, and is installed through the nyuctf Python package, which clones the repository; challenges need Docker. The paper’s llm_ctf_automation link now redirects to NYU-LLM-CTF/nyuctf_agents, whose reviewed 2026-07-17 snapshot is MIT-licensed and also contains the later D-CIPHER multi-agent system alongside the baseline. Challenge files keep their original authors’ terms. No code or experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.
Primary sources
- https://proceedings.neurips.cc/paper_files/paper/2024/hash/69d97a6493fbf016fff0a751f253ad18-Abstract-Datasets_and_Benchmarks_Track.html
- https://proceedings.neurips.cc/paper_files/paper/2024/file/69d97a6493fbf016fff0a751f253ad18-Paper-Datasets_and_Benchmarks_Track.pdf
- https://api.crossref.org/works/10.52202/079017-1832
- https://arxiv.org/abs/2406.05590
- https://arxiv.org/pdf/2406.05590v3
- README.md GitHub · NYU-LLM-CTF/NYU_CTF_Bench · Revision 4c5744f
- LICENSE GitHub · NYU-LLM-CTF/NYU_CTF_Bench · Revision 4c5744f
- README.md GitHub · NYU-LLM-CTF/nyuctf_agents · Revision 612190f
- https://zenodo.org/records/13930622