2026 · Preprint
CrackMeBench: Binary Reverse Engineering for Agents
Evaluates agents on executable-checked CrackMe tasks: infer validation logic from binaries and submit an accepted input or artifact, while recording tool use, time, and scoring attempts.
Scope and limitations
The v0 study uses 12 generated tasks and eight public calibration tasks, Linux x86-64 ELF binaries, a shell interface, and a five-minute budget. Its compact educational tasks do not establish performance on large applications. No paper-linked public artifact URL was verified.
Why it fits this collection
Directly measures whether an autonomous reverse-engineering workflow produces a verifiably accepted result.
Publication and review details
- Venue
- arXiv
- Status
- Preprint
- Publication / posting date
- May 11, 2026
- Date source
- arXiv v1 submission
- First arXiv submission
- May 11, 2026
- arXiv
- 2605.10597
- Reviewed arXiv version
- v1
No conference venue is stated in the reviewed arXiv record. Tool repositories cited in its bibliography are not treated as the paper's own code.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.