2025 · Conference paper
EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
EnIGMA extends SWE-agent for capture-the-flag challenges with interactive agent tools that keep a gdb debugging session or a pwntools server connection open across turns, alongside decompile and disassemble commands, an LM summarizer for long outputs, and category-specific demonstrations and guidelines. It is evaluated with four LMs on 390 challenges from NYU CTF, InterCode-CTF, Cybench and a HackTheBox set the authors collected. The paper ablates tools, summarizers and demonstrations, and analyzes solution leakage and “soliloquizing”, where the model writes its own observations instead of acting in the environment.
Scope and limitations
The benchmarks mix cryptography, forensics, pwn, reverse engineering, web and miscellaneous tasks (104 of the 390 are reverse engineering), so aggregate solve rates such as 13.5% on NYU CTF are not reverse-engineering accuracy. The per-category table reports 17.65% on NYU CTF’s 51 reverse-engineering challenges with Claude 3.5 Sonnet. Results are pass@1 with a $3 per-instance cost cap and 2024 models (GPT-4 Turbo, GPT-4o, Claude 3.5 Sonnet and LLaMA 3.1 405B Instruct); the NYU CTF baseline comparison uses pass@5. Many challenges have public write-ups, and the authors’ leakage and soliloquy measures are heuristics; on 21 later CSAW 2024 challenges the best configurations solved 2. Solving a CTF shows a captured flag, not recovered program structure or source.
Why it fits this collection
Evaluates how interactive debugger and connection tools change agent performance on CTF tasks that include reverse engineering, complementing the reverse-engineering-specific benchmarks in this section.
Publication and review details
- Venue
- ICML 2025
- Status
- Conference paper
- Publication / posting date
- October 6, 2025
- Date source
- PMLR states that Volume 267 was published on 06 October 2025, matching the paper page’s citation_publication_date of 2025/10/06. The 13–19 July 2025 conference dates are not used.
- First arXiv submission
- September 24, 2024
- arXiv
- 2409.16165
- Reviewed arXiv version
- v3
The 16-author order follows the PMLR page and the arXiv record; the PMLR metadata writes ‘Carlos E Jimenez’ and ‘Karthik R Narasimhan’, while the PDF prints ‘Carlos E. Jimenez’ and ‘Karthik Narasimhan’. PMLR lists pages 246–355 and no DOI; the arXiv v3 comment reads ‘ICML 2025’. The SWE-agent README still cites the earlier arXiv title ‘EnIGMA: Enhanced Interactive Generative Model Agent for CTF Challenges’. PMLR’s software link is the moving v0.7 branch; the reviewed head 084d5d1 (2025-04-28) is MIT-licensed and documents running challenges with Docker image sweagent/enigma:latest. The current main README says to use SWE-agent 0.7 while EnIGMA is updated for 1.0. The project site links trajectories (MIT); the separate benchmarks repository reformats Cybench and InterCode-CTF only and leaves their copyright with the original creators, so it does not supply the HackTheBox set. Table 9 reports the NYU-agent overall rates as 3.00 and 4.00 in the reverse order of the main text. No code or experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.
Primary sources
- https://proceedings.mlr.press/v267/abramovich25a.html
- https://raw.githubusercontent.com/mlresearch/v267/main/assets/abramovich25a/abramovich25a.pdf
- https://proceedings.mlr.press/v267/
- https://arxiv.org/abs/2409.16165
- https://arxiv.org/abs/2409.16165v3
- https://enigma-agent.com/
- README.md GitHub · SWE-agent/SWE-agent · Revision 084d5d1
- LICENSE GitHub · SWE-agent/SWE-agent · Revision 084d5d1
- docs/usage/enigma.md GitHub · SWE-agent/SWE-agent · Revision 084d5d1
- README.md GitHub · SWE-agent/SWE-agent · Revision 3ea751c
- LICENSE GitHub · enigma-agent/trajectories · Revision 431dda6
- README.md GitHub · enigma-agent/benchmarks · Revision c632a3c