2026 · Conference paper
Towards LLM-Resistant Software Protection: Agent Failure Patterns in CTF Reverse Engineering
Runs Claude Code, Codex CLI and ChatGPT-5.2 Pro on 24 crackme-style challenges selected from 15 international CTF events held in 2025, and attributes their failures to stages of an Observe–Comprehend–Plan loop and to three protection effects: concealment, complication and misdirection. From the failure logs, the authors derive four recurring weaknesses (training bias, over-trust in observations, context limitation and plan persistence). They rerun both CLI agents without a decompiler and compare outcomes with in-competition solve rates.
Scope and limitations
The study covers 24 locally solvable Linux x86-64 ELF crackmes and three commercial agents (Claude Opus 4.5 and GPT-5.1-Codex-Max in the CLI tools, and an undisclosed ChatGPT model), evaluated in December 2025 with network access; runs that consulted public write-ups were excluded. The binaries are small relative to real-world software, and malware analysis and vulnerability discovery are out of scope. Each configuration had at most three attempts with early stopping, which the authors call noisy; success means printing the correct flag or finding an accepted input. Weakness labels come from qualitative log analysis by two evaluators and describe these runs, not general properties of LLM agents. ChatGPT-5.2 Pro ran in an uncontrolled web sandbox and is excluded from the decompiler comparison. Solve rates are among teams that solved at least one challenge, unavailable for six tasks, and may include LLM-assisted teams. Training-data exposure remains a validity threat, and the paper does not publish its assembled challenge set, run logs or code.
Why it fits this collection
Links reverse-engineering agent failures to specific protection effects and analysis stages, complementing benchmarks that mainly report success rates.
Publication and review details
- Venue
- Workshop on Binary Analysis Research (BAR) 2026
- Status
- Conference paper
- Publication / posting date
- 2026 (exact date not verified)
- Date source
- The Internet Society’s Crossref deposit gives publication year 2026. The PDF footer names the 27 February 2026 workshop; no finer publication date is asserted from the event date.
- DOI
- 10.14722/bar.2026.23058
BAR 2026 is a workshop co-located with NDSS Symposium 2026, not the NDSS main track. The five-author order follows the PDF, the NDSS paper and program pages, and the Internet Society’s Crossref deposit. The DOI resolves directly to the PDF, whose footer gives ISBN 978-1-970672-08-4; Crossref lists no pages, and no page range was verified. No arXiv version was found. Table I gives per-task times and weakness labels for each configuration; the per-agent counts (15, 17, 14, 14 and 19 of 24) and the decompiler comparison are author-reported. The paper names no code or data release. Experiments were not rerun for this collection.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.