AwesomeREA.org

2026 · Preprint

Reveree: Diagnosing LLM Reverse-Engineering Agents

Hadjer Benkraouda; Hongyu Cai; Berkay Celik; Gang Wang

Evaluates reverse-engineering agent traces through flag recovery, eight analysis milestones and action profiles. Deterministic checks and an outcome-blinded LLM judge help locate failures beyond a single solve-rate score.

Scope and limitations

Uses 88 public CTF challenges and nine models; all four prompting strategies are compared on four models. Evaluation fixes the D-CIPHER harness and uses one attempt per setting. One expert checks 15 traces; agreement on judged comprehension stages is 53/60. Recall and surface-perturbation probes do not eliminate contamination.

Why it fits this collection

Adds process-level evaluation for binary-analysis agents.

Publication and review details

Venue
arXiv
Status
Preprint
Publication / posting date
September 1, 2026
Date source
First arXiv submission; no reviewed venue publication is established.
First arXiv submission
September 1, 2026
arXiv
2609.01185
Reviewed arXiv version
v1
DOI
10.48550/arxiv.2609.01185

Reviewed arXiv v1 and registered DOI. The paper announces a release, but the reviewed manuscript does not identify a dedicated artifact URL. No code, challenge binaries or experiments were run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Primary sources

  1. https://arxiv.org/abs/2609.01185v1
  2. https://arxiv.org/html/2609.01185v1
  3. https://api.datacite.org/dois/10.48550/arxiv.2609.01185