2026 · Conference paper
Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering
BinJudgeBench collects expert judgments for function naming, binary-code summaries and decompilation refinement. BinJudge learns to select a judge configuration for each input and task, evaluating agreement with human scores rather than executable correctness.
Scope and limitations
The router uses five-fold validation on 1,233 annotated outputs. Initial annotators are three authors, with an independent industrial expert checking a subset. Human scores are subjective, and fluent unsupported answers can mislead judges. Correlation on these tasks neither proves program equivalence nor establishes that judges can replace human review.
Why it fits this collection
Provides evidence-oriented evaluation resources for interpreting names, summaries and refined pseudocode.
Publication and review details
- Venue
- ASE 2026
- Status
- Conference paper
- Publication / posting date
- October 7, 2026
- Date source
- Publisher-deposited print publication date; online and arXiv dates are recorded separately.
- First arXiv submission
- August 7, 2026
- arXiv
- 2608.07038
- Reviewed arXiv version
- v1
- DOI
- 10.1145/3832783.3834339
The arXiv abstract still says accepted, while ACM’s Crossref deposit records ASE 2026 publication on 2026-10-07. Direct ACM access returned HTTP 403. The paper-linked Zenodo concept 19247812 resolves to version 19310393, titled Artifact for BinJudge, with anonymous creator metadata and CC BY 4.0. Its 2.36 GB archive was not downloaded; internal code/data licensing and reproduction were not audited.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.