2025 · Conference paper
BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models
BinMetric poses 1,000 questions across six binary-analysis tasks: call-site reconstruction, decompilation, signature recovery, binary code summarization, algorithm classification and assembly instruction generation. The questions come from 20 open-source C projects compiled for x86-64, stripped and processed with IDA Pro, and aligned with source code through DWARF information. The authors evaluate 12 LLMs from five model families with one-shot prompts, alongside IDA Pro’s decompiler, LLM4Decompile, BinT5 and HexT5 where those apply.
Scope and limitations
Scores are question-level metrics on isolated snippets, not end-to-end reverse-engineering workflows or agentic tool use. Preprocessing requires IDA Pro, and all binaries are x86-64 builds with each project’s default compiler settings on Ubuntu 22.04; other architectures, optimization sweeps and obfuscation are not studied. Most tasks score text overlap with Rouge-L, BLEU, METEOR or CodeBLEU, so they do not test semantic equivalence; only assembly generation checks compilation and test-case execution, where the average execution pass rate is below 1%. Summary labels were generated by ChatGPT and then manually reviewed, and algorithm classification uses 80 snippets from a single algorithms project. Leakage screening relied on exact-match web searches. The evaluated models predate 2025 (including gpt-3.5-turbo-16k-0613 and gpt-4-0613) and ran with a maximum length of 8,192 tokens, so the rankings are historical.
Why it fits this collection
Adds a multi-task benchmark spanning binary lifting, semantic recovery, algorithm identification and assembly synthesis, complementing single-task decompilation and agent benchmarks.
Publication and review details
- Venue
- IJCAI 2025
- Status
- Conference paper
- Publication / posting date
- September 2025
- Date source
- The IJCAI Organization’s Crossref deposit gives publication month September 2025. The proceedings page’s citation metadata separately gives 2025-09-16; this entry keeps the month that both sources state and does not use the August 16–22 conference dates.
- First arXiv submission
- May 12, 2025
- arXiv
- 2505.07360
- Reviewed arXiv version
- v1
- DOI
- 10.24963/ijcai.2025/858
The proceedings title reads “Binary Code Analysis Benchmark”; arXiv v1, the only version, is titled “A Comprehensive Binary Analysis Benchmark for Large Language Models” and is a 23-page extended version whose comment says it was to be published in IJCAI 2025. The proceedings PDF, page metadata and Crossref agree on the eight-author order, the DOI and pages 7715–7723 in the Main Track. Neither the proceedings PDF nor arXiv v1 links code, data or a leaderboard. A same-named GitHub repository describes itself as the IJCAI’25 benchmark but declares no license and is not linked from the paper, so it is not recorded as author-provided code. Per-task counts are 70 call-site, 250 decompilation, 250 signature, 250 summarization, 80 classification and 100 assembly-generation items. No benchmark data was downloaded, and no experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.