AwesomeREA.org

2023 · Preprint

Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models

Xin Jin; Jonathan Larson; Weiwei Yang; Zhiqiang Lin

BinSum benchmarks LLMs on summarizing binary functions in natural language. The authors compile 44 GNU projects with GCC 7.5 for x86, x64, ARM and MIPS at -O0 to -O3, match 557,664 binary functions to developer comments as reference summaries, and present each function as raw bytes, assembly, IDA microcode or decompiled code with and without symbols. GPT-4, ChatGPT, Llama 2 and Code Llama receive prompts selected from 320 GPT-4-generated candidates, with the fine-tuned BinT5 as a baseline; outputs are scored by embedding similarity alongside BLEU, METEOR and ROUGE-L. Decompiled code is the most useful representation, and stripping symbols lowers the average similarity score from 0.449 to 0.202.

Scope and limitations

Reference summaries are developer comments extracted from source, which the authors note may be inconsistent with the code. Embedding similarity with all-mpnet-base-v2 and n-gram metrics measure closeness to those comments, not whether a summary is correct or useful to an analyst, and no human evaluation is reported. All binaries are open-source GNU programs built with one GCC version, and the authors note that firmware, malware and obfuscated code are not covered. Prompts were chosen on 1,000 randomly sampled functions. The results describe the June 2023 gpt-4-0613 and gpt-3.5-turbo-16k-0613 snapshots and the 7B and 13B Llama 2 and Code Llama models, so they do not rank current models.

Why it fits this collection

Adds a dedicated large-scale benchmark for natural-language summaries of binary functions, a task that appears elsewhere in the bibliography only as one part of broader binary-analysis benchmarks.

Publication and review details

Venue
arXiv
Status
Preprint
Publication / posting date
December 15, 2023
Date source
arXiv v1 submission date; no later version, journal reference or proceedings publication was found.
First arXiv submission
December 15, 2023
arXiv
2312.09601
Reviewed arXiv version
v1

Reviewed arXiv v1, the only version, which has no comment, journal reference or DOI; a Crossref search found no peer-reviewed version. The paper says its code and dataset are released at github.com/xinjin95/BinSum, but at the reviewed commit, last pushed on 2023-12-18, that repository holds only a README whose Dataset section reads “Coming soon”. It has no license file, and its arXiv badge shows the wrong identifier 2312.02120 while linking to 2312.09601. No code link is recorded. The authors report 4,058,297,977 inference tokens, US$11,418 in model execution costs and 873 A100 GPU hours. No data was downloaded, and no code or experiments were run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Report an outdated entry Suggest a translation fix

Primary sources

  1. https://arxiv.org/abs/2312.09601v1
  2. https://arxiv.org/pdf/2312.09601v1
  3. README.md GitHub · xinjin95/BinSum · Revision bb2e567
  4. https://github.com/xinjin95/BinSum/tree/bb2e56778349da36a4609a223fa8d84d39517db4