2024 · Conference paper
How Far Have We Gone in Binary Code Understanding Using Large Language Models
Benchmarks 16 code and general LLMs on function name recovery and summarization of stripped binaries. The benchmark takes 12 popular C projects built for x86-64, strips and decompiles 28 binaries with IDA Pro, aligns 81,185 functions with source through DWARF, and keeps 2,000 functions after expert review. Zero-shot LLMs are compared with SymLM, NER, BinT5 and HexT5. Code Llama 34B leads name recovery with token-level F1 of 27.59, and ChatGPT leads summarization with BLEU-4 7.37. The study also measures few-shot prompting, LoRA fine-tuning of two 7B models on GNU binaries, and a case study on a stripped open-source Linux remote-access trojan.
Scope and limitations
The benchmark covers x86-64 builds with each project’s default compiler and optimization settings and IDA Pro pseudocode only, so the results do not establish behavior on other architectures, optimization levels, decompilers or obfuscated code, which the authors exclude. Reference summaries were generated by ChatGPT from source and screened by three experts, and ChatGPT is also an evaluated model. BLEU-4, METEOR and Rouge-L measure overlap with those references, which the authors say may not fit binary summarization. Name recovery uses token-level precision, recall and F1 rather than exact matches. Fine-tuning used 30,000 functions from O0 x86-64 GNU builds. Results describe model versions released between mid-2023 and early 2024, including gpt-3.5-turbo-16k.
Why it fits this collection
Adds a benchmark of LLMs on two core reverse-engineering subtasks, recovering function names and summarizing decompiled functions, with comparisons against earlier learned symbol-recovery models.
Publication and review details
- Venue
- ICSME 2024
- Status
- Conference paper
- Publication / posting date
- October 6, 2024
- Date source
- IEEE’s Crossref deposit gives published/print and issued date 2024-10-06, which is also the first day of the conference listed there (6–11 October). This entry uses the publisher-deposited date and does not derive it from the conference schedule or the arXiv history.
- First arXiv submission
- April 15, 2024
- arXiv
- 2404.09836
- Reviewed arXiv version
- v3
- DOI
- 10.1109/ICSME58944.2024.00012
Crossref establishes the nine-author order, DOI, ICSME 2024 proceedings title and pages 1–12. Reviewed arXiv v3 (2024-10-24), whose comment reads “to be published in ICSME 2024”; v1 and v2 date from 15 and 16 April 2024. The IEEE Xplore page refused automated requests, so the description rests on arXiv v3 and is not claimed to match the IEEE version page for page. The paper gives no download link for the benchmark, fine-tuning data or prompts, and no release was found. The evaluated pseudocode depends on IDA Pro, a commercial decompiler. No code, models or experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.