2022 · Preprint
Pop Quiz! Can a Large Language Model Help With Reverse Engineering?
Tests whether OpenAI’s Codex model code-davinci-001 can answer reverse-engineering questions about C code in a zero-shot question-and-answer prompt. Programs come from templates the authors wrote, covering cybersecurity snippets and industrial-control algorithms such as PID controllers and filters, plus five single-file Linux malware samples from vx-underground. Each is presented as original or name-randomized source, or as Ghidra decompilation of gcc -O1 builds with debug information, without it, or stripped. After exploratory open-ended questions, generated true/false and short-answer questions about purpose, capabilities and key variables are graded against an answer oracle; Codex answered 72,754 of 136,260 questions correctly.
Scope and limitations
All results come from one model, code-davinci-001, which OpenAI shut down on 2023-03-23, so they cannot be rerun against the original model and do not describe current LLMs. Accuracy is measured with a generated true/false and short-answer quiz, paired as positive and negated questions to expose answer bias, rather than with analyst tasks; the overall 53.4% led the authors to conclude that LLMs are not yet ready for zero-shot reverse engineering. Programs are small single-file C code that fits the model’s roughly 4,000-token window, compiled only at -O1; the real-malware class has five samples. The answer oracle combines the authors’ labels with capability matches from the capa tool.
Why it fits this collection
Records the earliest LLM-for-reverse-engineering evaluation in the bibliography, a baseline for the later LLM binary-analysis studies in this section.
Publication and review details
- Venue
- arXiv
- Status
- Preprint
- Publication / posting date
- February 2, 2022
- Date source
- arXiv v1 submission date; no later version, journal reference or proceedings publication was found.
- First arXiv submission
- February 2, 2022
- arXiv
- 2202.01142
- Reviewed arXiv version
- v1
Reviewed arXiv v1, the only version; its comment reads “18 pages, 19 figures” and links Zenodo record 10.5281/zenodo.5949075, a dataset DOI that is not used as the paper identifier. No peer-reviewed version was found in Crossref. That record (CC BY 4.0, a 692 MB codex-reverse.zip by the same six creators) holds the test framework scripts, recorded Codex responses and generated program instances. Its README, requirements file and build log were read through ranged requests: the README requires Ghidra and an OpenAI API key, and the build log lists compiled instances under experiments/malware, so the archive appears to include the malware samples and is not recorded as a code link. The archive was not downloaded in full or extracted, and no code or experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.