2024 · Conference paper
Large Language Models for Code Analysis: Do LLMs Really Do Their Job?
Evaluates GPT-3.5-turbo, GPT-4, LLaMA-2-13B, Code-LLaMA-2-13B-Instruct and StarChat-Beta on explaining what de-commented JavaScript, Python and C code does, then repeats the task on JavaScript obfuscated with JavaScript Obfuscator (identifier renaming, dead-code injection, control-flow flattening and string splitting) and with Wobfuscator’s WebAssembly transformation. GPT-3.5 and GPT-4 are also asked to rewrite 100 IOCCC winning programs as readable, compilable code. Case studies apply GPT models to a botnet and a benign browser-extension repository, an Android SMS stealer decompiled with JD-GUI, and WannaCry code decompiled with RetDec.
Scope and limitations
Ground truth is GPT-4’s own explanations after manual checking by four students; other models are scored against those texts with similarity metrics and a GPT-4 judge, which the authors say is unreliable without manual calibration. Correctness is judged at explanation level, not by recovered behavior or executable equivalence. Only JavaScript is obfuscated with generated transformations; neither GPT model deciphered the WebAssembly that Wobfuscator inserts, and the smaller models produced no meaningful explanations once obfuscation was applied. Author-reported figures include 97.4% GPT-4 accuracy on non-obfuscated code and 87% on obfuscated JavaScript; in IOCCC de-obfuscation, about 20% of GPT-3.5’s targets and 19% of GPT-4’s yielded compilable code, and only 8% and 4% produced correct results. The malware analyses are qualitative case studies on a few samples, with manual chunking for WannaCry, missed behavior when context was split, and a false alarm in the botnet repository. Models recognized code from popular repositories, so memorization is a confound. The evaluated GPT-3.5/GPT-4-era and LLaMA-2-13B-class models are dated. The study analyzes source or already-decompiled text; it does not perform binary decompilation.
Why it fits this collection
Measures how far LLM explanations survive JavaScript obfuscation and decompiler output, complementing JavaScript deobfuscation benchmarks and tools in the collection.
Publication and review details
- Venue
- USENIX Security 2024
- Status
- Conference paper
- Publication / posting date
- August 2024
- Date source
- Publication month in the official USENIX proceedings BibTeX; the August 14–16, 2024 symposium dates are not used to assert a publication day.
- First arXiv submission
- October 18, 2023
- arXiv
- 2310.12357
- Reviewed arXiv version
- v2
USENIX publishes no DOI for the paper; the arXiv identifier is recorded, and the final PDF name usenixsecurity24-fang.pdf does not use the secYY publisher profile. The USENIX page, its BibTeX and the final PDF agree on the 11-author order, the single-name author Asmita, pages 829–846 and ISBN 978-1-939133-44-1; the page also lists a prepublication PDF. arXiv v2 (2024-03-05) carries the comment “Accepted by Usenix Security 2024”. The paper names the online appendix; at the reviewed 2024-08-28 commit it holds dataset, result and case-study folders and declares no license, and its samples include third-party code that keeps its own terms. The appendix is recorded as the project link rather than as code. Model versions beyond family names are not specified in the paper. The appendix contents and malware samples were not downloaded, and no experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.
Primary sources
- https://www.usenix.org/conference/usenixsecurity24/presentation/fang
- https://www.usenix.org/system/files/usenixsecurity24-fang.pdf
- https://arxiv.org/abs/2310.12357
- https://arxiv.org/pdf/2310.12357v2
- README.md GitHub · aseec-lab/llms-for-code-analysis · Revision f1bdb01
- https://github.com/aseec-lab/llms-for-code-analysis/tree/f1bdb01f1da9143bda796c259f06c7364683a102