2025 · Conference paper
On Benchmarking Code LLMs for Android Malware Analysis
CAMA is a framework for benchmarking how code LLMs read decompiled Android malware. Androguard decompiles each APK into Java functions, and each model returns a structured output per function: a summary, a refined function name and a 0–10 maliciousness score. Two analysis tasks build on these outputs. Malicious-function identification is scored by consistency (whether scores and names stay stable when the model is re-prompted with its own summaries) and by fidelity (how much a LightGBM malware-category classifier loses confidence when the top-scored function descriptors are removed). Malware-purpose summarization is scored by semantic relevance (BLEU, METEOR and ROUGE-L against reference app descriptions). The benchmark holds 118 APKs from 6 categories and 13 families, with 7,542,799 decompiled functions. Of the four models evaluated, StarChat-Beta has the highest maliciousness consistency (0.813, against 0.381 for CodeLlama) and narrowly the highest semantic-relevance scores, while fidelity varies widely across models and removal sizes. Renaming functions with model-suggested names raises fidelity for CodeLlama and StarChat but lowers METEOR by about 10%.
Scope and limitations
This is an eight-page workshop paper that evaluates four open-source code LLMs: CodeLlama-7b-Instruct, StarChat-Beta and the base-size CodeT5 and PLBART, not current frontier models. CodeT5 and PLBART were fine-tuned for three epochs to produce the output fields, and their maliciousness scores come from a separate DeepSeek-R1-Distill-Llama-70B model, so score-based results do not measure those two models alone; both are also excluded from the renaming study. No function-level ground truth exists: malicious-function identification is measured indirectly through classifier confidence, and the reference app descriptions are GPT-4-generated summaries from the earlier LAMD dataset; the authors note that these surrogates can add noise and bias. The 118 samples are de-duplicated LAMD malware decompiled only with Androguard, and obfuscation is not studied separately. Fidelity scores have standard deviations close to their means, for example 0.332 ± 0.30 for CodeT5 at k = 2. The Zenodo record holds the structured model outputs (function summaries, refined names and maliciousness scores); neither it nor the LAMD record states that it contains the APK samples. CAMA does not detect malware and is not a decompiler.
Why it fits this collection
Adds a benchmark of code LLMs reading decompiled Android malware, complementing DeGuard's identifier recovery and the single Android case study in the USENIX Security 2024 code-analysis study elsewhere in the bibliography.
Publication and review details
- Venue
- ISSTA Companion 2025 (LLMSC Workshop)
- Status
- Conference paper
- Publication / posting date
- June 11, 2025
- Date source
- ACM's Crossref deposit gives online publication and the issued date as 2025-06-11, with a print date of 2025-06-25, the first day of the June 25–28 conference. This entry uses the publisher's online date; the arXiv dates are not substituted.
- First arXiv submission
- April 1, 2025
- arXiv
- 2504.00694
- Reviewed arXiv version
- v2
- DOI
- 10.1145/3713081.3731745
ACM's Crossref deposit gives the ISSTA Companion '25 proceedings, pages 153–160, a CC BY 4.0 license and the same seven authors in order; the arXiv v2 PDF prints the ACM reference and DOI. The workshop name rests on the arXiv comment and the Zenodo description, which both state acceptance to the LLMSC Workshop 2025 of the 34th ACM SIGSOFT ISSTA Companion; Crossref names only the companion proceedings. arXiv v1 is dated 2025-04-01 and v2 2025-04-23. The paper links no code repository, so code_url is null. Its outputs are the Zenodo dataset 10.5281/zenodo.15155917 (2025-04-05, CC BY 4.0, a 710 MB CAMA.zip that was not downloaded); that dataset DOI is not used as the paper identifier. The samples and reference descriptions come from the LAMD dataset 10.5281/zenodo.14884736, whose 2 MB archive is described as CSV sample information and logs. The ACM landing page refused automated requests, so the review relied on Crossref and the arXiv PDF. No code or experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.