AwesomeREA.org

2024 · Journal article

CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking

Zian Su; Xiangzhe Xu; Ziyang Huang; Zhuo Zhang; Yapeng Ye; Jianjun Huang; Xiangyu Zhang

CodeArt pre-trains a BERT-sized encoder from scratch on 26 million stripped x86-64 functions built from jTrans's BinaryCorp. Instead of relying on symbols, it uses IDA Pro control-flow graphs and a binary data-flow analysis to compute control and data dependences, then restricts self-attention with masks so a token attends only to instructions it transitively depends on or that depend on it, to tokens in its own instruction, and to a global [CLS] token; pre-training masks tokens and perturbs the dependence graph. The authors fine-tune it for binary similarity, malware family classification and type inference. They report improvements over prior models from 53% to 64% recall@1 for similarity in pools of 500, from 49% to 60% LRAP for malware families, and from 74% to 94% average F1 for type inference.

Scope and limitations

The model handles x86-64 only, with functions disassembled by IDA Pro and inputs capped at 512 tokens. Similarity is fine-tuned on BinaryCorp-3M and tested on six projects (Curl, Binutils, ImageMagick, SQLite, OpenSSL and PuTTY), with Coreutils for validation, against GNN, GMN, jTrans and DiEmph baselines; on PuTTY, whose functions are short, CodeArt is only comparable to DiEmph. Type inference uses the x86-64 part of StateFormer's dataset with 35 labels and compares with StateFormer's reported results rather than a rerun. Malware classification uses 5,484 VirusTotal and VirusShare samples from the 100 most frequent families, against jTrans and a re-implemented CNN. All gains are relative to these baselines and settings. CodeArt produces embeddings and task-specific predictions; it is neither a decompiler nor a malware detector.

Why it fits this collection

Adds general-purpose pre-training for stripped binary functions that uses program dependences in place of missing symbols, evaluated on similarity, type inference and malware-family tasks that support reverse engineering.

Publication and review details

Venue
Proceedings of the ACM on Software Engineering (FSE 2024)
Status
Journal article
Publication / posting date
July 12, 2024
Date source
ACM's Crossref deposit records print and online publication on 2024-07-12. The July 2024 issue date in the reference format and the FSE 2024 conference dates are not substituted for it.
First arXiv submission
February 19, 2024
arXiv
2402.11842
Reviewed arXiv version
v1
DOI
10.1145/3643752

Crossref records Proceedings of the ACM on Software Engineering volume 1, issue FSE, pages 562–585, and the seven-author order matches arXiv. The ACM landing page refused automated access, so publication facts follow Crossref and the reviewed text is arXiv v1 (2024-02-19), the only version, which carries the PACMSE reference format and DOI. The abstract lists the three gains in the order similarity, type inference, malware; the contribution list and evaluation assign 49%→60% LRAP to malware family classification and 74%→94% F1 to type inference, which this record follows. The paper links ziansu/codeart. At the reviewed 2025-03-10 commit the repository declares MIT and its README points to an artifacts branch; README-docker.md describes a 3.8 GB Docker image with a Gradio demo and links the PurCL CodeArt-Artifacts collection on Hugging Face, whose checkpoints and datasets, including top-100 malware-family data, declare no license. The REQUIREMENTS file asks for Docker, 64 GB RAM, a 24 GB GPU and up to 100 GB of disk. Nothing was downloaded or run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Report an outdated entry Suggest a translation fix

Primary sources

  1. https://dl.acm.org/doi/10.1145/3643752
  2. https://api.crossref.org/works/10.1145/3643752
  3. https://arxiv.org/abs/2402.11842
  4. https://arxiv.org/pdf/2402.11842v1
  5. README.md GitHub · ziansu/codeart · Revision 0f78714
  6. LICENSE GitHub · ziansu/codeart · Revision 0f78714
  7. README-docker.md GitHub · ziansu/codeart · Revision 0f78714
  8. REQUIREMENTS GitHub · ziansu/codeart · Revision 0f78714
  9. https://huggingface.co/collections/PurCL/codeart-artifacts-662e73dcd9b837e4b970c9be