2023 · Journal article
Revisiting Binary Code Similarity Analysis Using Interpretable Feature Engineering and Lessons Learned
Builds BinKit, a reproducible cross-compilation benchmark of 243,128 binaries and 36,256,322 functions from 51 GNU packages, covering 1,352 combinations of eight architectures, nine GCC and Clang versions, five optimization levels, and PIE, LTO, no-inline and Obfuscator-LLVM options. TikNib, an interpretable baseline over simple presemantic function features, measures how compilers, options and architectures change those features. The authors report that a few basic features approach a deep-learning tool’s accuracy, that type features push AUC above 0.99, and trace failures to IDA Pro analysis errors, compiler back-end differences and architecture-specific code.
Scope and limitations
Studies function-level, presemantic (syntactic and structural) features on Linux GNU packages compiled from source with debugging information for ground truth; semantic features, NLP-based models and whole-binary or other similarity tasks are outside the evaluation. The above-0.99 AUC uses argument and return types derived from source via Ctags and debugging information, not types recovered from stripped binaries. The state-of-the-art comparison sets TikNib’s AUC on VulSeeker’s ASE datasets against VulSeeker’s author-reported numbers rather than a re-run, and omits the original firmware images. Obfuscation covers Obfuscator-LLVM substitution, bogus control flow and flattening applied once. Experiments used IDA Pro 6.95, and the authors cannot rule out erroneous MIPS feature extraction. The results do not rank later learned or LLM-based models.
Why it fits this collection
Adds a public, reproducible benchmark and interpretable baseline for binary function matching, and documents how compiler settings and disassembler errors bias similarity results, complementing learned models such as jTrans.
Publication and review details
- Venue
- IEEE Transactions on Software Engineering
- Status
- Journal article
- Publication / posting date
- April 2023
- Date source
- IEEE’s Crossref deposit gives the volume 49, issue 4 print issue date as 2023-04-01, represented at month precision. The 2020-11-21 arXiv v1 date and the June 2022 acceptance note are not substituted for the journal issue date.
- First arXiv submission
- November 21, 2020
- arXiv
- 2011.10749
- Reviewed arXiv version
- v4
- DOI
- 10.1109/TSE.2022.3187689
IEEE TSE 49(4), pages 1661–1682; Crossref records a CC BY 4.0 license from 2023-04-01. The reviewed arXiv v4 (2022-07-07) notes acceptance in June 2022; its running header still reads November 2020, a template date rather than the publication date. The IEEE and Crossref title capitalizes “Using”, while arXiv uses “using”. At reviewed BinKit commit f6565a3 (2023-12-15, MIT), the README describes BinKit 2.0 with 371,928 binaries, 1,904 option combinations, 23 compilers and an added Ofast level, which the paper did not evaluate; its BinKit 1.0 section matches the paper’s 243,128 binaries and 1,352 combinations, counting four obfuscation options as compilers (13 in total), and was tested on Ubuntu 16.04. Paper-version datasets, feature pickles and toolchains download from Google Drive. TikNib, the analysis tool, is a separate MIT repository reviewed at 3b7c165 (2025-05-18); its README gives IDA Pro 6.95 scripts for the paper setup and separate IDA Pro 7.5 scripts. No datasets were downloaded and no code or experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.
Primary sources
- https://doi.org/10.1109/TSE.2022.3187689
- https://api.crossref.org/works/10.1109/TSE.2022.3187689
- https://arxiv.org/abs/2011.10749
- https://arxiv.org/pdf/2011.10749v4
- README.md GitHub · SoftSec-KAIST/BinKit · Revision f6565a3
- LICENSE.txt GitHub · SoftSec-KAIST/BinKit · Revision f6565a3
- README.md GitHub · SoftSec-KAIST/TikNib · Revision 3b7c165
- LICENSE.txt GitHub · SoftSec-KAIST/TikNib · Revision 3b7c165