Research
Binary BPE Tokenizer for Executable Analysis
Binary BPE Tokenizer for Executable Analysis studies byte-pair encoding of ELF, PE, Mach-O and APK files for transformer-based binary analysis. The repository provides paper sources, 4K to 64K tokenizer vocabularies, Python evaluation scripts and a sample archive. The author reports 2.89 bytes per token for the 64K vocabulary.
- Targets
- App · Binary
- Category
- Native analysis tools
- Relationship
- Learning resource
- Sources reviewed
Requirements & limits
- Prerequisites: Python 3.10+ with uv; TeX Live 2023+ is required to compile the paper.
- Prerequisites: Rust 1.70+ is optional for retraining the tokenizer from scratch.
- License: No licence file in the repository.
About this entry
The description was reviewed against primary sources.
Primary sources
- README.md GitHub · mjbommar/binary-tokenizer-paper · Revision 7ef1ec7
More in References and learning
- Assassin's Creed III online service docs · Documentation
- AC4 docs · Documentation
- eRepublik API Documentation · Documentation
- Garmin Strength Workout API · Documentation
- Substack API Reference (Unofficial) · Documentation
- hyperscan-homebrew · Documentation