Research
Glaurung Models
Glaurung Models contains tokenizer and model experiments for executable analysis. Its documented 64K BPE tokenizer encodes compiled PE and ELF bytes using Latin-1, with x86 and ARM64 training coverage; an accompanying analysis compares learned SSE and AVX instruction patterns across two tokenizers.
- Targets
- Binary
- Category
- Native analysis tools
- Relationship
- Learning resource
- Sources reviewed
Requirements & limits
- Prerequisites: The documented tokenizer workflow uses the tokenizers and transformers Python packages and Latin-1 conversion of binary bytes.
- Testing evidence: The document reports an average of 2.849 bytes per token for the 64K tokenizer.
- License: No licence file in the repository.
About this entry
The description was reviewed against primary sources.
Primary sources
- glaurung_binary_tokenizer_001_README.md GitHub · mjbommar/glaurung-models · Revision 1819020
- SSE_AVX_ANALYSIS.md GitHub · mjbommar/glaurung-models · Revision 1819020
More in References and learning
- CrackMes · Learning resource
- Binary BPE Tokenizer for Executable Analysis · Research
- Assassin's Creed III online service docs · Documentation
- AC4 docs · Documentation
- eRepublik API Documentation · Documentation
- Garmin Strength Workout API · Documentation