2026 · Conference accepted
RFCLLM: Evaluating LLMs' Reasoning Ability of Network Protocol State Machines
Tests reasoning over protocol state machines using 1,482 questions across 16 protocols. Four context settings distinguish RFC retrieval from reasoning over supplied transitions; tasks cover next states, conditions, predecessor states and paths.
Scope and limitations
Evaluates reasoning, not a new FSM extractor or implementation reconstruction. Gold FSMs were prepared by one author; judge agreement was checked on 50 instances by the three authors. Packet formats, error handling, configurations and real code mappings are outside scope. Results depend on context, judges and output budgets.
Why it fits this collection
Provides a bounded validation reference for using LLMs to interpret recovered or documented protocol models.
Publication and review details
- Venue
- Findings of EMNLP 2026
- Status
- Conference accepted
- Publication / posting date
- 2026 (exact date not verified)
- Date source
- Author arXiv record reports acceptance to Findings of EMNLP 2026; a proceedings publication date was not verified.
- First arXiv submission
- September 11, 2026
- arXiv
- 2609.13389
- Reviewed arXiv version
- v1
Reviewed September 11 v1. Acceptance is author-reported, not evidence of completed proceedings publication. The author-linked anonymous artifact exposes README, scripts and results; its UI lists access until December 16, 2026, but no full Git commit or root license was established. No model or benchmark was run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.