2026 · Preprint
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
Studies development of the Extractor reverse-engineering tool through commits, available session logs and code-provenance tracing. Taxonomies separate instruction intent, change triggers and agent claims, exposing errors corrected before commits that ordinary repository history misses.
Scope and limitations
One codebase with human-directed work, no human-coded comparison. The 14.3% figure is 15 of 105 audited generation events, including test defects and stale tests, not a defect rate per source line or in shipped code. LLM scoring has sampled human calibration; transcripts are incomplete and the archive omits per-response evidence bundles. This companion study does not independently corroborate the related schema-recovery results.
Why it fits this collection
Adds provenance and claim-auditing methods for agent-built reverse-engineering tools.
Publication and review details
- Venue
- arXiv
- Status
- Preprint
- Publication / posting date
- September 24, 2026
- Date source
- First arXiv submission; the reviewed version is v1, not a verified proceedings publication.
- First arXiv submission
- September 24, 2026
- arXiv
- 2609.29744
- Reviewed arXiv version
- v1
Reviewed v1 and the author's pinned reproducibility-archive README, which documents lost sessions and unreproducible response-scoring evidence. The repository exposes an archive and Git-history material; neither was downloaded. No repository-wide license was verified, so code_url remains null and project_url identifies the reviewed artifact landing.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.
Primary sources
- https://arxiv.org/abs/2609.29744
- https://arxiv.org/html/2609.29744v1
- README.md GitHub · doug-leith/extractor-code-development · Revision 4f74f79
- README.md GitHub · doug-leith/ios-protobuf-decoding · Revision 9a4a688