AwesomeREA.org

2026 · Conference accepted

ProgramBench: Can Language Models Rebuild Programs From Scratch?

John Yang; Kilian Lieret; Jeffrey Ma; Parth Thakkar; Dmitrii Pedchenko; Sten Sootla; Emily McMilin; Pengcheng Yin; Rui Hou; Gabriel Synnaeve; Diyi Yang; Ofir Press

Evaluates black-box program reconstruction across 200 tasks: agents query an executable and read its documentation, then independently implement a codebase whose behavior is checked by generated tests.

Scope and limitations

Binary inspection, decompilation, and internet access are forbidden during inference. Passing finite tests does not prove complete behavioral equivalence or recovery of the original source. Later live-leaderboard runs are separate from the reviewed arXiv results.

Why it fits this collection

Provides an executable benchmark for recovering software behavior through experiments, complementing benchmarks that permit static binary analysis.

Publication and review details

Venue
NeurIPS 2026
Status
Conference accepted
Publication / posting date
2026 (exact date not verified)
Date source
The official NeurIPS 2026 poster listing verifies conference acceptance; no proceedings publication date is asserted.
First arXiv submission
May 5, 2026
arXiv
2605.03546
Reviewed arXiv version
v1
DOI
10.48550/arXiv.2605.03546

NeurIPS lists a December 2026 poster; this records acceptance, not proceedings publication. The July 2026 DL4C oral was a separate non-archival workshop appearance. The DOI identifies the arXiv preprint. The pinned harness is version 1.2.5 under MIT; its generated-test dataset preserves per-task attribution and upstream license terms, including copyleft obligations for derivative portions.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Primary sources

  1. https://arxiv.org/abs/2605.03546
  2. https://arxiv.org/html/2605.03546v1
  3. https://neurips.cc/virtual/2026/poster/139090
  4. https://neurips.cc/Downloads/2026
  5. https://dl4c.github.io/papers/
  6. https://dl4c.github.io/callforpapers/
  7. https://programbench.com/
  8. https://github.com/facebookresearch/ProgramBench/blob/895f65916bc5ba34a915d36da321fde0301125ad/README.md
  9. https://github.com/facebookresearch/ProgramBench/blob/895f65916bc5ba34a915d36da321fde0301125ad/docs/README.md
  10. https://github.com/facebookresearch/ProgramBench/blob/895f65916bc5ba34a915d36da321fde0301125ad/pyproject.toml
  11. https://github.com/facebookresearch/ProgramBench/blob/895f65916bc5ba34a915d36da321fde0301125ad/LICENSE
  12. https://huggingface.co/datasets/programbench/ProgramBench-Tests/blob/de0ddfb637590c7ecb54fa0b5301f6dc7dfbcee5/README.md