AwesomeREA.org

2026 · Preprint

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

Jeonghye Kim; Minseon Kim; Young Jin Kim; Matheus Pereira; Marc-Alexandre Côté; Alessandro Sordoni; Xingdi Yuan; Zhengyan Shi

Evaluates recovery of web-app behavior from a live reference with hidden source. It provides 4,063 feature-restoration tasks across 26 applications and a separate 12-application reconstruction setting, scored by replaying dependent browser workflows.

Scope and limitations

Tests cover finite, model-mined behaviors in self-contained apps, including SaaS clones. Persistence failures can be detected through replayed workflows, but passing does not establish all backend or database invariants. Evaluated agents receive structured browser observations without screenshots; visual fidelity is not measured. Public verifiers and reference solutions are not a contamination-resistant test set.

Why it fits this collection

Directly studies reference-guided behavioral reconstruction beyond screenshot-to-HTML, including dependent state changes and fresh verification of submitted implementations.

Publication and review details

Venue
arXiv
Status
Preprint
Publication / posting date
September 16, 2026
Date source
arXiv v1 submission history; the earlier author blog date is separate.
First arXiv submission
September 16, 2026
arXiv
2609.18805
Reviewed arXiv version
v1
DOI
10.48550/arXiv.2609.18805

The DOI resolves and DataCite records registration on 2026-09-17, despite arXiv’s stale pending banner. Separately, the paper was accepted to the non-archival Workshop on Interpreting Agent Behavior (IAB) at NeurIPS 2026, scheduled for December 12, 2026; its OpenReview record was published on 2026-10-04. That record and the workshop listing put Xingdi Yuan before Alessandro Sordoni; the authors here follow arXiv v1. The public release contains executable benchmark packages, not the mining, task-generation or orchestration pipeline. ProgramDistill-authored artifacts declare MIT; included applications and assets retain upstream terms. The paper and later packaged release are distinguished; no experiments were reproduced.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Primary sources

  1. https://arxiv.org/abs/2609.18805
  2. https://arxiv.org/html/2609.18805v1
  3. https://microsoft.github.io/debug-gym/blog/2026/09/programdistill/
  4. https://huggingface.co/datasets/microsoft/ProgramDistill/blob/dc782f7003ae00738260996d02d77920d08a5db1/README.md
  5. https://huggingface.co/datasets/microsoft/ProgramDistill/blob/dc782f7003ae00738260996d02d77920d08a5db1/tasks/full-reconstruction/README.md
  6. https://huggingface.co/datasets/microsoft/ProgramDistill/blob/dc782f7003ae00738260996d02d77920d08a5db1/tasks/full-reconstruction/baserow_baserow/tests/scorer/trace_schema.py
  7. https://huggingface.co/datasets/microsoft/ProgramDistill/blob/dc782f7003ae00738260996d02d77920d08a5db1/tasks/full-reconstruction/baserow_baserow/tests/scorer/trace_replay_validator.py
  8. https://api.datacite.org/dois/10.48550/arXiv.2609.18805
  9. https://openreview.net/forum?id=FbKJFN0R5L
  10. https://github.com/iab-agents/iab-agents.github.io/blob/0437015a568d8d541e277833c68ffd213b117c53/src/App.jsx
  11. https://github.com/iab-agents/iab-agents.github.io/blob/0437015a568d8d541e277833c68ffd213b117c53/src/data/siteData.js