AwesomeREA.org

2026 · Preprint

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

Grace Man Chen; Litao Guo; Yifan Wu; Yiyu Chen; Yenchi Tseng; Sicheng Liu; Yuyu Luo; Ying-Cong Chen

UI2App evaluates runnable React/TypeScript applications generated from 600 screenshots across 95 web applications. It separates build executability, visual fidelity and screenshot-implied interaction behavior, including cross-route state, using a reference interaction taxonomy and public evaluation pipeline.

Scope and limitations

Static screenshots underdetermine behavior; the rubric accepts multiple valid implementations rather than proving equivalence to the original application or backend. Auto IIS judges generated source and screenshots without tool execution; human runtime validation covers a 45-application subset. Cross-route state is not necessarily durable persistence across reloads or sessions. Main results use one generation per application-model pair, with repeated runs on a subset.

Why it fits this collection

Tests reconstruction from observed web interfaces while distinguishing visual copying from inferred interaction logic; its explicit behavior ambiguity makes it a bounded complement to trace-guided reconstruction benchmarks.

Publication and review details

Venue
arXiv
Status
Preprint
Publication / posting date
July 7, 2026
Date source
First arXiv submission date. The reviewed v2 was submitted on 2026-10-04; it is not a separately verified conference publication.
First arXiv submission
July 7, 2026
arXiv
2607.06306
Reviewed arXiv version
v2
DOI
10.48550/arXiv.2607.06306

Use v2’s 95 applications and 600 screenshots, not v1’s 45/327 counts. The arXiv DOI is registered. Public code is MIT; the dataset and reference annotations are CC BY 4.0, with upstream code and third-party asset rights separate. The documented pipeline needs Python 3.10+, Node/npm, pnpm, Chromium, Tesseract and model access; Auto IIS additionally needs an authenticated Codex CLI. No pipeline or benchmark was run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Primary sources

  1. https://arxiv.org/abs/2607.06306
  2. https://arxiv.org/html/2607.06306v2
  3. https://arxiv.org/pdf/2607.06306v2
  4. https://api.datacite.org/dois/10.48550/arXiv.2607.06306
  5. https://chenmancm169.github.io/UI2App-ProjectPage/
  6. https://github.com/chenmancm169/UI2App/blob/520a59aad0899b1eb61e80351aa1ffbfad9322a4/README.md
  7. https://github.com/chenmancm169/UI2App/blob/520a59aad0899b1eb61e80351aa1ffbfad9322a4/LICENSE
  8. https://github.com/chenmancm169/UI2App/blob/520a59aad0899b1eb61e80351aa1ffbfad9322a4/requirements.txt
  9. https://github.com/chenmancm169/UI2App/blob/520a59aad0899b1eb61e80351aa1ffbfad9322a4/iis/evaluate.py
  10. https://huggingface.co/datasets/UI2App/UI2App/blob/32e6f6193c5c265cf6d8afcbce12b641e3c2a758/README.md
  11. https://huggingface.co/datasets/UI2App/UI2App/blob/32e6f6193c5c265cf6d8afcbce12b641e3c2a758/annotations/rii.jsonl