AwesomeREA.org

2026 · Conference accepted

ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents

Yilei Jiang; Yaozhi Zheng; Yuxuan Wan; Jiaming Han; Qunzhong Wang; Michael R. Lyu; Xiangyu Yue

Splits screenshot-to-code generation among grounding, planning and generation agents that label UI regions, build a hierarchical layout tree and write HTML/CSS. The pipeline also produces the Screen-10K training pairs used for supervised and reinforcement fine-tuning, and the paper introduces the 1,000-page ScreenBench.

Scope and limitations

Output is static HTML/CSS layout; interaction, JavaScript behavior and backend logic are not reconstructed. Results are visual-similarity scores (CLIP and OCR-matched text blocks) on ScreenBench and Design2Code, which do not establish code-level or functional equivalence. The released pipeline calls a hosted model API, Doubao by default. The reviewed text is arXiv v2; the accepted EMNLP version may differ.

Why it fits this collection

Represents staged, agent-based screenshot-to-code reconstruction for websites, separating visual grounding and layout planning from code synthesis.

Publication and review details

Venue
EMNLP 2026
Status
Conference accepted
Publication / posting date
2026 (exact date not verified)
Date source
The official EMNLP 2026 detailed program lists a main-conference poster, verifying acceptance; no proceedings publication date is asserted.
First arXiv submission
July 30, 2025
arXiv
2507.22827
Reviewed arXiv version
v2

The official EMNLP 2026 detailed program lists paper 7281-MAIN as a poster on 2026-10-26; this records acceptance, not proceedings publication. The arXiv v2 comment 'ScreenCoder-v2' is a version label, not a venue, and an earlier ICLR 2026 submission was withdrawn. The abstract-linked repository is Apache-2.0 at the pinned commit and bundles UIED, LLaMA-Factory and VLM-R1 code with their own license files. ScreenBench is published on Hugging Face under Apache-2.0.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Primary sources

  1. https://arxiv.org/abs/2507.22827
  2. https://2026.emnlp.org/program/
  3. https://docs.google.com/spreadsheets/d/1aXGTy_7Xeh-OXIs3iJSbDpUsZ6YhbfA76b17kBZH0Yk/edit?gid=943432490#gid=943432490
  4. https://openreview.net/forum?id=29VraifZkB
  5. https://github.com/leigest519/ScreenCoder/blob/22816fdfb4ee5982973f9a09d67498a64823e65d/README.md
  6. https://github.com/leigest519/ScreenCoder/blob/22816fdfb4ee5982973f9a09d67498a64823e65d/LICENSE
  7. https://jimmyzhengyz.github.io/ScreenCoder-Project-Page/
  8. https://huggingface.co/datasets/Leigest/ScreenCoder/blob/b0e49d2a5d937e93f712d4c281d4036ec2652a30/README.md