AwesomeREA.org

2025 · Conference paper

Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping

Ryan Li; Yanzhe Zhang; Diyi Yang

Sketch2Code benchmarks vision-language models on turning rough hand-drawn wireframe sketches into HTML webpage prototypes. Crowd annotators drew 731 sketches of 484 real-world webpages from the Design2Code test set. Ten models, eight commercial and two 8B open-source, generate pages directly and in two multi-turn settings with a GPT-4o simulated user: following its feedback, or asking it clarification questions. Outputs are scored by Design2Code visual similarity and a new IoU-based layout similarity over component types such as text blocks, images, navigation bars, forms and buttons. Claude 3.5 Sonnet and GPT-4o lead single-turn layout similarity, and feedback rounds help commercial models more than question asking does.

Scope and limitations

The generated pages are prototypes scored by visual and layout similarity to reference screenshots, not recovered source code; behavior, interactivity and code equivalence are not evaluated, and page images are replaced with a placeholder. Sketches deliberately omit most text and styling, so similarity scores are low in absolute terms, with best single-turn layout similarity around 22%. The simulated user is GPT-4o, judged faithful in 83.3% of feedback and accurate in 86.7% of answers by human annotators. Only 8B open-source models were run, and a multi-turn example needs 40,000 to 160,000 input tokens. The user study interviewed eight UI/UX experts. Results describe 2024 model versions.

Why it fits this collection

Adds sketch-based and multi-turn interactive webpage generation, complementing the screenshot- and design-based website reconstruction benchmarks in this section.

Publication and review details

Venue
NAACL 2025
Status
Conference paper
Publication / posting date
April 2025
Date source
The ACL Anthology record gives the proceedings month as April 2025; Crossref records only the year. The conference ran from 29 April to 4 May 2025, and no publication day is inferred.
First arXiv submission
October 21, 2024
arXiv
2410.16232
Reviewed arXiv version
v1
DOI
10.18653/v1/2025.naacl-long.198

The ACL Anthology page, final PDF and Crossref establish the three-author order, DOI, NAACL 2025 Volume 1 proceedings and pages 3921–3955. Reviewed arXiv v1 (2024-10-21), the only version, commented “preprint, 9 pages”; the description follows the 35-page proceedings PDF. The paper links its code and data through the project page. At the reviewed 21dfd1d commit, SALT-NLP/Sketch2Code is MIT-licensed, pins Python packages including torch 2.4.0, transformers 4.44.1 and playwright 1.43.0, and requires OpenAI, Anthropic or Gemini API keys for the commercial models. The Hugging Face dataset card declares ODC-By, and the paper says the source webpages were collected by Design2Code under ODC-By. No data was downloaded, and no code or experiments were run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Report an outdated entry Suggest a translation fix

Primary sources

  1. https://aclanthology.org/2025.naacl-long.198/
  2. https://aclanthology.org/2025.naacl-long.198.pdf
  3. https://aclanthology.org/2025.naacl-long.198.bib
  4. https://api.crossref.org/works/10.18653/v1/2025.naacl-long.198
  5. https://arxiv.org/abs/2410.16232v1
  6. https://salt-nlp.github.io/Sketch2Code-Project-Page/
  7. README.md GitHub · SALT-NLP/Sketch2Code · Revision 21dfd1d
  8. LICENSE GitHub · SALT-NLP/Sketch2Code · Revision 21dfd1d
  9. requirements.txt GitHub · SALT-NLP/Sketch2Code · Revision 21dfd1d
  10. https://huggingface.co/datasets/SALT-NLP/Sketch2Code/blob/ce28bf4e62f9242cde5c5beaa8775efccfa8502b/README.md