AwesomeREA.org

2025 · Conference paper

WAFFLE: Fine-tuning Multi-Modal Model for Automated Front-End Development

Shanchao Liang; Nan Jiang; Shangshu Qian; Lin Tan

WAFFLE fine-tunes multimodal LLMs to generate HTML/CSS from UI screenshots. A quarter of the decoder's attention heads use a structure-aware mask that limits each element's tokens to its parent, preceding siblings and its own earlier tokens. A contrastive objective aligns image and code embeddings over groups of four mutated WebSight-v0.1 pages. With VLM-WebSight and Moondream2 backbones, the authors report up to 9.00 pp higher HTML-Match and 0.0982 higher CW-SSIM than standard fine-tuning on WebSight-Test, plus gains on every Design2Code metric.

Scope and limitations

Generates static HTML/CSS from a single screenshot; it does not recover a site's original source, scripts, interactions, navigation or backend. HTML-Match strips styles and attributes before requiring pixel-identical renders, while CW-SSIM, CLIP and LLEM are similarity scores; the authors note that these metrics do not fully capture human judgment. Training data and the 500-page WebSight-Test are synthetic WebSight-style pages; Design2Code supplies 484 real-world pages. Only two backbones were fine-tuned, and on Design2Code the Moondream2 model trained with WAFFLE stays below GPT-4o and GPT-4o mini. The Gemini 1.5 Pro baseline answered 384 Design2Code instances. Human ranking used 50 samples per dataset and two annotators. Commercial-model prompting results are historical baselines, not current rankings.

Why it fits this collection

Adds a training method for screenshot-to-HTML website reconstruction that builds DOM hierarchy into attention, complementing the collection's screenshot-to-code benchmarks, datasets and prompting pipelines.

Publication and review details

Venue
ACL 2025
Status
Conference paper
Publication / posting date
July 2025
Date source
The official ACL Anthology record gives July 2025. Crossref records only the year; the July 27–August 1 conference dates printed in the PDF are not treated as a publication day.
First arXiv submission
October 24, 2024
arXiv
2410.18362
Reviewed arXiv version
v3
DOI
10.18653/v1/2025.acl-long.1208

Publication identity follows the ACL Anthology record and PDF. Reviewed arXiv v3 of 2026-03-02 (v1 2024-10-24, v2 2025-06-24) is titled “WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development”; the repository README and model card use “WAFFLE: Multi-Modal Model for Automated Front-End Development” and cite the arXiv preprint. The paper's availability statement links lt-asset/Waffle. The reviewed 2025-05-28 snapshot is BSD-3-Clause and provides VLM-WebSight inference, Selenium/Chrome rendering, the HTML tree-attention utilities, a contrastive dataset class reading a local load_from_disk dataset, and 500 WebSight-Test PNG screenshots without reference HTML. Although README describes a training file, the snapshot has no training script, Moondream2 code, metric implementation or mutated training set; no public copy of the 231,940-pair set was found in the lt-asset Hugging Face organization. Weights are on Hugging Face as lt-asset/Waffle_VLM_WebSight (7B). The GitHub README releases them under Apache-2.0 subject to the base model's terms, while the model card declares bsd-3-clause-clear; no single license is inferred. Documented dependencies include Python 3.10.14, PyTorch 2.3.0, Transformers 4.41.1, PEFT 0.11.1 and DeepSpeed 0.14.1; the inference script expects a CUDA GPU and has placeholder paths. Source files were read without downloading weights or data, running code or reproducing results.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Report an outdated entry Suggest a translation fix

Primary sources

  1. https://aclanthology.org/2025.acl-long.1208/
  2. https://aclanthology.org/2025.acl-long.1208.pdf
  3. https://api.crossref.org/works/10.18653/v1/2025.acl-long.1208
  4. https://arxiv.org/abs/2410.18362
  5. README.md GitHub · lt-asset/Waffle · Revision 5ca57f8
  6. LICENSE GitHub · lt-asset/Waffle · Revision 5ca57f8
  7. vlm_websight GitHub · lt-asset/Waffle · Revision 5ca57f8
  8. vlm_websight/eval_websight.py GitHub · lt-asset/Waffle · Revision 5ca57f8
  9. vlm_websight/dataset.py GitHub · lt-asset/Waffle · Revision 5ca57f8
  10. WebSight-Test GitHub · lt-asset/Waffle · Revision 5ca57f8
  11. https://huggingface.co/lt-asset/Waffle_VLM_WebSight/blob/296f6de57f8e646b1455ca00e3bc596ac0ea9429/README.md
  12. https://huggingface.co/HuggingFaceM4/VLM_WebSight_finetuned