AwesomeREA.org

2024 · Conference paper

Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search

Nicola Dainese; Matteo Merler; Minttu Alakuijala; Pekka Marttinen

GIF-MCTS generates executable Python transition and reward models from natural-language environment descriptions and offline state-transition trajectories, using prediction feedback to generate, improve and repair code. CWMB evaluates 18 reinforcement-learning environments and downstream planning; a separate experiment uses a simplified Read to Fight Monsters grid world.

Scope and limitations

Assumes deterministic, fully observable dynamics and sufficiently detailed descriptions; this is not recovery from pixels or binaries alone. Accuracy uses the replay transitions supplied during synthesis, alongside planning-return evaluation, rather than proving correctness on all unseen states or full-engine equivalence. RTFM fixes the manual across episodes and keeps monsters stationary. CWMB also includes non-game control and physics tasks; some planning returns are unavailable and complex continuous-state models remain difficult.

Why it fits this collection

Shows how documented rules and observed transitions in game-like environments can become executable behavior models with test feedback. It complements game reconstruction without claiming recovery of original engines, graphics or assets.

Publication and review details

Venue
NeurIPS 2024
Status
Conference paper
Publication / posting date
2024 (exact date not verified)
Date source
The final NeurIPS proceedings and publisher-deposited Crossref record identify 2024. No publication month or day is asserted; conference event dates are not substituted for publication dates.
First arXiv submission
May 24, 2024
arXiv
2405.15383
DOI
10.52202/079017-1933

Reviewed the final proceedings paper, pages 60429–60474. Its WorldCoder baseline follows the first arXiv version, not later revisions. The Apache-2.0 repository provides prompts, replay-buffer files and evaluation scripts; bundled RTFM carries CC BY-NC 4.0. The pinned evaluator balances reward classes, grants partial credit over next-state components, skips terminal next-state checks, and accepts nonempty valid-action subsets for RTFM; continuous states use 1e-5 relative/absolute tolerance. Its score is not a strict full-transition match rate. The planning script defaults to two episodes, while the paper reports ten. Reproduction needs the Linux-oriented Conda/Python 3.10 environment, Gymnasium/MuJoCo and model access. No upstream code, serialized buffer or model was executed.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Primary sources

  1. https://proceedings.neurips.cc/paper_files/paper/2024/hash/6f479ea488e0908ac8b1b37b27fd134c-Abstract-Conference.html
  2. https://proceedings.neurips.cc/paper_files/paper/2024/file/6f479ea488e0908ac8b1b37b27fd134c-Paper-Conference.pdf
  3. https://api.crossref.org/works/10.52202/079017-1933
  4. https://arxiv.org/abs/2405.15383
  5. https://sites.google.com/view/code-world-models/home
  6. https://github.com/nicoladainese96/code-world-models/blob/e17e787b41137793bcef9f860bdfe9e6fc07b768/README.md
  7. https://github.com/nicoladainese96/code-world-models/blob/e17e787b41137793bcef9f860bdfe9e6fc07b768/LICENSE
  8. https://github.com/nicoladainese96/code-world-models/blob/e17e787b41137793bcef9f860bdfe9e6fc07b768/RTFM/LICENSE
  9. https://github.com/nicoladainese96/code-world-models/blob/e17e787b41137793bcef9f860bdfe9e6fc07b768/environment.yml
  10. https://github.com/nicoladainese96/code-world-models/blob/e17e787b41137793bcef9f860bdfe9e6fc07b768/src/mcts/gif_mcts/simulators/simulator_cwm.py
  11. https://github.com/nicoladainese96/code-world-models/blob/e17e787b41137793bcef9f860bdfe9e6fc07b768/src/cwm_benchmark/generate_env.py
  12. https://github.com/nicoladainese96/code-world-models/blob/e17e787b41137793bcef9f860bdfe9e6fc07b768/src/experiments/eval_cwm_planning.py
  13. https://github.com/nicoladainese96/code-world-models/blob/e17e787b41137793bcef9f860bdfe9e6fc07b768/data/prompts/rtfm/generate.md