AwesomeREA.org

2020 · Conference paper

Learning to Simulate Dynamic Environments with GameGAN

Seung Wook Kim; Yuhao Zhou; Jonah Philion; Antonio Torralba; Sanja Fidler

GameGAN learns an action-conditioned neural simulator of a game from recorded frames and the actions taken, without access to the game’s logic or engine. An action-conditioned LSTM dynamics engine with a stochastic latent variable predicts the next state, an optional external memory module builds an internal map so that revisited locations stay consistent, and a rendering engine decodes frames while separating static from dynamic components. Training is end-to-end with adversarial losses, including an action-conditioned discriminator, and a temporal cycle loss for memory consistency. The authors evaluate on a modified Pac-Man, a ghost-free Pac-Man-Maze and VizDoom “take cover”, training RL agents inside the learned simulators and testing them in the real games, and measure long-term consistency with a come-back-home task; a human-playable model of the official Pac-Man is shown qualitatively.

Scope and limitations

GameGAN imitates a game’s pixels and dynamics; it does not recover game code, symbolic rules, assets or an executable. Training needs large logged datasets with actions: 40K Pac-Man episodes from a high-entropy DQN agent that sees the full environment, and 10K random-policy VizDoom episodes, at 84×84 and 64×64 resolution. Quantitative results cover small grid-based Pac-Man variants and one VizDoom scenario, with an external classifier supplying game outcomes. Agents trained in the simulators can exploit modeling errors: the full model sometimes let Pac-Man pass through walls, and its Pac-Man agents scored below those of the memoryless variant, while on VizDoom the World Model baseline scored higher than GameGAN. The released code supports VizDoom training only, relies on a deprecated WorldModels data setup, and is under the non-commercial NVIDIA Source Code License; Pac-Man data and the official Pac-Man model are not included.

Why it fits this collection

Adds a neural approach to reconstructing a playable game simulator from observed frames and actions, complementing the section’s program-synthesis and black-box game-replication research.

Publication and review details

Venue
CVPR 2020
Status
Conference paper
Publication / posting date
June 2020
Date source
IEEE’s Crossref deposit and an archived copy of the IEEE Xplore page give June 2020, represented at month precision. The May 2020 arXiv submission and the conference dates are not substituted.
First arXiv submission
May 25, 2020
arXiv
2005.12126
Reviewed arXiv version
v1
DOI
10.1109/CVPR42600.2020.00131

Crossref and the archived IEEE Xplore page give the five authors, CVPR 2020 and pages 1228–1237; IEEE metadata capitalizes the title as “Learning to Simulate Dynamic Environments With GameGAN”, while the CVF open-access paper and arXiv print “with”. The 10-page CVF PDF is the Computer Vision Foundation’s open-access version; IEEE Xplore holds the version of record. arXiv has only v1, submitted 2020-05-25 with the comment “CVPR 2020”. The repository README at the reviewed 2021-11-11 commit gives BibTeX pages 1231–1240, which differ from IEEE, and its Project Page link points to an unrelated lift-splat-shoot page, so no project URL is recorded. The README lists Ubuntu 18.04 and Python 3.8.3, and requirements pin torch 1.4.0 and torchvision 0.5.0. LICENSE section 3.3 limits use to non-commercial research or evaluation. No code, data or experiments were run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Report an outdated entry Suggest a translation fix

Primary sources

  1. https://ieeexplore.ieee.org/document/9156900
  2. https://web.archive.org/web/20240914161135/https://ieeexplore.ieee.org/document/9156900/
  3. https://api.crossref.org/works/10.1109/CVPR42600.2020.00131
  4. https://openaccess.thecvf.com/content_CVPR_2020/papers/Kim_Learning_to_Simulate_Dynamic_Environments_With_GameGAN_CVPR_2020_paper.pdf
  5. https://arxiv.org/abs/2005.12126
  6. readme.md GitHub · nv-tlabs/GameGAN_code · Revision 1195c39
  7. LICENSE GitHub · nv-tlabs/GameGAN_code · Revision 1195c39
  8. requirements.txt GitHub · nv-tlabs/GameGAN_code · Revision 1195c39