2017 · Conference paper
Game Engine Learning from Video
Learns a rule-based forward simulator of a 2D game from gameplay video. Using a supplied sprite palette, OpenCV locates sprites in each frame; matched sprites are converted into hand-designed fact types for animation, position, relative distance, x and y velocity and camera offset. Game Engine Search then scans the frames, and whenever the current engine’s predicted next frame differs from the real one by more than a threshold, it runs a greedy search over engines that add or modify condition–effect rules or mark them as player-controlled. Engines are ranked by pixel error and rule count. Trained on one Super Mario Bros. Level 1-1 video, the learned engine predicts frames of a different player’s video more accurately than a CNN frame predictor and a previous-frame baseline, and a value-iteration agent planning with it shows no significant difference from one using the Infinite Mario engine’s native forward simulation.
Scope and limitations
The evaluation covers one level of one game: training uses a single “speedrunner” video of Super Mario Bros. Level 1-1 (1,569 frames), and testing uses an “explorer” video of the same level. Frame parsing depends on a fan-authored sprite sheet and hand-authored fact types, which the authors say makes the method less applicable to more complex domains such as real video. The learned engine is a set of search-derived rules for on-screen sprite behavior, not recovered game code or a binary; it does not represent player death or level transitions, and mechanics absent from Level 1-1 are not expected to transfer. Search can be slow at low thresholds, up to two weeks for one level on a 2013 iMac, and the reported run used a zero threshold. The CNN comparison used downscaled grayscale frames, and the gameplay comparison is indirect, through agents in the Infinite Mario engine.
Why it fits this collection
Adds a search-based method for reconstructing game rules from gameplay video, complementing the section’s neural and LLM-based world-model and code-synthesis research.
Publication and review details
- Venue
- IJCAI 2017
- Status
- Conference paper
- Publication / posting date
- August 2017
- Date source
- The IJCAI-deposited Crossref record gives August 2017 for print publication, represented at month precision. The IJCAI proceedings page gives no publication day, and the conference dates are not substituted.
- DOI
- 10.24963/ijcai.2017/518
The IJCAI proceedings page and the 7-page PDF give the three authors, the IJCAI-17 main track and pages 3707–3713; the Crossref record matches the authors and DOI. The paper links no code, learned engines or parsed data, and the Super Mario Bros. footage and fan-made sprite sheet are third-party materials. No code or experiments were run.
Sources reviewed
Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.