Mirage: I Built a Game That Runs Inside a Neural Network

This project started with me killing another one. I had built a GPU zombie-survival RL environment — technically solid, thousands of lines of hand-written CUDA, PPO training, the works. But it had a flaw I couldn't unsee: the task didn't actually need machine learning. "Fly toward the beacon, dodge the zombies" can be solved by a 15-line hand-written controller. I had trained a model to do something I could have just... programmed.

So I scrapped it, name included, and asked the question properly: what is a problem where learning is the only way in?

The answer became Mirage: write an arcade game as a CUDA simulator, then train a neural network to be the game. Not to play it — to replace the game engine itself. Given the last four frames and the button you're pressing, the network paints the next frame. Once it's good enough, you take the real game away and the network keeps going, feeding on its own output. You are now playing inside the model's imagination, live, in a browser:

Left is the real game. Right is the dream — no game code is producing those pixels. They start from the same opening frames and drift apart, because the dream is a valid future, not the future.

Nobody can write "the rules of the next frame" by hand — they're the physics, collisions, splitting rocks, and glow rendering all entangled in 12,288 pixel values. That's the test the old project failed and this one passes: learning is structurally irreplaceable here.

The whole thing — simulation, training, and live inference — runs on my GTX 970. A 4 GB card from 2014.

Step 1: The game itself (and why it's written in CUDA)

The game is a deliberately classic design: an inertial ship (rotate / thrust / fire), rocks that split twice before vanishing, a torus arena where flying off one edge wraps you to the other. 64×64 pixels, 12 discrete actions, high-contrast palette.

Training a world model needs millions of example frames. That's the real reason for CUDA: a GPU is thousands of weak processors that all run the same instruction on different data — useless for most programs, perfect for "step this game" ×512 and "shade this pixel" ×4096. The sim steps 512 independent games in parallel (one GPU thread per game) and rasterizes them (one thread per pixel) at about 206,000 frames per second, never touching the CPU.

That speed flips the usual ML workflow on its head: there is no dataset. Every training step, the simulator plays a couple of fresh ticks in all 512 arenas and the new frames stream straight into a ring buffer the trainer samples from. Infinite fresh data, zero disk, nothing to memorize. The simulator isn't the product — it's the data factory.

One detail I'm fond of: the arena is a torus, so I made every convolution in the network use circular padding — the network's wiring literally matches the world's topology. There's a unit test proving the model is indifferent to sliding the whole world around the edges.

Step 2: Make the fast thing provably correct

Before training anything, I needed to trust the simulator — a subtly broken data factory poisons every conclusion downstream. Two ideas did the heavy lifting.

First, the entire game logic exists twice: once in CUDA (fast, hard to debug) and once in plain numpy (slow, readable). Tests march both implementations 300 steps and demand agreement to four decimal places. Two independent calculators agreeing is how you trust the fast one.

Second — my favorite trick in the whole project — randomness is stateless. Random number generators are normally like a shuffled deck dealt in order, and two implementations drift out of sync the moment one draws a card the other doesn't. Instead, every random number here is a pure hash of the question being asked:

// "What's the random value for THIS arena, THIS step, THIS purpose?"
// Same question -> same answer, in CUDA and numpy alike. Nothing to desync.
__host__ __device__ inline float rnd01(unsigned int seed, unsigned int arena,
                                       unsigned int step, unsigned int salt) {
    unsigned int h = pcg_hash(seed ^ pcg_hash(arena * 0x9E3779B9u
                       ^ pcg_hash(step * 0x85EBCA6Bu ^ salt)));
    return (float)(h >> 8) * (1.0f / 16777216.0f);
}

GPU↔CPU parity passed on the first build of the kernel, and the sim is bitwise-deterministic across runs. The renderer is guarded by golden-image tests — fixed scenes whose exact pixels are committed to git:

Step 3: The world model, attempt one — and a beautiful failure

The first model (v1) is a U-Net — the standard image-to-image architecture: shrink the image to understand the big picture, re-expand it to paint the output, with skip connections carrying crisp detail across. The action you pressed is injected into every layer (FiLM conditioning), because the same four frames have different futures depending on whether you're thrusting or firing. 2.6M parameters, trained for 2.3 hours on the 970 with the simplest possible objective: make the predicted frame close to the real next frame, on average.

It works scarily well one step ahead — 37 dB PSNR, which is "you can't tell them apart" territory. And the dream genuinely understands the game: I built a probe that stages a scenario inside the dream — rock dead ahead, press fire — and checks whether the game's law holds. 24 out of 24 times, shooting a large rock in the hallucination splits it into two mediums. The network learned a game rule purely from pixels.

But watch what happens over a longer dream:

Top: the first seconds, indistinguishable. Bottom: nine seconds in, the dream (right panel) has melted — the ship is gone, the rocks are ghosts.

This is the most instructive failure in machine learning, and it's not a bug. When a rock splits, the shards fly at randomized angles — several futures are all possible. A model graded on average error has one mathematically optimal answer: predict the average of all futures, a semi-transparent smear of every possibility. Then the dream loop feeds that smear back as input, and you're photocopying a photocopy. Blur compounds until the world dissolves.

The lesson: the loss function doesn't just measure your model. It defines what your model wants, and "be close on average" wants ghosts.

Step 4: Diffusion — learning to commit

The fix changes the question. Instead of "what is the expected next frame?" (which invites averaging), ask "give me one plausible sample from the distribution of next frames." That's what diffusion models do — the same family behind modern image generators.

The training recipe is beautifully indirect. Take a real next-frame. Pour random noise over it — sometimes a little, sometimes so much it's nearly pure static. Hand the noisy mess to the network with the context frames and the action, and demand the clean frame back:

def loss(self, ctx, action, target):
    x0 = scale(target)                       # clean frame in [-1, 1]
    sigma = exp(P_MEAN + P_STD * randn(n))   # random noise intensity
    noised = x0 + sigma * randn_like(x0)     # destroy it
    denoised = self.denoise(noised, sigma, ctx, action)
    weight = (sigma**2 + SIGMA_DATA**2) / (sigma * SIGMA_DATA) ** 2
    return (weight * (denoised - x0) ** 2).mean()   # restore it

To get good at restoring frames at every noise level, the network is forced to deeply learn what this game world looks like — that knowledge is exactly what fills in whatever the noise destroyed.

Then the punchline: at play time you hand it 100% pure static and politely insist there's a very noisy frame in there. It "restores" a frame that never existed — steered by the context and your keypress — refining from coarse to fine over 4 passes. The randomness of the static picks which future you get; the network's knowledge makes it a valid one. No averaging. The shards fly one specific way.

Nine seconds into a dream, same seed, same inputs as before:

Still sharp. Still the game. It's showing a different arrangement than the real sim by now — it committed to its own future — but that's what sampling means.

One extra trick mattered (borrowed from the GameNGen paper): during training I sometimes added noise to the context frames too, telling the model how much. A dreaming model's context is its own slightly-imperfect previous outputs — by training on smudged context, it learns to treat its own small mistakes as noise to see through rather than gospel to amplify.

The dreaming loop itself, by the way, is almost embarrassingly simple:

ctx = deque(last_4_real_frames, maxlen=4)
while playing:
    frame = model.predict_next(stack(ctx), action=your_keypress)
    ctx.append(frame)        # the model now feeds on its own imagination
    send_to_browser(frame)

The numbers, honestly

I evaluated both models on held-out data the training never saw, and the results taught me more than a clean victory would have:

• One-step accuracy: ~42 dB PSNR for both — at one step ahead, you cannot tell either model from the real game.
• Ten steps ahead: v2 wins, 27.3 vs 25.6 dB.
• The split-rule probe: v1 wins, 24/24 vs 18/24. On a mostly-deterministic game, futures are usually single — averaging barely hurts, and the simple model nails the laws.
• Sharpness after 10 seconds of dreaming: v2 retains 100% of the real game's edge crispness; v1 has visibly melted.
• Speed: v1 dreams at 200+ fps; v2 at 54 fps with 4 denoising passes — both far above the 15 fps the game needs, on a 4 GB card from 2014.

So diffusion bought commitment, not rule fidelity, at this scale — and the humble regression baseline is genuinely strong on a near-deterministic game. I also tried two automated "sharpness" metrics that completely failed to capture what your eye sees instantly, and I kept that failure in the writeup instead of shopping for a friendlier metric. If I've learned one meta-lesson from ML it's this: the field's main failure mode is fooling yourself, and the most valuable skill is refusing to. PSNR famously rewards blur; a stochastic model "diverging" from the one true trajectory is doing its job, not failing it. You have to know what your metrics actually measure.

What I have learnt

• Pick problems where ML is load-bearing. If a heuristic solves it, the model is decoration. The strongest answer to "why does this need ML?" is "the deliverable is the learned function."
• The loss function is the product spec. v1's blur wasn't a bug — it was the L1 loss getting exactly what it asked for. Diffusion isn't "better"; it answers a different question.
• Fast and correct beats fast. The numpy oracle, stateless RNG, and golden images cost a day and bought me the right to trust every result after.
• Honest evaluation is a feature you build, not a paragraph you write at the end. The in-dream rule probes were more informative than any pixel metric.
• A 2014 GPU is plenty for real ML systems work — batched simulation, streaming training, diffusion inference at interactive framerates. Constraints are a feature; they force you to understand instead of scale.

The build log — every decision, bug, and dead end, including two pkill incidents I'd rather not discuss — lives in the repo as a generated "journey" PDF, and the stack is: hand-written CUDA kernels (sim + rasterizer), PyTorch for both world models, FastAPI + a WebSocket streaming raw frames to a vanilla-JS canvas. No frameworks were harmed in the making of the frontend.