A ~4,000-step horizon
The ingot arrives around step 3,600 on average, and the budget is 18,000. Credit for the final smelt has to propagate back past thousands of intervening decisions — far beyond what standard RL assigns reliably.
CS175 · Project in AI · Video Pre-Training on MineRL v1.0
An agent that mines
its own iron ingot.
Dropped into a fresh, randomly generated Minecraft world with an empty inventory, a 0.5B-parameter transformer policy chops wood, crafts a pickaxe, digs to the stone layer, hunts iron ore, builds a furnace, and smelts an ingot — from nothing but 640×360 pixels and a keyboard-and-mouse action space. No API access, no scripted subroutines, no reward shaping.
95.6%
- 4,246avg. steps
- 1,592fastest run
- 8milestones
Live demo
Run the agent in the browser
Minecraft itself can't run in a web page — MineRL launches a real Java client with an OpenGL context. So it runs headless on the server and streams the agent's own point of view here, frame by frame. Nothing opens on your machine.
How the live mode works, and why it's off by default
A single MineRL environment runs inside a container with Xvfb providing a
virtual X display and software OpenGL. The VPT policy steps it in a loop; each
640×360 POV frame is JPEG-encoded and pushed over a WebSocket alongside step count
and reward events. Every connected viewer watches the same episode — one
environment broadcast to many clients, rather than one Minecraft instance per visitor,
which would need several GB of RAM each.
That backend needs a persistent GPU box (~7 min per episode on a T4; 30+ min on CPU),
so the public site ships in replay mode — a recorded episode with the
identical HUD and milestone timeline. Point DEMO_BACKEND in
config.js at a running server and the same viewport goes live automatically.
The problem
Why this is hard
"Get an iron ingot" sounds like a tutorial task. As a learning problem it sits in the worst corner of the difficulty space: long horizon, sparse reward, partial observability, and an action space built for human hands.
Eight-link dependency chain
Ingot needs a furnace and ore; ore needs a stone pickaxe; stone needs a wooden pickaxe; that needs planks, sticks and a crafting table; all of it needs a log. Break any link and the run is dead — there's no recovering a missing pickaxe.
Pixels only
The policy sees the same 640×360 screen a person sees, GUI and all. Inventory state must be read off rendered menus. There is no symbolic observation, no item list, no coordinates.
A human action space
24 binary keys plus continuous mouse movement, quantised into camera bins. Crafting
means physically moving a cursor across a GUI grid — not calling craft().
Iron is genuinely scarce
Ore only generates below y≈63 and peaks in a narrow band. The agent has to dig down, survive, and search — the step from stone tools to iron ore is the single longest gap in the chain, averaging 1,447 steps.
A fresh world every time
Every episode is a new seed: different biome, terrain, cave layout, time of day. Nothing can be memorised. The policy has to generalise across the whole world generator.
Where the time actually goes
Gap between consecutive milestones, averaged over successful episodes. Two steps dominate everything else.
Method
How VPT learns to play
Video Pre-Training solves the label problem: there are 70,000 hours of Minecraft video online, but none of it records which keys were pressed. VPT learns to infer the actions, then trains on its own labels.
-
Stage 1
Inverse Dynamics Model
Contractors play Minecraft while their keyboard and mouse are recorded — 2,000 hours of video with ground-truth actions. A non-causal model learns to look at frames before and after a transition and name the action that caused it. Seeing the future makes this vastly easier than acting, so it needs far less data.
pIDM(at | o1…T)non-causal · 2,000 hrs labeled -
Stage 2
Behavioral cloning at scale
The IDM is run over 70,000 hours of unlabeled web video, producing pseudo-labels for every frame. A causal transformer policy is then trained to predict those actions from past frames alone. This is where the general Minecraft competence comes from — the foundation model that already knows what a tree is for.
ℒBC = −𝔼(o,a)∼𝒟[ log πθ(a | o≤t) ]causal · 70,000 hrs pseudo-labeled -
Stage 3
RL fine-tuning with PPO
The BC policy is fine-tuned with PPO on a reward that pays out once per new item in the crafting chain. Starting from a competent prior is what makes this tractable: random exploration would essentially never see an iron ingot. A KL penalty against the frozen BC policy keeps it from collapsing onto a degenerate strategy.
ℒPPO = 𝔼t[ min( rt(θ)Ât, clip(rt(θ), 1−ε, 1+ε)Ât ) ]checkpoint: rl-from-early-game-2x
What runs at inference
Frame640×360 RGB
Resize128×128
IMPALA CNN3 blocks · width 8
Transformer4 layers · 16 heads · 128 ctx
Action heads24 buttons + camera bins
Hidden size 2,048 with a 256-step attention memory carried across steps, so the policy
keeps a working memory of where it has been. Weights: rl-from-early-game-2x
(948 MB). The policy itself is OpenAI's, unmodified — the work here is the evaluation
harness and IronIngotWrapper, which terminates the episode the moment
iron_ingot > 0 appears in the inventory and logs the full milestone trace.
Results
50 episodes, measured
Each episode: a fresh seed, an 18,000-step budget, terminate on iron ingot. Five runs crashed on MineRL environment errors and are excluded rather than counted as failures — the rate below is over the 45 that ran to completion.
Completed episodes45of 50 attempted
Reached iron ingot4395.6% success
Average steps4,246of 18,000 budget
Fastest run1,592slowest 12,413
Every completed episode
One cell per episode, in run order. Filled = reached the ingot; hollow = stalled.
Success rate vs. baselines
Share of episodes reaching an iron ingot, same environment and budget.
Milestone progression
Average step at which each reward first fires, over successful episodes. The chain is cheap until iron.
Steps to iron ingot
Distribution across the 43 successful episodes. Strongly right-skewed — most runs finish fast, a few wander.
What the two failures have in common
Both reached iron ore and never smelted it. The agent mined the ore, and in one case placed a furnace, but never completed the fuel-and-smelt GUI interaction before the step budget ran out. Everything upstream of smelting is essentially solved at this model scale; the remaining error mass sits in one place.
Qualitative
The agent at each stage
Frames captured from evaluation episodes at the moment each reward fired.
Drop your demo video at web/assets/demo.mp4
and a poster frame at web/assets/video-poster.jpg. See web/assets/README.md.