ARC-AGI-3 · 25-game public evaluation · no training, no demonstrations

OPINE-World: programmatic world models, learned online from raw interaction

A single scored run per game. Two cooperating LLM agents play an unknown game from raw 64×64 frames, write its dynamics as a Python program, admit the program only when it replays all observed transitions exactly, and steer exploration by a Bayesian measure of object-type adequacy we call ontology error.

David Courtis · Wenhao Li · Scott Sanner · University of Toronto, D3M Lab

arXiv 2607.01531 PDF Code Reproduce

–

games won, of 25 in the public evaluation set

–

mean action-efficiency score

1.7×

human pace, averaged over the 20 wins

gradient updates, demonstrations, or per-game engineering

Results · score vs levels cleared

ARC-AGI-3 scores a run by how few actions it spends against a per-level human baseline, so the score rewards clearing games and clearing them efficiently. OPINE-World reaches a mean score of 78.4 and wins 20 of 25 games.

Frontier-LLM baselines are the official single-run sessions published per task at arcprize.org (fetched 2026-07-11, mirrored in assets/arcprize_baselines.json).

Method · hypothesis, test, repair

An object-centric world model, synthesized as code, verified by exact replay

01 · THE LOOP

Two agents, a shared replay buffer, an exact-replay verifier

Game transitions are recorded into a buffer D_t = {(s, a, s′)}. A goal-directed agent reads the log and picks the next action sequence. A world-model agent reads the same buffer and writes the dynamics as a Python program: o′ ≈ f_τ(o, a, u), one rule per type τ in local context u. A candidate engine T̂ is admitted when

T̂(s, a) = s′ for every (s, a, s′) ∈ D_t

acceptance is exact replay of the buffer, a misprediction is a counterexample and the trigger for a rewrite

A rejected proposal's mispredicted transition becomes the counterexample for the next round. After the first level clear, a planner searches the verified program, validating planned steps against the live game.

OPINE-World architecture: goal-directed agent, replay buffer, world-model agent, exact-replay verifier, planner, with ontology error in the loop

02 · OBJECTS AND EFFECTS

States are objects, changes are abstract effects

A state is a tuple of objects O = (o₁, …, o_n), and each object o_i = (κ_i, τ_i, p_i) carries a tracking key κ_i, an inferred type τ_i, and properties p_i (position, pixel pattern). The extractor producing these is itself synthesized from the raw 64×64 grids. Under action a, the changed fields of o_i form its delta Δ_i, and the observed effect keeps the shape of the change

e_i = ρ(Δ_i(O, O′))

ρ discards magnitudes, x : 12 → 15 becomes "x", making outcomes comparable across objects and levels

The outcome alphabet E grows from observation: no_change · x · y · pixels · gone · born. A few transitions suffice to show that two objects respond alike, the signal the type layer uses.

Observed effect abstraction: object deltas mapped through rho to a dynamically grown outcome alphabet

03 · ROWS AND CONTEXTS

Effect rows keyed by type, action, and local context

Object-transitions are filed into rows keyed by type, action, and local context, j = (τ, a, u). Row counts c_j(e) with a symmetric Dirichlet prior give a posterior over the row's effect distribution

q_j ∼ Dir(α₀·1 + c_j)

α₀ is a small pseudocount per effect, sparse rows remain uncertain, and the posterior is closed form

A row that remains mixed after many samples indicates a conflated context: a piece both "stays" and "recolours" under one action because a hidden condition, its lit or unlit pixel state, was pooled away. Splitting on that context u separates the mixture into near-pure rows, implementable as per-type rules f_τ. The Dirichlet form gives calibrated uncertainty from small counts without fitting.

Pooled effect distributions are mixed until the row is split by the piece's own context, unconfounding the rule

04 · ONTOLOGY ERROR

Ontology error η: joint type and effect uncertainty

Two normalized entropies measure what is unresolved: type uncertainty H(τ_i) / log K from the posterior Pr(τ_i | D) over K candidate types, and effect uncertainty H(q_j) / log m, with m the number of observed effects. Both lie in [0, 1]. Ontology error combines them

η = 1 − (1 − H(τ_i)log K)(1 − H(q_j)log m)

the product form drives η to 0 when both type and effects are resolved, and either uncertainty keeps it high

Exploration is prioritized toward high-η rows, where the model mispredicts, rather than re-deriving known dynamics. Over a run the curve falls as posteriors sharpen, is perturbed during CEGIS repair, and declines once the engine replays the buffer. The annotated curve below is the ar25 run.

Anatomy of the ar25 eta curve: initialization, perturbation under CEGIS repair, steady fall once the model is accurate

Observations · what the run artifacts show

01 · η ACROSS ALL 25 GAMES

η trajectories across the 25 runs

One line per run, rescaled to its own η range and trace span, so the graph shows trajectory shape. Hovering a game in the selector (tapping, on a touch screen) transitions the chart to absolute coordinates with level breakpoints, synthesis events, and the run's ending. Three runs (ar25, tr87, vc33) recorded η from a late-engaging extractor onward, after the ontology had settled, and are drawn faded.

won errored level cleared, η valued (on hover) synthesis event (on hover) hover = absolute η, rest = relative shape

Replays · all 25 scored runs, from the released artifacts

Replays of all 25 scored runs, with per-step reasoning and engine state

Each card replays the scored run, including intermediate animation frames. The Reasoning tab follows the playhead and shows the move set the acting agent chose at that step, with its rationale. The Engine tab shows the synthesized world model as of that step. Diamonds on the timeline mark synthesis rounds, and added lines are highlighted against the previous round.

Click a game to open its replay · dots are levels: filled = at or above 50% level score, gray = cleared slower, hollow = budget ended there

LEVELS VS ACTIONS

Levels cleared against cumulative actions, per game

Levels cleared against cumulative actions, as on the official task pages. Black is OPINE-World, gray the human reference, light gray baseline1. A flat tail is budget spent on an uncleared level. Clicking a game opens its replay.

OPINE-World human baseline1

SCORE PER LEVEL

Per-level scores across all 183 levels

More area is better. A level scores min(1.15, (human/agent)²), where human and agent are the number of actions the human baseline and the agent took to clear the level; uncleared levels score 0. ARC-AGI-3 gives no extra credit for clearing a level in far fewer actions than the human (1.15 cap): OPINE-World beats the cap on 109 of its 160 cleared levels, often by large margins the score cannot reflect.

05 · EFFORT HEADROOM · estimate

Room for increased effort

Both agents ran Opus 4.8 at high, the third of five reasoning-effort tiers. With xhigh and max efforts, gains are expected. We show this against the measured scores for the recent frontiers, as unfortunately, we are unable to report our results for higher-effort tiers due to financial constraints.

BUDGET · effort vs performance

Running Opus 4.8 on max is expected to deliver significant gains. Official per-task sessions at arcprize.org, levels cleared across the 25 games. Effort raises Sol from 9 to 48 levels, with the top two tiers adding roughly 60% over high. Anthropic recommends max for this kind of work (announcement).

ERRORS · hypothesizing recovered runs

Some promising runs that failed to win ran into API rate limits well before a 2000 move limit was reached. Due to the runtime and idle limitations of ARC-AGI-3's competition mode, this cut their solutions short. As an upper bound, if each errored game is set to the average score of the rest of the games, our final score would be 92.1. We expect our true score to reasonably lie between these ranges.

Citation

@misc{courtis2026opineworldprogrammaticworldmodeling,
  title={OPINE-World: Programmatic World Modeling with
         Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3},
  author={David Courtis and Wenhao Li and Scott Sanner},
  year={2026},
  eprint={2607.01531},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2607.01531},
}