live training · extended eval · DAPO metric layer
Every published checkpoint, benchmarked.
On a fixed cadence, the grader scores the latest published checkpoint against a frozen suite, math and code, that it never trained on. The charts below report pass@1 (correct on the first try), pass@G (the ceiling across G attempts), the headroom between the two, and the in-zone population — problems solved only sometimes, the pool still worth training on. A score here measures the checkpoint; it does not, by itself, show that training caused the difference.
math — training
best conversion – at ckpt –
chart 01 · headline
Mean accuracy vs best-of-G ceiling — math and code
Solid = avg@G (mean accuracy); dashed = the pass@G ceiling (at least one correct in G). The shaded band between them is the headroom the selector is meant to convert, and it narrows as the model converts it. The gold line is the math and code mean. Shaded checkpoint spans were never evaluated; the curves break there rather than draw through them.
chart 02 · in-zone fuel · math
Teachable fuel rarefies — and the RPI climbs
The amber in-zone fuel (0<k<16, still learnable) shrinks as the model masters the suite. The white RPI (Reliquary Prospecting Index = Δmastered × fuel₀/fuelₜ) climbs: capability keeps being extracted from an ever-scarcer pool.
chart 03 · conversion · math
Headroom conversion
How much base headroom turned into reliable skill. Above 1.0× the model surpassed its own base best-of-16 frontier — genuinely new capability.
chart 04 · distribution · math
k-histogram heatmap — where the probability mass lives
A density grid: each column is one checkpoint's distribution of k = #correct out of 16, each row a value of k (mastered k=16 at top, dead k=0 at bottom). Cell brightness is the probability mass; watch it migrate upward — from the red k=0 band to the green k=16 band — as training advances.
chart 05 · learning progress · math
What the model learned that it couldn't do before
Against the base checkpoint on the frozen suite: newly solved (once impossible, now reachable, as a share of the once-unsolved pool), regressed (once reachable, now impossible), and net learned across the whole suite.
chart 06 · reasoning allocation · math
Where the model spends its thinking
Median completion length against difficulty (k = #correct out of 16). A model that reasons well spends its budget on what it cannot yet do and stops early on what it has mastered — a curve that descends from k=0 to k=16. The amber band is the p25–p75 spread; the dashed line is the earliest measured checkpoint.
Read the red bars first. A rollout that hit the token cap is long by definition, so a model looping until it is cut off draws the same tall k=0 bucket as one genuinely thinking harder. Buckets ringed in red are majority-truncated: their length is the cap, not a choice.
requires per-rollout completion lengths; cannot be backfilled from stored rewards
chart 07 · allocation over training · math
Where the effort goes, checkpoint by checkpoint
Chart 06 shows the shape at one checkpoint. It cannot say whether the shape is improving, because two points always describe a direction. This tracks the effort focus — tokens spent on prompts it never solves divided by tokens spent on prompts it has mastered — across every measured checkpoint.
Focus rising means effort is concentrating on what the model cannot yet do. Focus flat while both raw medians rise means every answer got longer and none got better targeted. A mean completion length cannot tell those apart; this ratio can — subject to the caveat under the chart, which is why what follows is a measurement and not a verdict.
The checkpoints were measured in separate campaigns. The grey band is the stretch nobody evaluated, compressed and labelled with the span it hides; all three curves break there rather than draw through it. Each campaign then carries its own amber band: one ordinary checkpoint-to-checkpoint step for that campaign, laid around its starting value. It is a ruler, not an envelope — the curve is free to wander outside it, because a single step is exactly that big. What a campaign is worth is read off where its last checkpoint (ringed dot) lands. Inside the band means that whole campaign is worth less than one step of noise, however dramatic the axis makes it look. Ends of different campaigns are not subtracted from one another.
requires per-rollout completion lengths; cannot be backfilled from stored rewards
chart 08 · accuracy per reasoning token · math
What the accuracy cost
There is no single accepted metric for reasoning quality. The defensible reading is multi-axis — outcome, process validity, faithfulness, robustness, efficiency — and this run measures two of those: outcome and efficiency. So those two are what is plotted, and the rest are named here rather than implied by the axes that exist.
Each point is one checkpoint at what it scored against what it spent. Moving up and to the left is the model getting better. Moving up and to the right is the same answers bought with a bigger budget — which a chart of accuracy over time would have shown as pure progress. Ringed points are on the Pareto frontier: nothing measured here is both cheaper and at least as accurate.
The vertical rule through each point is its 95% accuracy interval. Where those overlap, the accuracy axis is not resolving the difference between two checkpoints and only cost separates them.
Not measured on this run, and therefore not claimed: whether the intermediate steps are individually valid (that needs the trace split into atomic steps and graded by an independent verifier), whether the trace actually caused the answer rather than being written after it (that needs causal interventions — delete a step, corrupt it, paraphrase it — and a check that the answer moves), and whether any of this survives out-of-distribution prompts. A checkpoint can reach these answers through broken or post-hoc reasoning without moving either axis below.
the cost axis needs per-rollout completion lengths, which cannot be backfilled
archive · every run published