skip to content
Reliquary

forge · training

The training loop

Healthy balanced batches can become GRPO steps. The validator trains only on rollouts that survive verification—PPO-clipped and KL-penalized against a frozen reference—and publishes every four successful balanced optimizer steps, with earlier publication on behavior-policy drift.

FIG.01 · the GRPO loop · selection to safe publication
VERIFIEDsurviving rolloutsRECOMPUTErewards, validator-sideGRPO STEPPPO-clip + KLCHECKPOINTsigned manifestHUGGING FACEpublished weightshealthy balanced batches train · publish every 4 successful steps

1 window

one possible GRPO step

10 windows

cadence to publish

frozen ref

KL-penalized policy

/checkpoint

signed custody chain

FIG.02what each stage does · the parts the loop can't draw
  • Market-selected batch. A window seals once enough valid distinct-prompt rollout groups land. The validator recomputes every reward itself, then assembles the GRPO batch from the survivors — never trusting a miner's reported score.
  • The GRPO step. A PPO-clipped surrogate loss with a KL penalty against the frozen reference, run on healthy full batches. Selected windows that look suspicious are quarantined from training rather than poisoning the gradient.
  • Published checkpoint. Updated weights push to Hugging Face (ReliquaryForge/qwen3.5-2b-reliquary-v3) every four successful balanced optimizer steps, with earlier publication on behavior-policy drift. The signed /checkpoint manifest records the chain of custody from the base model through every step.
▸ live cycles · /forge/cycles

outer-loop telemetry · team-only

The GRPO step emits per-step telemetry —loss_mean_window,grad_norm,advantage_variance,kl_vs_reference — to Weights & Biases. It renders on the team trainer dashboard, which requires credentials and is not a public surface. We draw no chart here on purpose: inventing training curves would defeat the protocol.

The public surface is the loop's output — the verified rollouts, the verdicts, and the signed checkpoint manifest. That's what FIG.03 below lets anyone re-run.

▸ team · live trainer dashboard
FIG.03 · verify the loop yourself · public artifacts only
R2 DATASETrollouts the trainer sawVERDICTSper-window accept/rejectMANIFESTsigned /checkpoint chainpull · re-derive · walk the chain — no credentials required
  1. 01 · pull

    Pull any recent window's sealed archive via /api/r2/window/<id> — the exact prompts + per-rollout commitments (merkle roots, GRAIL sketches) the trainer sealed.

  2. 02 · re-derive

    Each entry carries checkpoint_hash + grail_sketch. Re-derive the sketch from the announced policy and confirm the rollout came from that checkpoint.

  3. 03 · walk the chain

    Walk the signed /checkpoint manifest to trace every published checkpoint back to the base model — see /docs/scoring for how survivors turn into weight.