skip to content
Reliquary
RELIQUARY
BlogDocsOpen App Open Console

research · article

Teutonic-I-Reliquary: benchmarks and asynchronous RL run

A 9B model post-trained on Bittensor. The final checkpoint, the eight published benchmarks and the conditions behind each score.

Reliquary ·

Run dates
22 Oct – 2 Nov 2026
Optimizer steps
158
Trained tokens
15.7B
GPU models
11+

Final checkpoint benchmark results

Teutonic-I-Reliquary scored 43.6% on 486 SWE-bench Verified tasks, compared with 35.1% for Qwen3.5-9B under the same harness and protocol. It scored higher on five of the eight published evaluations; Qwen3.5-9B scored higher on GPQA Diamond, MMLU-Pro and TMax.

Pass@1 (%), with tasks and samples per task for each model.
BenchmarkTeutonic-I-ReliquaryQwen3.5-9BTasks × samples
SWE-bench Verified43.6%35.1%486 × 2
AIME 202591.2%86.2%30 × 8
AIME 202689.6%86.7%30 × 8
GPQA Diamond68.1%81.6%198 × 4
LiveCodeBench62.0%57.0%300 × 2
IFBench74.7%67.5%300 × 2
MMLU-Pro79.1%80.3%500 × 2
TMax (terminal)74.8%82.2%200 × 4

Model lineage and subnet roles

SubnetContribution
Teutonic SN3Provided the pretrained Teutonic-I base model.
Reliquary SN81Miners generated the SFT data, RL rollouts and evaluation samples. The network verified rollouts and evaluation samples.
Lium SN51Provided compute for the supervised fine-tuning and reinforcement learning trainer.
Targon SN4Provided the sandboxes used for software engineering and terminal tasks.

See the published model card and the SFT model card for the checkpoint lineage.

Training and evaluation protocol

Supervised fine-tuning used about 4.5 billion tokens generated by Reliquary miners. During asynchronous reinforcement learning, independent miners generated rollouts across 11+ GPU models and seven environments. The trainer completed 158 optimizer steps in 11 days.

In this asynchronous RL run on heterogeneous GPUs, the trainer published a checkpoint after each step. Miners on consumer and datacenter cards kept generating without waiting for each checkpoint. The network computed the rewards and verified rollouts before they could be used for training. The model card describes the run and identifies Gumbel substitution as the correction method.

Both models were evaluated on the network with matching harnesses, budgets and sampling settings: temperature 1.0, top-p 0.95, top-k 20 and thinking mode enabled. SWE-bench Verified used the mini-swe harness. SWE and TMax allowed up to 48 turns and 16,384 tokens per turn. Missing or truncated answers counted as failures.

What these scores cover

This SWE-bench Verified evaluation used 486 of the 500 tasks. Seven required network access; another seven failed in the sandbox with the reference patch. TMax used a held-out split. The results apply to the published tasks, settings and checkpoint.

The console also tracks a separate 100-problem SWE series during training. Use the table above for the release comparison and the console for progress across checkpoints.

Weights, source and run archive