skip to content
RELIQUARY

research · article

How to prepare tasks and rewards for LLM reinforcement learning

Start with a pinned checkpoint, a task you can grade, and an evaluation set reserved for the final comparison.

Reliquary ·

An LLM reinforcement learning workload starts with an existing checkpoint, a repeatable task, and a reward that can evaluate the model's attempts. Prepare a versioned task set, test the grader against known outcomes, and reserve separate questions for the final checkpoint comparison.

Decide what you are trying to teach

Pretraining builds the base model, commonly by predicting tokens in a large corpus. Post-training starts from an existing checkpoint and adapts its behavior. Supervised fine-tuning (SFT) learns from example responses. Reinforcement learning with verifiable rewards (RLVR) samples new attempts and scores them using a checkable outcome, such as passing software tests or matching a math answer. SFT can precede RL when useful demonstrations are available. It is a separate stage, and the training setup determines whether to use it.

The Llama 3 technical report describes pretraining and post-training in one model family. The DeepSeek-R1 report documents verifiable rewards and compares approaches with different starting conditions.

Turn one real task into a repeatable reward

Suppose the task is to fix a Python function that removes duplicates while preserving order. Ask the model for a patch, apply it in a pinned test environment, and award a pass when the required tests succeed. Check the case [3, 1, 3] expecting [3, 1], plus an empty list and an input whose correct output is not sorted. Record invalid patches, timeouts, and environment errors separately so a broken runner does not look like a wrong model answer.

Test the grader against a known correct patch and against plausible wrong answers before using it in a run. Sample several attempts from the starting checkpoint and inspect both the outcomes and the errors. With a group-relative method such as GRPO, a group whose attempts all receive the same reward offers no relative contrast on that task. Inspect the distribution instead of assuming a universal pass-rate threshold. GRPO reward documentation

Reserve a final test before you tune

Use separate training, validation, and final test sets. Keep near-duplicates in the same split so the final test stays independent of training and tuning. Compare the starting and trained checkpoints on the same final tasks using the same harness, grader, generation budget, and sampling settings. Report the number of tasks you actually evaluated and explain exclusions. Dataset splitting guidance · Compare two checkpoints on your own questions

For a published example of a shared evaluation protocol, the Teutonic report compares the final Teutonic-I-Reliquary checkpoint with Qwen3.5-9B on 486 SWE-bench Verified tasks. It reports exclusions and distinguishes the final release evaluation from the smaller series used to monitor training. The model lineage identifies the pretrained Teutonic-I model from SN3 and the subsequent post-training on SN81.

Prepare your RL workload

Bring the starting model and revision, a sample of the tasks, the expected output, the grader and its version, the execution limits, and the proposed final evaluation. Review the RL on demand prerequisites. Customer RL execution is planned. If you have a workload in mind, register your interest.