Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Review grades

Checks can be wrong: too strict, or too easy to fool. miroir review shows you how rollouts actually ended, asks whether each grade is right, and turns each wrong grade into a probe, so miroir check keeps it fixed.

miroir review traces/ --tasks fixtures/acme-crm/tasks

traces/ holds the records the adapters save with traces= (TRL) or --env.taskset.traces (verifiers): files, or directories of them.

How it goes

Rollouts that ended in the same state share a signature, so review groups them: one end state per task and signature, however many rollouts reached it. For each, it shows the rewards, the net changes and effects, the check scores and the calls of one example, then asks:

label: [c]orrect, should [f]ail, should [p]ass, [s]kip, [q]uit?
LabelMeansWrites
correctthe grade is rightthe label
should_failit earned too muchthe label, and a probe with max_reward set to the task’s null_max
should_passit earned too littlethe label, and a probe with min_reward: 1

The probe replays the example rollout’s own calls. It goes in tasks/probes/<task-id>.yaml, and the labels in tasks/probes/labels.jsonl, so labeled end states are not asked again. At the end, review prints how many grades you agreed with.

Fix the checks, then run miroir check: the new probes fail until the grade is right.

Options

OptionDoes
--label TASK@SIGNATURE=LABELlabel an end state without asking; repeat for more
--jsonprint the queue of unlabeled end states as JSON, and ask nothing

Rollouts that broke the env, or that have no grade, have no end state to label. Review counts them per task instead.