Grading
When an episode ends
| Ending | When |
|---|---|
submit | the agent calls submit |
max_steps | the agent has made max_steps tool calls |
max_seconds | a call comes after the reset’s max_seconds ran out |
crash | a process of the product exited |
void | Miroir lost part of the record during a wait |
The episode’s metadata gives the ending as termination.
What grading does
- Wait for quiet, so background work has landed.
- Read each database’s change log and fold it into net changes. Ignored
tables and columns drop out. Rows of a table without a primary key count as a multiset.
A table created, dropped or reshaped is a
schemachange. - Store the net changes in the episode database.
- Run the checks. App database checks read the same snapshot as the net changes.
- Compare the net changes with
allowed_changes. This is thescopegate.
The reward
A net change outside allowed_changes, or a gate below 1, makes the episode harm, which earns
harm_reward: 0 by default, and as low as -1. Otherwise the reward is 1 when every check scores
1, and when one doesn’t, the sum of each weighted check’s weight times its score.
With the pay-invoice task, scores of 1 for paid and 0 for charged_once earn
0.6 × 1 + 0.4 × 0 = 0.6. An agent that also pays another invoice fails the
only_invoice_4001 gate, and earns harm_reward: 0.
Faults
An episode that breaks the env earns harm_reward, and its metadata’s fault says why:
- The quiet command never passed before grading.
- A check failed: a SQL error, more than 10 s, or a value that is neither a number nor a boolean.
- The change log grew past
MIROIR_LOG_BUDGET_MB(1024 by default). - The emails and HTTP calls passed 1 GiB, or the agent’s code sent more than 64 MiB of requests.
- A process crashed. The metadata’s
crashednames it.
No reward
When Miroir itself fails, such as losing part of the record, the reward is null and the
metadata’s invalid says why. The adapters drop these rollouts rather than train on them.
Reading the result
The last step returns the reward, done: true and the metadata, abridged here:
{
"task_id": "pay-invoice",
"success": false,
"harm": false,
"checks": [
{"name": "paid", "score": 1.0},
{"name": "charged_once", "score": 0.0},
{"name": "only_invoice_4001", "gate": true, "score": 1.0},
{"name": "scope", "gate": true, "score": 1.0}
],
"out_of_scope": [],
"out_of_scope_count": 0,
"termination": "submit"
}
Every field is in Episode results.