Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Task file

Each *.yaml or *.yml file in /miroir/tasks/ is a task. Probes can also go in /miroir/tasks/probes/<task-id>.yaml, as a list. Unknown fields are errors.

Top level

FieldDefaultWhat
idthe file nameunique among the tasks
splittraintrain, validation or test
worldthe only worldthe world each reset restores; required with several
personathe only personawho the agent acts as; required with several
instructionrequiredwhat the agent is asked to do
answernonethe normal form of the submitted answer: integer, ids or date
max_steps50the tool calls the episode allows
setupnoneSQL run on the only database at each reset, before the apps start
checksrequiredthe checks, at least one
allowed_changesrequired, or any_changethe net changes the task may make
any_changefalseallow every change, in place of allowed_changes
harm_reward0the reward for harm, from -1 to 0
null_max0the most an immediate empty submit may earn, from 0 to 1
goldenrequireda run that must earn 1, ending with submit
probes[]runs whose reward must stay within bounds
coworkers[]colleagues a model plays; see Coworkers
events[]requests sent at clock minutes; see Clock and events

With several worlds, a world’s tasks are all test, or none are.

checks

FieldWhat
nameunique; scope is reserved
sqla query returning one value: a boolean, or a number from 0 to 1
commanda command that scores what it prints on exit 0 (1 if nothing), and 0 otherwise
appthe database sql reads, when several apps have one
dbepisode: run sql on the episode database instead
weightits share of the reward, in (0, 1]; the weights sum to 1
gatetrue: it must score 1, or the episode is harm
bythe clock minute it is scored at, from 1 to 525,600; weighted checks only

A check has exactly one of sql and command, and exactly one of weight and gate. A check runs for up to 10 seconds. A command check gets MIROIR_EPISODE_DB and MIROIR_DATABASE_URL.

allowed_changes

FieldWhat
tablethe table, schema-qualified outside the default schema
opinsert, update, delete, truncate or schema
columnsfor update: the columns it may change; any when unset
appthe app whose database it is, when several apps have one

Steps

golden and each probe’s steps are lists of tool calls:

- {tool: api_request, arguments: {method: GET, path: /deals}}
- {tool: submit, arguments: {answer: "3"}}

tool is one of the agent tools, a tool of the app’s MCP server, or coworker_reply (with a coworker and a text) to stand for a coworker’s answer. The last step, and only the last, is submit.

probes

FieldWhat
nameunique within the task
stepsthe tool calls, ending with submit
max_rewardthe most it may earn
min_rewardthe least it may earn

A probe has at least one of max_reward and min_reward.

coworkers

FieldWhat
name1 to 64 lowercase letters, digits, _ or -
descriptionwho they are
scriptwhat they know, and how they act
headerstheir headers, by app
inbox.sqla query returning a JSON array of messages, each with an id and a text
inbox.appthe database to query, when several apps have one
replythe request that posts an answer: app, method, path and body, with {text} and {field} placeholders

events

FieldWhat
name1 to 64 lowercase letters, digits, _ or -
atthe clock minute it is sent at, from 1 to 525,600
headersthe sender’s headers, by app
requestapp, method, path and body