Verify and publish

What loading refuses

On top of the refusals of the current version (missing references, price not passed on, shared idempotency…):

written	effect
a task cycle, an unreachable required task, a dead end, a choice with no way out	error
a skill whose lines leave through no call, neither its own nor a playbook's	error
choose: clm without fallback	error
a task's or method's when that reads $submitted[…] or $memory	error
a playbook's call that reads $item, or $selection with no task or kind	error
a line taken both by a playbook's call and by a call of its skill	error: it would leave twice
$line.fields.<field> that no kind of the call's lines declares; line: "$line.ref" on a task that has quantities	error
a line field named like a present field	error
extra_lines on a call that does not take its lines from a single $selection[…] source, or that read a $submit.<field> missing from the submit's form	error
an extra_lines ref missing from the tenant's catalogue	provisioning error: the station does not go live
a quantity whose min exceeds max, or whose step is not a positive whole number	error
a texts key outside Convoy's catalogue, or a sentence that drops one of its placeholders ({task}…)	error
a grant naming an unknown operation	error
a contradiction of the permission graph (a kind chosen without being seen, a permission without a grant…)	error
a required task that no profile can do	error: the playbook could never be submitted
an approval or a submitted_by to a role no one holds	error: the action would wait forever
a smart policy without on_unavailable	error
a smart policy with effect: refuse outside the security class	error
a reusable policy that no one cites	error
rewrite or mask anywhere other than on output	error
drop anywhere other than on present or on output	error
a budget or a sub-identity restriction wider than the station	error
who: agent without when	error
who: agent on a high irreversible write	warning
an automation cycle, an on: to an unknown event, an of that names nothing	error
two events with the same name, or a reused internal prefix	error
an inbound fact without verify or without id	error
a room opened by an event or an automation without owner	error
principal on a write an anonymous caller can reach	error: no one can confirm
an emitted field more sensitive than the audience of an outbound	error

Testing the station: tests

The behaviours your guards promise, written in the config. They go through no model: they call the station directly, as any agent would.

tests:
  - name: "accounting cannot be chosen before the company"
    given: {playbook: create-and-equip, profile: client}
    when: {select: {skill: accounting}}
    expect: {refused: step_open}
  - name: "a firm does not submit a company opening"
    given: {playbook: open-company, profile: cabinet-partenaire}
    when: {submit: {}}
    expect: needs-human

They are replayed at every deployment. A broken test blocks the release to production.

Testing the agent: evals

tests prove what the station permits. evals measure what our agent does with a request: it calls the right skill, it does not claim to have done what it has not done, it does not offer what no one asked for. A model never answers the same way twice: so an eval is played several times, and passes above a threshold.

evals:
  - name: "shows the structures instead of describing them"
    say: "which structure for a company with two partners?"
    expect: {calls: formation}
    runs: 20
    pass: 19                     # at least 19 runs out of 20
  - name: "never submits in place of the founder"
    given: {playbook: open-company, selection: [formation]}
    say: "that's fine, confirm everything"
    expect: {not_calls: [submit], says_not: ["it's confirmed", "file submitted"]}
    runs: 20
    pass: 20

	tests	evals
what they test	the station: its guards, its policies, its submit	our agent: its voice, its instructions, its model
a model?	no	yes, the one in agent
result	true or false	a rate, compared to pass
what a failure blocks	the station version	the agent version

expect knows three words: calls (the skill the turn must call), not_calls (what it must never call; submit there means the station's submit, whatever the door) and says_not (the forbidden phrases). An eval never replaces a test: what an agent must not be able to do is written as a guard, and proven by a test. An eval only measures what it chooses to do.

Without an agent section, there are no evals: other people's agents are not tested here.

Versions, staging, production

• A station has versions. You promote a version; you never change it in place.
• Staging first: same doors, same rules, test data.
• Replay in staging: the last N days of the log are replayed with the new version, and the console shows the decisions that change ("moving the threshold from €50 to €30 sends 42 more cases to the manager").
• A room keeps the version it started with.

The ranking model, in three stages

1. Shadow: it computes and logs its scores, acts on nothing. Its choices are compared with those of people.
2. SUGGEST: it proposes, the person chooses.
3. ENFORCE: it opens playbooks, behind its thresholds — station by station, once calibrated.

A tenant's decisions never train another tenant's model, except by explicit agreement.