Writing a validator¶
A validator turns a finished run into named metrics. Three modes, one contract.
Mode script¶
Any executable, in any language. One argument: a path to a context file.
score.py /path/to/run/validation/script/context.json
{
"repo": "/tmp/trysquare/rule-vs-ticket_.../a7f3/repo",
"etalon": { "tag": "etalon-v1", "checkout": "/path/to/my-repo" },
"prompt": "/tmp/.../prompt.txt",
"session": "/tmp/.../session",
"trace": "/tmp/.../trace.jsonl",
"cell": "rule / high",
"repetition": 3,
"test_command": "node --test 'src/**/*.test.js'",
"prepare": [],
"artefacts": ["node_modules"],
"touched": ["src/basket.js"],
"files": ["README.md", "src/basket.js", "src/theme.js"],
"given": ["src/probe.test.js"],
"declared": ["in_scope", "delivered", "tests"]
}
Read it with trysquare.assay rather than by hand - run.touched, run.etalon,
run.sources_at_etalon("src/*.js"). Four names cover a whole validator, and the module
carries the error contract with them. examples/validator.py is a complete one written
that way, and the suite runs it against tests/fixtures/tiny so it cannot go stale.
touched and files are computed by the harness. Three validators written before the
base existed each reimplemented the first with a raw subprocess, one of them landing
on a different answer for the reference side, while repo.changed_files had held
that knowledge all along. A fact computed in one place cannot be got three slightly different ways.
declared is what the scenario contracted for, so a validator can be told which metric it
forgot before anything is recorded rather than after the tokens are spent.
Subtract the by-products before scoring scope¶
artefacts is what the scenario declared as the leavings of running the task. Subtract
them, and score the remainder:
work = run.touched - run.artefacts
Warning
Skipping this cost a whole matrix. An agent asked to fix a defect ran the declared suite
to check itself, which left __pycache__/*.pyc in the clone; scope scoring counted the
bytecode as its work, and in_scope was false in every run of every cell. The
criterion saturated at zero and nothing was concludable.
And it is not random noise: the runs that scored out of scope were the ones where the agent verified itself. The metric ended up anti-correlated with the behaviour it was meant to reward.
run.artefacts is the subset of run.touched matching those patterns, and touched
itself is never reduced - the filter decides what a verdict rests on, not what is
recorded. Both metrics of the shipped example need the subtraction: delivered too,
since bytecode is not a delivery.
Tell what was given from what was written¶
given is what a files brick
put in the tree before the agent started, and it is empty in every cell that was handed
nothing. Read it whenever a metric depends on that material still being there:
if FICHIER not in run.given:
return Metric.unjudged("no probe was given to this cell")
An absent file is one fact with two causes - never given, or deleted along the way -
and without this key a run that removed the test it was handed scores exactly like a
run that was handed no test at all. Given files are committed on top of the etalon, so
an edit to one appears in touched like any other change, and comparing the tree’s
copy against the brick’s says whether it was weakened.
It is withheld from a judge’s context: naming a brick names the configuration.
A scenario that declares nothing gets an empty set, so this line is safe to write everywhere.
Score a skill on both ways it arrives¶
A metric of process reads the archived session, so it replays: run.tool_calls() is
every call the agent made, and run.first_write(path) the index of the first one that
wrote a given file.
A skill needs both halves, because the two ways of loading one leave opposite traces.
Loaded by name with a skills brick,
only its name and description enter the system prompt, so the agent has to read the
SKILL.md itself - and may not. Referenced as /skill:<name> in the prompt, the agent
expands it before the first turn and the whole body arrives in the message, so there is
nothing left to read and no call to count. run.skills_expanded names the ones that
arrived that way:
opened = {c for c in run.tool_calls() if not c.failed and reads_a_skill(c)}
return Metric(bool(opened or run.skills_expanded), ...)
Warning
Counting only the call scored “the skill was never opened” for five of five runs of a cell whose prompt carried the body from end to end - while the cell next to it, loading the same skill by name, scored three of five. Read as written, that column said forcing a skill into the prompt stopped the agent using it.
The two are worth separating in a scenario, and not in the metric: “did the agent go and fetch it” has no answer in a cell where the operator answered on its behalf.
Important
The context is handed as one file, and it is archived with the run. That is why this form was chosen over environment variables or named options: a validation replays by hand months later with the same command and the same file, which is what makes “fix a signature and re-score runs already paid for” true.
etalon.checkout is a path, not just a tag, because scoring needs the reference side
as text and not every validator can run git.
test_command is the suite that decides the tests metric, declared by the scenario and
never guessed here. It is carried as the scenario wrote it, so a context read against the
scenario file says the same thing, and so a validator gets the shape its own runtime prefers:
a shell splits a string for free, where a JSON array would have to be parsed and rebuilt.
No shell runs it, though - the loader refused any word that only means something to one. In
Python, trysquare.scenario.split_command() is the rule the loader vetted it with, and
run.tests() already uses it. Elsewhere, $cmd unquoted does the same job.
prepare is what has to run before the suite - usually nothing. Its failures mean
something else: no network or a dependency that will not install says nobody judged, so
the metric is unjudged rather than false. The suite failing is a measurement.
Warning
Do not fall back to npm test, or to any command read out of the repository. Its
meaning lives in package.json, which is inside the perimeter the measured agent may
edit: broken code plus a test script of echo ok scores green, and nothing in the
output says so. The scenario declaring the command is what stops the agent being
measured from choosing how it is measured.
The key is absent when the scenario names no suite, and an absent key is a different fact from an empty command. It says this experiment scores no test suite - which is something to refuse over, not something to score as a failure.
A minimal validator, by hand. Any executable in any language may do this, and this is what the contract is:
#!/usr/bin/env python3
import json, sys
from pathlib import Path
context = json.loads(Path(sys.argv[1]).read_text())
repo = Path(context["repo"])
metrics = {"has_readme": (repo / "README.md").is_file()}
json.dump({"metrics": metrics, "reasons": {}}, sys.stdout)
In Python, write it with trysquare.assay instead. The same validator, plus the
error contract, plus a reason attached where the value was found:
#!/usr/bin/env python3
from trysquare.assay import Assay, Metric, validator
SCOPE = frozenset({"counter.py"})
@validator
def evaluate(run: Assay) -> dict:
outside = run.touched - SCOPE
return {
"delivered": bool(run.touched),
# The reason belongs to the failure. Attached unconditionally it reads
# `also touched ` on every run that stayed in scope; an empty reason is dropped.
"in_scope": Metric(
not outside,
f"also touched {', '.join(sorted(outside))}" if outside else "",
),
"tests": run.tests(),
"touched": run.touched,
}
if __name__ == "__main__":
raise SystemExit(evaluate.cli())
The whole thing is in examples/validator.py, and tests/test_example.py runs it against
tests/fixtures/tiny on every CI build - so unlike a snippet in a document, it cannot rot.
It is also where to see the pattern that matters: wrapping a metric the run cannot answer
so that one unanswerable metric does not refuse the run and take every other metric
with it.
Make it executable. The contract says “any executable”, so the bit is part of it.
Note
The working directory is deliberately not the measured clone: a validator that wrote a stray file there would be counted as the agent’s work by scope scoring. Every path it needs is absolute in the context.
Failure¶
Exit non-zero, print unreadable JSON, exceed the timeout, or omit a declared metric,
and the run becomes invalid. Its stderr is kept in that run’s
validation/script.stderr, the cell is not publishable, and the matrix continues.
Mode judge¶
For what no script can score.
[[validation]]
mode = "judge"
provider = "ilaas" # pinned, and distinct from the model under test
model = "gemma-4-31b"
thinking = "off"
rubric = "rubric.md"
pieces = ["prompt", "response", "diff"]
metrics = ["note_usable", "cites_paths", "says_what_is_missing"]
A judge that is the model being judged is not a judge, so pin it separately.
The verdict is a tool call, not parsed prose¶
The agent offers no schema option and no response format, so prompt discipline plus parsing would be the fallback - and it is the wrong one. A judge answering in prose with a stray code fence produces an unreadable verdict, and an unreadable verdict is indistinguishable from a negative one unless something catches it.
So trysquare/judge-tool.ts, shipped inside the package, registers a verdict tool
whose parameters are built from the metrics the scenario declared. The runtime validates
the call before it reaches the harness.
The format is guaranteed. The call happening is not. A judge that never calls the tool leaves the run invalid rather than scoring zero - a broken judge must not read as a negative verdict.
pieces¶
What is assembled into the judge’s prompt, and it is declared because what the judge is given to read is half of what it measures.
promptthe task the agent was given
responsethe agent’s final prose, not its transcript. A judge asked whether a note is usable must score the note, not the work behind it.
diffwhat the agent changed
Blinding¶
The judge’s context carries no cell name and no configuration. But blinding is not always achievable, and the harness says which pieces leak rather than pretending:
judge: blind over 4 cells (pieces: prompt, response, diff)
! judge is only partially blind: prompt varies between cells,
so the judge can identify the treatment
When the treatment is the prompt, handing the judge the prompt reveals the cell. That is allowed, provided it is said.
One call, one vote¶
One judge call per run, returning every metric at once, and no repetition of the judge. The matrix already repeats, so the judge’s noise lands in the cell’s dispersion where the resampling accounts for it.
A majority vote would smooth that noise away and leave the dispersion unable to say whether the instability came from the agent or the judge. A noisy instrument should cost certainty, visibly.
Writing a rubric¶
State the failure conditions, not only the successes - a judge given only what good looks like rewards confidence. And tell it what to do when the material is empty:
If the material is empty or unreadable, still call the tool: set every metric
false and say so in the reasons. A missing verdict is treated as a broken judge,
not as a negative one.
Mode form¶
A human filling a blind, shuffled TOML file. See Command line reference.
A manual metric may fill a metric nothing else produces. It may never overwrite a measured one - that is “the harness computes the verdict, the author does not work around it” applied to the interface.
Composition¶
Validators are independent: each receives the same context and cannot see what the others found.
That is the whole point. A judge told the script’s verdict is anchored on it, and its agreement stops being an independent signal - which was the only reason to have it.
Two validators cannot declare the same metric; the collision is refused at load, before
any measurement. Rename one (overflow_judge).