trysquare

A try square tells a joiner whether a joint is true. This one tells you whether a measured difference is.

A scenario harness for measuring coding agents reproducibly.

One scenario is one self-contained experiment in one TOML file: the task, the configurations to compare, the protocol, and the validation. The harness runs it, scores it, and refuses to publish a difference that does not survive resampling.

Why this exists

Measuring an agent is easy to get wrong in ways that look right.

The tool this one replaces produced six published conclusions that collapsed on rerun, and twenty catalogued defects. Most were variations on a single mistake: a run that did not do the work reading as a run that worked well. A provider cuts the stream, the agent retries and returns turns that are real but empty. That run breaks no rule, touches no file, and fails no test - so a naive harness records it as exemplary.

Every rule in this tool is a defect that was paid for. The documentation says which one, because a rule whose reason is lost gets removed by the next person who finds it inconvenient.

Start here

Getting started

Install, write a skeleton, plan a run for free, measure, read the output.

Getting started

Core concepts

Scenario, cell, brick, validator, verdict. The vocabulary the rest assumes.

Core concepts

Writing a scenario

Grids, variants, bricks, protocol. Every key explained.

Writing a scenario

The invariants

The eight rules that make a number publishable, and the defect behind each.

The invariants

Cheat sheet

Every command, every flag, and what it costs, on one page.

Cheat sheet

Reference

User guide

• Getting started
  • Install
  • Start from a skeleton
  • Point the config at your repository
  • Check it, before there is an output directory
  • Plan a run without spending anything
  • Measure
  • Read the output
  • Rebuild a table without remeasuring
  • Next
• Core concepts
  • Scenario
  • Etalon
  • Cell
  • Brick
  • Validator
  • Metric
  • Verdict
  • How they fit together
• Writing a scenario
  • Start from the question
  • Pin the task
  • Declare the agent completely
  • Declare the protocol, including the load
  • Choose grid or variants
  • Declare the bricks once
  • Declare the validation
  • Choose the criterion, the reference, and the validity
  • Check it before spending
• Writing a validator
  • The shared contract
  • Mode script
  • Mode judge
  • Mode form
  • Composition
• The invariants
  • 1. A run counts only if it consumed tokens
  • 2. Nothing that changes a measurement may be inherited
  • 3. A validator that could not judge never yields a verdict
  • 4. Two verdict states, and only two
  • 5. Repetitions are declared in advance
  • 6. What the harness injects is excluded from scoring
  • 7. A judge is blind, and where it cannot be, the harness says so
  • 8. Durations compare only within one matrix
  • 9. A run that was cut short is not recorded at all
  • And one that is not a rule but a habit
• Parity with the previous bench
  • Neither tool is the reference
  • Running it
  • Layer 3, in the test suite
  • Layer 1 is a real test, not a tautology
  • Layer 2 re-scores the archive, and needs a scenario
  • Layer 4 is a smoke pass, not a comparison
• Troubleshooting
  • “these files the scenario references do not exist”
  • “init never overwrites”
  • “refused: the scenario declares thinking = … “
  • “these agents declare no model”
  • “metric declared by two validators”
  • “value ‘x’ declares no delta”
  • “reference cell … has no run left to compare against”
  • “pi is not on PATH”
  • Runs marked empty
  • A validator failed
  • “this scenario names …, and you asked to re-score …”
  • A validator refuses “the context carries no …”
  • The synthesis warns about cost columns
  • No synthesis was written
  • “--only names no cell of this scenario”
  • “these cells changed since their runs were measured”
  • A harness clone failed
  • A repository URL could not be cloned
  • A resume against a URL wants the network again
  • “[repos] has no entry”
  • “no trysquare.toml was found”
  • A repository path does not exist
  • “refused: different etalons”
  • No progress bar appears
  • The cursor is gone after a hard kill
  • Stopping a matrix

Reference

• Cheat sheet
• Command line reference
  • init
  • run
  • validate
  • render
  • replay
  • compare
  • parity
  • form
  • Exit codes
• Scenario schema
  • [scenario]
  • [task]
  • [agent]
  • [protocol]
  • Cells: [axes], [values], [variants]
  • [harness.<name>]
  • [[validation]]
  • [verdict]
• Config schema
  • [repos]
  • [harness]
  • [defaults]
  • Absent config
• Outputs
  • Two layouts for runs/
  • The directory name is the guard
  • state.json
  • measures.json
  • synthesis.md
  • synthesis.html
  • One run’s directory
  • Where clones and sessions live
  • A known gap in the naming scheme
  • Not implemented yet
• Python API
  • The pure core
  • The effectful shell

In one page

pip install -e .                                   # or: uv sync
cp trysquare.toml my-trysquare.toml                      # point [repos] at your repository

uv run trysquare run my-scenario.toml -o out --dry-run     # plan, spend nothing
uv run trysquare run my-scenario.toml -o out               # measure

The shape of a scenario

[scenario]
name = "rule-vs-ticket"

[task]
repo = "my-repo"             # a logical name, resolved by the config file
etalon = "etalon-v1"         # a tag, cloned; never the working tree
prompt = "tickets/vague.md"  # relative to the scenario; inline text works too

[agent]
provider = "ilaas"           # mandatory, never inherited
model = "gemma-4-31b"        # mandatory
thinking = "off"             # mandatory

[protocol]
repetitions = 10             # declared in advance
concurrency = 5
timeout = 900

[axes]                       # a grid; declaration order fixes the table's order
context = ["nothing", "rule"]
thinking = ["off", "high"]

[values.context.rule]
context = "AGENTS.md"

[values.thinking.high]
thinking = "high"

[[validation]]
mode = "script"
command = "score.py"
metrics = ["in_scope", "delivered", "tests"]

[verdict]
criterion = "in_scope"
reference = { context = "nothing", thinking = "off" }
validity = ["delivered", "tests"]

Requirements

Python >= 3.11: TOML parsing is tomllib from the standard library, which is why the floor is there. uv sync - or pip install -e . - installs the rest.

Note

Measuring anything also needs the agent binary (pi) on PATH and a provider you have access to. Everything else - loading, scoring, aggregation, verdicts, the parity checks - runs offline, which is why the methodological rules have tests at all.

Status

Working and tested, with gaps that are listed rather than left to be discovered. See Outputs for what is produced today and Not implemented yet for what is not.