Harness query

Use this when a Built Harness already exists and the operator wants to see it answer something. It builds nothing: no Builder session, no campaign, no adoption decision. Making a new harness is bun run fullrun -- --prompt "...", a different and much larger spend.

How it works

Every bundle keeps the starter-pack layout (starters/pi-built-harness): agent/ plus correctness-model/tasks.json and correctness-model/controls.json, recorded by an optional conformance.json. Bundles recorded before 2026-08-20 spell the model directory grader/; the script reads either. Because the layout is shared, one wrapper serves every bundle. The script runs four steps:

1. select the tasks — recorded ones you name, or written queries wrapped into task shape;
2. derive a probe bundle — a copy in .scratch/harness-query/<runId>/ made self-consistent for exactly that battery;
3. measure through measureHarness from the tree under test, shipping-only, judge off — the same function the controller calls for a climb battery, so the solve path, isolation wall and host verifier are the product's own;
4. report per-case verdicts, the artifact paths, and harness-query.json.

Only step 2 knows the bundle layout. Three product guards would otherwise refuse a narrowed battery, and the derivation answers each honestly instead of weakening a check: the copy's tasks.json becomes the selection (its own honest task-set hash), conformance.json is dropped (the claim then says conformance-unprobed), and the control corpus keeps only rows bound to tasks that still run. The adopted tree is never written to; the guards stay untouched in source.

Which harness. Name a bundle by its adopted version (campaigns/<campaign>/versions/<name>), not by an epoch commit: a Builder session's gate commits usually carry the harness it started with. The adopted versions are the directories under campaigns/<campaign>/versions/; order them by recorded commit time, never by directory mtime. An adopted version whose .toolchain link dangles (the epoch trees were removed on 15 September for the truss project) cannot be selected by the product until the tool tree is back.

Usage

Find the bundle (a directory with agent/ and correctness-model/, or grader/ in older bundles, usually domains/<slug>/), then list — free, no model turn:

bun .claude/skills/harness-query/scripts/harness-query.mts \
  --harness domains/<slug> --list

Read each task's publicInput keys before writing a query: most bundles take a structured object, not free text. Then invoke — an unnamed selection is refused rather than spending the whole battery by accident:

bun .claude/skills/harness-query/scripts/harness-query.mts \
  --harness domains/<slug> --task <taskId> --model claude-opus-5 --effort medium

Selectors: --task <id> (repeatable), --family <name>, --all, --query <text or json> (repeatable), --queries <file>. Pins: --model, --effort, --max-built-turns. --json prints the machine-readable report. Run the helper through a prepared checkout's scripts/worktree.sh run, using the repository's pinned Bun, so the dynamic import binds the tree under test.

The judge

The battery runs with no reviewer unless --judge names one, and each backend lives in its own file under judge/, so switching is a flag rather than an edit:

--judge	reviewer	spend
off (default)	none; the battery records judge:"off"	none
configured	the repository's own review slot, resolved exactly as a real battery resolves it	that account's credential

--judge-model and --judge-effort name the reviewer; judge/README.md describes both backends. Judge verdicts stay advisory here as everywhere: the host verifier decides each pass, and this skill writes no claim.

Measured: one recorded task of consumer-hardware-f8714546 on claude-opus-5 at medium effort took 43 seconds and returned 1/1 graded pass.

Written queries

--queries <file> reads the shapes people paste, without a format flag: a JSON array or object, one JSON object per line, a CSV with a header row, or one plain query per line. A row may be a whole task (taskId, family, publicInput, hidden) or a bare publicInput, which gets a generated id, the family from --family or the bundle's first task, and no hidden expectations.

A harness checks the shape it was authored for, so a written query is fitted to that shape before it runs. It borrows the structure and omitted values from a recorded task in its family: free text goes into the template's text field (objective and similar, or the first string field), an object's own fields override the template's, and a template caseId is replaced by the query's own id. What was filled is printed before the run.

--fit chooses the behaviour: template (default, as above), strict (refuse a query that does not already match, naming the fields it lacks), and raw (send exactly what was written). Use raw to probe how the harness behaves on an input nobody authored — that is what surfaced the oracle defect below — and template when you want the query answered.

A written query has no controls, so recorded tasks accompany it as anchors. The script selects one task carrying both accept and reject controls, or two when those controls belong to different tasks. Each anchor costs an extra solve. These controls exercise the recorded task; they do not calibrate the new query's expected answer.

Shaping can satisfy the input schema; it does not establish correctness or answerability. Measured on 2026-08-14: a firmware plane-radar query was written into the objective of a desktop-parts task and passed 2/2, but the artifact it produced was a desktop build — cpu, board, ram, gpu, psu — because the template's catalog, requirements and rules are what the agent actually solves against. The injected text moved almost nothing. Read a templated pass as "the harness completed its own kind of task", never as "the harness answered my question".

That is the honest limit of this skill: a harness is a domain and its grader together, so a query outside that domain has no gradeable meaning inside it. To get a real answer to an out-of-domain question, build a harness for that domain with bun run fullrun.

State the evidence limit plainly. Written queries default to hidden: []. Under the current check-program/v1 contract, a required hidden operand that is missing refuses execution; it does not remove that check from the denominator. Older bundles used different applicability rules. In one historical run the anchor failed against its hidden expectations while the written query returned {"ok": true, "issues": []} — same harness, same model, one verdict carries discrimination and the other does not. When the question is "can this harness do the job", use recorded tasks; use written queries to see behaviour on a shape nobody authored.

Expect a first refusal on an unfamiliar bundle

The Harness Builder authors each bundle, and the layout details drift between generations: one bundle binds every control to a task, an older one binds none; hidden expectations, families and input shapes all vary. So a first invocation of a bundle generation this script has not seen may be refused. That is the working loop, not a fault:

1. run it; if it throws, read the message and the record it points at (runs/<runId>/battery.json in the probe copy holds terminalReason and findings);
2. compare what the guard expected against what this bundle's model files actually hold — the mismatch is always visible in those bytes;
3. adapt the derivation or selection in the script (deriveProbeBundle and selectTasks are the two places), keeping the rule: make the probe copy honest, never weaken a product guard;
4. rerun. Selection mistakes cost nothing — the product refuses before any model turn is spent.

Two adaptations are already built in from bundles measured so far: a task-bound control corpus is narrowed with the battery and written queries get recorded anchor tasks, while a task-unbound corpus (older generation) is carried whole and needs no anchor.

One drift the script cannot bend around: a brief written in an older predicate vocabulary (for example a controlReference field the current validator refuses) fails validation before any turn is spent. Rewriting the brief would change the grader, so do not. Either point --repo at the source generation that adopted the bundle, or pick a bundle the current tree accepts — check correctnessContract and the declared check inputs in the brief. The absence of one retired field does not establish compatibility with the current contract.

One drift that is not the bundle's fault: its declared tools may not exist here. Every committed bundle links .toolchain into a campaign workspace that has since been removed, so tool resolution falls through to the host PATH. A bundle whose checks name only clang++ measures normally; one naming platformio resolves nothing, every applicable check throws, and the control census records 29 of 29 controls as non-result (verifier-throw) and refuses with DISCRIMINATION_NOT_PROVEN. That reads as a bundle defect and is a missing program. The script now names the unresolved tool ids and refuses first, before the probe copy and before any turn. Install the tool or measure a bundle whose declared tools this host has.

Recorded bundles written before the solve-side package was renamed to @ana/agent-bundle kept the old specifier, so fingerprintSlug refused every one of them with unvetted-import before a turn was spent. Those bundles were rewritten in place on 18 September rather than shimmed at read time, so the name on disk is the name that resolves. The guard keeps its full strength: a bundle importing anything outside the vetted list still refuses, as it should.

Spend and evidence

The Built slot spends turns, the judge spends them only when --judge names one, and no Builder session exists. All three slots are still resolved, so backends.json names them — catalogue resolution, not spend. Before running, check that no live fullrun spends the same credential; running both starves them.

Expect the claim to be refused on a narrowed battery (TRUTH_CHECK_NEVER_FIRED and similar): one task does not exercise every truth check, and that is correct. The result is the per-case verdict, never a claim, an improvement, a climb or a promotion. The controller owns domains/ and campaigns/; do not hand-edit what this writes.