The learning loop

Overview

Everything this team has already done is data. Every execution, failed run, escaped bug and retro is a recorded case that can be replayed against the next version of a workflow. The owner put it this way on 2026-10-06: "everything that we've done is basically data, and we can use those as evals to improve", and "we need the repeatable workflows to rely on, and then use that to compound."

A lesson compounds only if it ends up somewhere the next run reads or a gate enforces: an eval case, a test, a fitness function, a prompt line, a skill. If it ends up anywhere else, someone learns it again later at full price. The 2026-10-06 retro is the cautionary example. The same lesson (a test double does not behave like its backend) cost four separate PRs before anyone wrote it down.

The loop has six steps. Each one produces a specific artifact in this repo, and a step that produced nothing did not happen:

Step	Produces
1. Capture	a paper cut or failure record with an execution id and a commit SHA
2. Classify	one category per failure, counted in the retro Scorecard
3. Convert	an eval case for an escaped bug, or an issue with a test or fitness function for a recurring class
4. Change	one variable changed, as a new workflow id or one PR
5. Measure	eval results plus Scorecard numbers, before the change is adopted
6. Retro	docs/retrospectives/YYYY-MM-DD-<slug>.md with a Scorecard comparable to the last one

workflow-experimentation covers how to run and judge one change (step 4). This skill covers where the change comes from and where its result goes.

Outcomes we are looking for

Outcome 1: every lesson can be traced to a record

A lesson starts as a stored fact, never as something remembered.

• Signal: every paper cut, failure count and escaped bug in a retro cites an execution id, a commit SHA, a PR or an issue.
• Signal: a retro's timeline comes from GitHub, deployment or command output. The 2026-10-04 retro had to correct orchestrator log entries that were stamped by hand and ran up to 40 minutes ahead.

Outcome 2: a lesson ends up where the next run will meet it

• Signal: every retro's "What we changed" lists a PR, test, fitness function, eval case or prompt or skill edit, and none of its items say only "be more careful".
• Signal: a failure class that recurs gets an issue with a proposed test or fitness function, and that test eventually lands.

Outcome 3: escaped bugs become a regression suite for the verifiers

A bug that verification certified and someone found later is the most valuable data this team has. It is a known defect at a known commit, so it can be used to measure any future verifier.

• Signal: each escaped bug becomes an eval case (pre-fix commit + expected finding) within the retro that records it.
• Signal: a change to a verify prompt, verifier model or gate is not adopted until it has been run against the eval set.

Outcome 4: improvement is measured, not felt

• Signal: consecutive retros share Scorecard rows, so the trend can be read off two tables.
• Signal: a claimed improvement cites the window and the sample size behind it.

The loop

1. Capture

Record the event at the time it happens, with its identifiers:

• execution id (syn execution show <id>), and the phase that failed;
• commit SHA the run checked out, read from that execution's own record: its git_checkout event, checked_out_commits on WorkspaceProvisionedForPhase, or the SHA its verify artifact names. A PR's current head is not this: a fix round moves the head after verification;
• what was observed, as pasted output (log line, exit code, error text), not paraphrase;
• what was expected.

Use the stored record. syn execution show, the event stream, gh pr view and git log are evidence; your memory of the run is not. Recording later is fine, but the identifiers have to be looked up, not recalled.

Where it goes: the PR, an issue, or the retro's working notes, wherever the next reader will see it. A paper cut that only exists in a scratchpad gets lost when the session ends.

2. Classify

Every failure gets exactly one primary category. The retro scorecards so far use finer, ad hoc rows ("Codex capacity / quota", "phase deadline (exit 124)", "OOM at the workspace cap", "orphaned by an API restart"). This five-way split is the summary those rows fit into, and each one sends the fix somewhere different:

Category	Means	Seen in the retros as	Fix goes to
Platform gap	the platform could not do what a reasonable task needed	phase deadline, OOM at the cap, orphaned by restart, compose drift	a platform issue, with the image or deployment it was measured on
Vendor outage	a provider was down, at capacity or out of quota	Codex capacity (29) and quota (3) on 2026-10-06	retry/fallback policy, never a workflow rewrite; excluded from the run failure rate
Agent error	the agent had what it needed and got it wrong	a defect it wrote or certified	an eval case (step 3), then a prompt, model or gate change
Orchestrator error	whoever dispatched or merged got it wrong	the ~3 h idle on a blocking question; batch approval of 6 PRs	orchestrator practice, written into a skill or runbook
Spec error	the task or brief was wrong	a false premise an agent correctly refused	the brief or the workflow's premise phase

An agent that refuses a false premise or stops at a platform gap has not made an agent error. The 2026-10-06 retro counts 20 such refusals as a good outcome, and two of them turned into needed fixes (#1650/#1651, #990/#1652). Score the refusal as correct, and classify whatever caused it.

3. Convert

An escaped bug becomes an eval case. An escaped bug is one that a verification phase certified as clean, and then a later reviewer, a second verifier or production found. Each case records:

• the PR, and the head commit before the fix (the commit verification certified, or the fix commit's parent on the PR branch);
• the expected finding: the defect in one sentence, specific enough that a verdict either names it or does not;
• the pass condition: re-verifying that commit ends in a blocked verdict that names the defect. Being blocked for some other reason does not count as a pass.

Run it with sdlc-reverify-pr-v1, pinned to that commit through an eval:

syn eval create --name "escaped: #1652" --goal "verify blocks: binary reads 404" \
  --repo syntropic137/syntropic137@<pre-fix-sha>
syn workflow run sdlc-reverify-pr-v1 -t "#1652" --eval <eval-id>
syn eval runs <eval-id>

The baseline pins the SHA, and every run of the case shows up under syn eval runs.

This does not work yet. Issue #1658 tracks it. The prepare phase refuses a merged PR, checks out the PR's live head instead of the pinned SHA, and merges origin/main, which for an escaped bug already contains the fix. Until #1658 lands, a seed case run as written passes for the wrong reason. Record the cases now so the set exists when the mode does.

Seed cases, from the 2026-10-06 retro:

Case	Commit to verify	Expected finding
#1574 (fixed by #1641)	aab6e95b, #1574's head as merged (merge df7a99db1)	ExecutionRequest keyed by the execution's own id shares the execution's stream in a store keyed by aggregate id alone, so every direct start is dropped as a duplicate
#1649	7047b1c3, the parent of fix bd302d76	an execution id is accepted as an eval id; archive writes into the execution's stream
#1652	b2f680f0, the parent of fix 721a8014c	binary artifact reads use the artifact id, but MinIO keys by workflow, execution and artifact, so every binary read returns 404
#1654	123b2520, the parent of fix 4dae285b2	codex exec reports usage once, after the run, so the per-phase cost limit can never stop a Codex phase

The three parents are merges of origin/main into the PR branch, which is the head verification was handed. All four passed their own verification against a test double or fixture that did not behave like the real backend or CLI. The eval asks whether a verifier will catch them now.

A recurring failure class becomes an issue with a test or fitness function. "Recurring" means a second occurrence, or a first one that obviously belongs to a class. The issue proposes the check that would have failed:

• a static property across the whole codebase belongs in ci/fitness/, ratcheted;
• a behaviour of one code path is a test beside the code (see AGENTS.md, "Why the typing ratchets are fitness functions").

The 2026-10-06 class (an in-memory double that does not behave like its real backend) became contract tests run against both the fake and the real backend (draft #1655) and ESP #344. Writing "be careful with doubles" in a retro would not have counted.

4. Change

Change one thing at a time (a prompt, a model, a gate) and follow workflow-experimentation: a new workflow id for the treatment, the control left untouched, outcomes registered in advance. A change that touches two variables cannot be credited to either.

5. Measure

Before adopting the change:

1. Run the eval set against it. A verifier change that misses a case the old verifier caught is a regression, whatever else it gains.
2. Record the Scorecard numbers for the window it ran in:

Measure	Definition	Why this one
Escaped defects per merged PR	defects found after a PR was certified, divided by PRs merged	the quality bar; rework dominates every other cost
Cost per merged PR	sum of execution cost in the window, divided by PRs merged	spend that shipped nothing is included, on purpose
Run failure rate, excluding vendor outages	failed executions minus vendor-outage failures, over executions started minus vendor-outage failures	otherwise a Codex quota day looks like a bad workflow
Owner minutes per PR	owner time spent reviewing, approving and unblocking, divided by PRs merged	the scarce resource the loop exists to save

These four have not all been recorded yet. The 2026-10-06 Scorecard has spend, failures by cause and PRs merged, but no escaped-defect count and no owner minutes. Record whatever you can measure, and write "not recorded" for the rest rather than leaving a row out. That is how the 2026-10-06 retro handled the 10-04 retro's missing spend.

Quality comes first. A cost or latency gain counts only if escaped defects per merged PR held flat (workflow-experimentation, Outcome 3).

6. Retro

Follow the retrospectives README template (What happened, Timeline, Root cause, What we changed, Open follow-ups), and add:

• a Scorecard with a stated window and a column for the previous retro, as the 2026-10-06 retro does against 2026-10-04. Reuse that retro's rows and add new rows; do not rename them, or the comparison is lost;
• the eval cases added in this window, by PR and commit;
• an index row in the README.

The README's rule applies here too: a retro has to link a concrete change. If nothing changed, the lesson goes under "Open follow-ups" in an existing retro.

Principles

1. Judge a past run by its stored record, not by current code. The question is what that run saw: its checked-out commit, its pinned phases, its artifacts. Today's code is a different system. Pinned launch records checked_out_commits on WorkspaceProvisionedForPhase (#1615) for this reason.

2. "Verified" means the gates ran. A verdict counts only if the verifier ran the tests and gates and pasted their output. A read-only review that ran no tests is a review. Call it that, and do not count it toward the eval or the Scorecard as verification.

3. A green double proves the double. A behaviour certified only against an in-memory adapter or a fixture is not certified. A double needs a contract test that runs the same assertions against the real backend.

4. A lesson has to land somewhere enforced. Choose a test, a fitness function, an eval case, a gate, a prompt line or a skill, in roughly that order, because the earlier ones fail a build and the later ones only ask.

5. Escaped bugs are worth the most. Each one is a labelled defect at a pinned commit, and you cannot buy those. Convert every one.

6. Count vendor outages, then leave them out. Record them in the Scorecard so the cost is visible, and exclude them from the failure rate so they do not hide what a workflow change did.

7. Keep Scorecards comparable. A new retro with different rows starts a new series. Add rows, keep the old ones, and write "not recorded" when a number was not measured.

Anti-patterns

All of these have been observed here.

• A lesson written only in a scratchpad. The 2026-10-04 orchestrator kept 58 paper cuts in a papercuts.md outside the repo. The ones that later became issues got fixed. The rest are only as durable as that file.
• "Verified" meaning a read-only review that ran no tests. The 2026-10-06 batch approval ("assume the other 6 are good, assuming they've had the review passes") trusted verify phases that could not reach a real store. One of those PRs broke every start for about 3.5 hours.
• A test double certifying behaviour the real backend lacks. Four cases on 2026-10-06: #1574 (stream keying), #1649 (same), #1652 (artifact keying), #1654 (Codex usage timing). All four were green on their own verification.
• Judging a past run by current code instead of its stored record. A run is declared wrong, or right, because of what the code does now. The run's commit and pinned phases are the evidence, and an execution does not store its prompt (#1030), so write the prompt down when you dispatch.
• A retro with no linked change. A list of observations nothing acts on. The README forbids it, and the Outcome 2 signal catches it.
• A failure count with no cause split. "66 failed" says nothing until the 32 vendor outages are separated from the rest.

Recommended tools by outcome (as of 2026-10-06)

Outcome: every lesson can be traced to a record

• syn execution show <id> and syn execution list. These give the stored phase state, cost and failure for a run.
• The execution's stored checkout, not the PR's current head. The commit a run certified is the one in its own record: the git_checkout event in its stream, checked_out_commits on WorkspaceProvisionedForPhase (#1615), or the exact SHA in its verify artifact. gh pr view <n> --json headRefOid gives the head as it is now, which a later fix round moves, so use it only when you are verifying the current head. mergeCommit is the merge, not anything a verifier saw.
• git log --grep finds the fix commit for an escaped bug, so its parent can be the eval's pinned commit. PR numbers and issue numbers differ (#1649 fixed issue #967), so look both up.
• The execution's event stream and phase artifacts. These show what the agent did and reported, as opposed to the run's status.

Outcome: a lesson ends up where the next run meets it

• ci/fitness/ with fitness-exceptions.toml for whole-codebase properties, ratcheted.
• A test beside the code for one path's behaviour, driven against the real backend where a double was the problem.
• This skill, dogfooding and workflow-experimentation. Edit their anti-pattern lists when a lesson is about how we work.

Outcome: escaped bugs become a regression suite

• syn eval create, syn workflow run --eval <id>, syn eval runs <id>. Pinned launch, the eval aggregate, and runs as a filtered execution list (evals plan, steps 4 to 6).
• sdlc-reverify-pr-v1 as the verifier under test, once #1658 adds a mode that verifies the pinned commit as found.

Outcome: improvement is measured

• The previous retro's Scorecard, copied as the starting table.
• workflow-experimentation's judging order before reading any cost number.

References

• workflow-experimentation - running and judging the change this loop proposes.
• dogfooding - where work runs, and how delivery is confirmed.
• docs/retrospectives/README.md - the retro template and index.
• 2026-10-04 retro
  • the first Scorecard.
• 2026-10-06 retro
  • the comparable Scorecard and the seed eval cases.
• Evals plan - #967.
• #1658 - re-verification cannot yet replay a case at a pinned pre-fix commit.
• #534 - insight loop: analysing stored execution data automatically.
• #1655, ESP #344 - contract tests for in-memory adapters.

Continual improvement

This skill is maintained in the syntropic137 repository: https://github.com/syntropic137/syntropic137/blob/main/.claude/skills/learning-loop/SKILL.md

The seed-case table and the #1658 caveat are the parts that will go stale. When an escaped bug is converted, add its row. When #1658 lands, replace the caveat with the actual invocation and the first run's result. When a Scorecard measure is first recorded, remove it from the "not recorded" note. Apply the loop to this skill too: if a lesson had to be re-learned because the skill missed it, add it to Anti-patterns with the execution id that found it.