Post-deploy verification

A deploy that went green tells you the process started. It says nothing about whether the change it shipped actually works, or whether it broke something else that happened to keep running. Verification is what closes that gap, and it happens every time — not only when something looks wrong.

1. Procedure — exact calls

1. get_release({ task_id }) → read deployed_at, component, checks (health samples, smoke results, new_errors, notes, early_stop), profile.auto_rollback, and tasks[] — every task this release covers, not only the one you were woken on.
2. get_environment only if you need to confirm nothing is bound (checks.notes already says so in most cases).
3. list_runtime_errors({ component, since: "<window that starts well before deployed_at>" }) — see "Reading error groups" below for the window and how to judge new.
4. query_runtime_logs({ component, since: deployed_at, min_severity: "error" }); re-query with text: set to a route or module the release changed when the error list alone is not enough.
5. run_smoke_checks({ task_id }) when the stored evidence is older than ~30 minutes, or when early_stop itself came from a smoke failure and you need a fresh read before deciding.
6. One read-only acceptance check per observable criterion of every task in release.tasks (list_acceptance_criteria per task) — a release with several tasks is verified in full, not just the card that woke you.

2. Reading error groups

list_runtime_errors' new: true is accurate on a provider with native grouping (gcloud). On a provider without one (Vercel, AWS — fallback fingerprint grouping), it only means "this group's first occurrence fell in the back half of whatever window you queried" — querying since: deployed_at puts a regression that starts right after the deploy in the window's front half, so it can read new: false and never show up as a concern.

• Judge a group by its own first_seen against release.deployed_at, never by new alone.
• Read with a window that starts well before deployed_at — roughly twice the time elapsed since the deploy, or at least the soak period — so an occurrence from before the deploy is visible and can be told apart from one that started with it.
• Before rolling back on one low-count group, re-read with since: "1d". If the same message recurs before deployed_at, it is chronic, not a regression. Once you have confirmed a group is chronic and unrelated to any release, save_memory (project scope) a one-line note of its fingerprint message ("redis timeout in apps/api recurs hourly; not a release signal") so a future run of this skill recognizes it without a full 1-day re-read.
• checks.new_errors (the sweeper's own field) can be empty on Vercel or AWS even when errors genuinely began with the deploy — your own read in this run is the check, not a formality that confirms what the sweeper already decided.

3. Verdict table

Evidence maps to one of four outcomes — finish, re-check once, roll back, or hand to a human — the same shape as Argo Rollouts' Successful/Failed/Inconclusive and Flagger's "N failed checks before rollback": one blip is not a trend, and an unreadable signal is not a pass.

Evidence	Verdict
early_stop: smoke check failed	Re-run run_smoke_checks once. Passes, and no error group has first_seen >= deployed_at → finish, naming the transient failure and the re-check. Fails again → rollback_release (verify_failed).
Health check failed twice	fetch_url the checks.health_url once (GET). Still failing → roll back. Recovered → look for a restart/crash line in the logs since deployed_at; clean → finish, noting the blip.
An error group with first_seen >= deployed_at whose message, path or stack touches a file or route this release's tasks changed (get_task_pull_request({ task_id, include_diff: false }) → files)	Roll back (verify_failed).
Same, but unrelated and it also recurs before the deploy in a 1d read	Finish, naming each group and why it is not a regression.
Only structural gaps in notes (no bound environment, no health URL, no base URL)	Finish, stating the gap verbatim — it is a different kind of verification, not a failed one.
A runtime tool itself errored	Retry once. Still failing, and health and smoke are both green → finish, saying the logs were unreadable this run.
Nothing at all is readable for an on_merge/dispatch service (no health, no smoke, no logs, and the retry above also failed)	One comment: "verdict needs a human: no evidence was readable (<what you tried>)" — and stop. A human finishes or rolls back from the Deploy tab. This is the one case where "finish with the gap stated" is not the default: with nothing readable there is no evidence to finish on.

Recording a missing deploy target

A notes gap like "no health URL" or "no base URL" does not always mean nobody can fix it — if the deploy's own output (get_deploy_logs, a local_run.tail, or a workflow log) names the real URL the service answers on, update_deploy_target({ env, base_url / health_url / logs_url }) records it so the next soak and verdict actually have something to read, instead of hitting the same gap every release. Still finish or roll back on the evidence you have in THIS run — recording the target fixes the next one, it does not retroactively create evidence for this one.

4. Feature-flagged changes

If a task's after_deploy (on release.tasks[]) enables a flag or config, its new behaviour is not reachable yet — that is expected, not a failure. Check that the old path still works, and say "criterion X is behind flag Y, enabled after release by a human" in your note. A flag flip is a manual step for a human, never something you do yourself.

5. Notes that count as evidence

✅ "query_runtime_logs (apps/api, since 2026-10-01T10:02Z, ≥error): 0 entries; list_runtime_errors 40m: 2 groups, both first_seen before deploy (chronic redis timeout, 1d read shows it hourly); smoke 3/3 (re-run 10:41); T-12 GET /api/orders?status=paid → 200 with paid_at field."

❌ "Looks fine, logs clean."

6. Rationalizations (adapted from obra/superpowers)

• "Deploy went green" → that is the process starting, not the behaviour it shipped.
• "Soak was green" → read it yourself, in THIS run — the sweeper's own fields are a starting point, not a substitute.
• "new_errors is empty" → on Vercel or AWS that can miss an error that genuinely started with the deploy; read first_seen yourself.
• "The error is probably old" → show the first_seen, don't assert it.
• "One failed probe" → re-check once, then decide — never roll back, and never finish, on a single sample.

Common Mistakes

• Omitting component in a monorepo — silently reads the root component's logs, not the release's own.
• Reading errors with the default 24h window instead of a window anchored before deployed_at.
• Judging novelty by new instead of first_seen against deployed_at.
• Verifying only the task you were woken on when the release carries several.
• Writing to production (seeding, creating records) to check a criterion.
• "Should", "probably", "seems" anywhere in a finish/rollback note.

Batch releases with no runtime environment

Most desktop and mobile components have no bound runtime environment at all — there is no host to sample health from, no log stream to query, no error groups to list. That is not an incomplete verification, it is a different one: see batch-artifact-verification for what to check instead (the published artifact, store build, or local run), and say explicitly in the finish_release/rollback_release note that no runtime environment is bound — the same rule as any other gap: never let silence read as a pass.