<!-- SYNCED COPY — do not edit here. Canonical: common-docs/skills/forcing-function-tests/SKILL.md This file is distributed to every consuming repo by common-docs/meta/scripts/sync_skills.py. Edit the canonical, run the sync, and commit each repo. Edits made here are overwritten and lost. -->

Forcing-Function Tests

A test's only job is to FAIL when the system is broken. The expected result must exist only at the end of a chain of real work — a forcing function — with no shortcut to it through the harness. A green reachable without the system doing its job is fiction: worse than no test, because it ships false confidence. Applies to pytest, jest, node --test, check:* scripts, and --self-test guards.

A real test around a big change — a playbook a fresh agent walks in the live product, written first and proven to fail on a planted break — is the safety-net skill; this one is the bar for the automated kind. Tests are guards, not verification. Green never flips anything to done. Done = someone who did not build it, on the live surface, with real data (campaign-pattern). A test keeps one named break caught forever. Fixture names look real, and test people come only from the persona factory (test-data-looks-real: aidream/aidream/testing/persona.py, matrx-frontend/scripts/lib/persona.mjs; never hand-make an account).

1. Before the body: name the break

Write the production change that makes this test fail — in the name or a one-line comment.

• Cannot name one → no test; find an observable outcome.
• Only an intentional decision fails it (a constant, message wording, private structure) → change detector. Test the behavior that depends on the decision: not MAX_RETRIES == 5, but "the 6th attempt never happens".
• Only a crash or a missing selector fails it → it guards nothing.

Method. (1) Name the SUT precisely — "the function foo", "the orchestrator", never "the feature". (2) List what it OWNS: ordering, routing, decisions, joins, persistence, transformations — never stubbed. (3) List external dependencies: network, registered functions, third-party SDKs, time, randomness — stubbed with known returns. (4) Design input and environment so the expected output is reachable only if every owned responsibility executes; any misstep (wrong order, dropped branch, swapped data, silent corruption) diverges observably. (5) Assert on that forcing output, never on an intermediate value the harness set up. (6) Gut check (§3).

2. The expected value comes from outside the code

A literal, a worked example, the spec, or a payload captured from the live system. Never:

• computed by the code under test or its helpers (expect(build(x)).toBe(build(x)), expected = items.reduce(...));
• the fixture or input echoed back (User(name="Alice").name == "Alice" proves assignment, not User; input url asserted on output url).

3. Gut check, then mutation pass

Gut check — RUN it, every test, before you call the test written. Replace the SUT with return expected and execute the suite. Still green → rewrite the test. Reasoning that it would fail is not the gut check, and a mutation pass does not stand in for it: the two are both required, and the pass most tests survive is this one. One forcing input never survives it — a constant returns that input's expected value. So when every case shares one expected value, a second input with a DIFFERENT expected value belongs in the SAME change (it.each / parametrize, a second captured page), never as a "before done" follow-up. (Observed 2026-09-10: 5 of 6 skill-guided reps shipped a single forcing input under "keep it small"; the two that named the second input deferred it, and the runner's executed return expected mutant passed their merged tests.) Mutation pass: for each realistic mutation, at least one test must go red — wrong constant or argument · wrong branch · missing side effect or state change · empty/default return · missing refusal for empty, zero, null, unauthorized, malformed · stages swapped or data between them corrupted · work silently skipped or dropped. A mutation nothing catches is an unguarded behavior: add the forcing input; do not bolt assertions onto a weak test. Shared checkout → every mutation runs through plant.py beside this skill, never a hand edit, one test file per command (usage in its docstring; proof: plant_selftest.sh). Exit 3 (restore failed) or 5 (mutation committed) → stop and revert it. Hand plants were left on disk and captured by peer commits, one of them an auth bypass (2026-09-10).

4. Doubles replace what the SUT CALLS, never what it IS

Ask of every double: does it replace a dependency, or the logic I am verifying? Only the first.

• Legitimate: third-party API, LLM, network, registered function, clock, randomness, filesystem, slow startup.
• Illegitimate: anything the SUT owns. If the stub's return IS the answer, you are testing the stub (mock.run.return_value = 42; assert mock.run([1,2,3]) == 42 — the pipeline did nothing).
• Before mocking, list the real method's side effects; keep the ones the test depends on real and mock the slow/external level beneath them.
• Never assert on the double's own presence (a *-mock test id, "the mock rendered").
• Call args, counts, and order ARE assertable when they are the contract (dedupe, exactly-once send, not-reinvoked-on-resume). Make the double specific enough that the wrong branch cannot satisfy it — one fixture per branch (success, error, malformed).
• Mock setup outgrowing the test logic → use real components (real Scheduler + MemoryCheckpointer; real store + real reducers with only the API module mocked).

4a. Proving RED never weakens the shared working tree (added 2026-09-12, Data Doctrine chair)

A RED proof that edits the real guard on disk is one interruption away from shipping the weakening: on a shared checkout a peer sweeper commits whatever is on disk the moment a lane dies (it happened — a frame guard reached main deleted; DD-158). So: prove RED by injecting the weakening in memory (patch/mock inside the test), in a scratch copy, or in a git worktree you delete after. If the real file must be touched, weaken and restore in the SAME tool call, then verify the restore.

5. Fixtures: captured, complete, typed

The done law bans manufactured data as PROOF. Controlled inputs inside a guard are fine — shaped from reality:

• Captured, not invented. Record a live payload or a sanitized real row (RecordReplay mode="record") and commit it beside the test.
• Complete. Every field the real thing carries, not just the ones this test reads. A partial double passes while integration breaks.
• Typed against generated contracts. TS fixtures satisfies the generated DB/API type — never as / as unknown as a partial object (matrx-frontend type-safety skill). Python fixtures go through the real Pydantic model.
• Environment conspiracy (nondeterministic systems — agents, schedulers, solvers): choose stubbed dependency values whose composition leaves exactly one correct output. The SUT must consume, combine, and route every one of them to land it; no partial-credit path exists. This tests that the orchestration fed the nondeterministic component the right inputs and used its output correctly, without testing that component itself.

5a. Every value comes from a REAL USE CASE (the owner's law, 2026-09-21)

Before entering any test data, name the real use case in one sentence: who the user is, what business they run, what job this exact feature does for them. A recycling company's pickup schedule. A dental practice's new-patient intake. A property manager's maintenance requests. Then make the values true to that business — field names a practitioner would use, statuses that follow a real lifecycle, money that adds up, dates that make sense together, and relationships true to the domain (an invoice belongs to a job that belongs to a client).

Banned outright: Acme, foo/bar/baz, lorem ipsum, test table, ZZZ …, Job 001, sequential Item 1…N, .example mailboxes, and one stand-in person repeated everywhere. A screenshot of "Job 001 / Ada / 42" tells the owner nothing; a picture of a recycling route with today's pickups tells him whether the product works. Placeholder data does not fail a test — it destroys the only way anybody can tell whether the thing is finished.

Never real personal data. Names, emails, phones and addresses are synthesized to look real and belong to nobody; no row is copied from a real customer, a real inbox, or the production tables. Realistic is the standard; real people are forbidden. Test identities stay admin@admin.com and test@test.com.

Name everything after the use case. The organization, the tables, the form, the pipeline, the document, the notification — "the Harbor Dental new-patient intake", never "ZZZ FORMS-2 safe to delete". Cleanup finds a test organization by a tag in its settings, never by a junk name.

Do not invent your own. The datasets live once, in @ai-matrx/records/use-cases (aidream/apps/shared/records/src/use-cases/, with a README on adding one). Draw from it; add a use case there when none fits, and the library's validator will refuse a dataset whose field types the store cannot resolve, whose relations point at rows that do not exist, or that carries junk.

Enforced: aidream/scripts/check_no_placeholder_data.mjs and matrx-frontend's pnpm check:no-placeholder-data, both reading their banned patterns from that one library so they cannot disagree. They scan the data that reaches a SCREEN — demo harnesses, seeds, screenshot scripts and recorded fixtures — not every unit test's opaque handles: user-1 in a cache assertion is a handle, and example.com in a URL-parser test is the reserved domain doing its job. A line genuinely needing a banned word carries matrx-real-data:allow <reason>, and the reason is required.

6. Bug fixes and guards: red first, for the right reason

• Fixing a defect: write the test or guard FIRST and watch it fail on the unfixed code. It must sit at a seam that reproduces the real bug pattern (the real caller chain, the real boundary); a seam too shallow to exhibit the bug is false confidence — record the missing seam instead (diagnose §5).
• The red must be an assertion failure that names the defect. An import error, typo, missing fixture, or timeout is not a red — fix it and re-run until it fails correctly.
• Fix, watch green. For guards, commit the sabotage as --self-test: plant the violation → exit non-zero with file:line and the fix → restore → clean (strictness-law §7–8).
• A test green on its first-ever run has proven nothing yet — sabotage it once.
• A multi-rule guard must prove each rule SEPARATELY, and the sabotage is per-rule. Break one rule at a time and watch the self-test go red for that rule. Whole-suite red proves nothing about which rule earned it.
• A fixture caught by a rule OTHER than the one it names is a fake proof. Observed 2026-09-11: a byte-size guard grew four rules; the fixture for the fourth put the unit label three lines from the arithmetic, inside an older rule's six-line window, so the older rule matched it and the new rule was never exercised. Deleting the new rule entirely left the self-test PASSING. The rule was real; only its proof was fake. Write each fixture so it can be caught by nothing but its own rule — then delete that rule and confirm the self-test goes red.
• New features: test-first order is not mandated; the §3 mutation pass is the proof. Never delete working code to replay a red-green ceremony.

7. Source-reading tests: census, never change detector

Reading source is legitimate when it derives a CLASS from the code and fails on any new member (every field the client sends is writable; every postgres_changes table is published). Asserting that one specific line or symbol exists — or stays removed — is a change detector: replace it with behavior or delete it. Census tests carry a self-test.

8. Where tests go

• Your code's boundary contract — the query emitted, payload produced, row persisted, event emitted — not framework mechanics (Pydantic raising, a router calling a handler, React rendering a prop).
• Persistence: assert the LIVE row or DB machinery, not the repo's belief (verify-live-state §1).
• Vertical slices: one behavior → its test → its code → next. Never a batch of tests for imagined behavior up front.
• Test-only helpers (reset, destroy, inject) live in test utilities, never on production modules.
• One behavior per test — a name that needs "and" gets split. The name states the forcing function: test_<must_be_true>_when_<condition> / it("refuses X when Y"); never test_it_works / test_happy_path.

Reject on sight — the four failure modes and their cousins

• Tautology: asserting what the fixture set up; assert True, is not None, toBeDefined(), toBeTruthy() as the primary behavior assertion
• Over-fitted input: one cherry-picked input a hardcoded return satisfies (apply_discount(100, "SAVE10") == 90 passes against return 90) — use several inputs, parametrize, or property-based tests
• Mock owns the logic: mocking the SUT or a layer inside it (§4)
• Coincidental ordering: dependence on dict/set iteration, thread scheduling, harness step order, or test order the SUT does not guarantee; shared mutable state across tests
• Waiting with sleep/fixed delays for async work — wait on the actual condition with a timeout whose error names what never happened; a fixed delay is legitimate only when the behavior under test IS timing (debounce, tick interval), with a comment saying why
• Broad except / catch asserting "didn't crash"
• Snapshots nobody hand-checked
• Real LLM / paid API calls in CI (use committed recordings)
• skip, xfail, .only, it.todo left on a behavior-critical test — an escape hatch
• "Tests are green" offered as done

Rationalizations

Excuse (verbatim, observed 2026-09-10)	Reality
"A second saved page, via parametrize, would stop a hard-coded answer from passing" — listed as a follow-up	A known hole deferred is a shipped hole; a constant passes the merged test today.
"The test still checks two things in one, because the brief said not to restructure it"	"Keep it small" never waives §3 or one behavior per test. A split is one more it.
"…which works because the node imports scrape at call time"	That is the SUT's own pipeline. Stub the fetch beneath it (§4).
"My lines add nothing — the existing test already catches this bug"	Run return expected against it first: that test used the SUT's own helper as its oracle and passed a hard-coded mutant.

Red flags — in the brief or in your own reasoning

• "Keep it small — just add an assertion to the existing test."
• "A second input can come later."
• "The existing test already guards this" — said before a return expected run.
• "The stub works because it is looked up at call time."
• CI is red on something else / the release train leaves in 20 minutes.

Done criteria — every one true, or the test is rewritten, not patched

1. It fails when the execution path is wrong.
2. It fails when a piece of work is silently skipped or dropped.
3. It fails when data between stages is swapped or corrupted.
4. It fails against a return expected replacement of the SUT — executed, not reasoned.
5. Its expected output is a forcing function on everything the SUT owns, from a source independent of the SUT.
6. It sits at a seam that reproduces the real bug pattern.

Review sequence (anyone's test, including yours)

1. What exactly is the SUT? 2. What does it own — is any of it stubbed? 3. Name the one-line bug this catches. 4. Does return expected pass it — run, not reasoned? 5. Is the expected output forced by correct execution and independent of the SUT, or arranged by the fixture? 6. Are fixtures captured, complete, and typed? Any unanswerable → not mergeable.

Stack notes

• Python: pytest, not unittest; fixtures set up, never assert; @pytest.mark.parametrize one forcing function over many inputs; pytest.raises(..., match=...) asserting type AND the message/attribute proving the right path fired; mark @pytest.mark.integration (and e2e) so they can be selected or excluded. Workflow actions → aidream matrx-action-testing.
• TypeScript: jest (pnpm test <path>), node --test for scripts. RTL queries by role/text, never by mock test id. Redux: real store + reducers.
• Guards: exit code is the verdict; unreachable data = UNMEASURED = fail (strictness-law §7).