Mutation cleanup

Surviving mutants are the engine's untested behaviour, listed. This skill is the method for working them down, so that every round is worked the same way and the weekly count means the same thing from week to week.

What a surviving mutant is

cargo-mutants applies one small change to the engine at a time (flip a comparison, delete a match arm, return Default::default() from a function) and runs the test suite. A mutant that survives is a change no test noticed. That is either a gap in the tests or code nothing depends on.

It is not a bug report. The mutated code is not in the repository, and the count on its own says nothing about correctness. It says how much of the engine's behaviour the tests actually pin down.

Two workflows produce them:

• .github/workflows/mutation-testing.yml runs the full set every Monday and opens an issue labelled mbot:survivors with the counts and a report.
• .github/workflows/mutation-diff.yml runs on every pull request that touches packages/engine/**, over the changed lines only. That gate is why new gaps should not reach this backlog.

Step 1: Get the list

From the weekly issue, take the run id and download the report:

gh run download <RUN_ID> -n mutation-report

missed.txt has one line per surviving mutant: file, line, column, and the change that was applied. caught.txt, timeout.txt and unviable.txt hold the rest. README.md in the same artifact has the counts and the per-file distribution.

Working from a failing pull-request gate instead? The gate splits its mutants over shards, and each shard leaves its files in its own artifact, mutation-report-diff-shard-N. Only a red shard has something to fix; the list is short by construction.

Step 2: Work one file at a time

Not one line at a time. Mutants cluster in a handful of files, and the context built reading a file pays off across all of its mutants. Take the files in descending order of surviving count; the distribution is in the report.

Step 3: Classify before touching anything

Every mutant is one of three things. Decide which before writing code.

A real gap in the tests

Write a test that fails on the mutated code and passes on the real code. Verify it in that order: apply the mutation by hand, watch the test go red, revert the mutation, watch it go green. A test you did not see fail proves nothing.

Put the test where the behaviour lives. Unit tests for a single function, a BDD scenario when the gap is about how a law evaluates end to end. A mutant in operations.rs that changes a comparison usually wants a unit test; one in service.rs that changes resolution order usually wants a scenario.

Before writing a new test, check whether one already claims to cover this. Often a mutant survives next to a test that reads as though it pins the behaviour. Then the gap is that test, and the fix is to make its fixture construct the premise its own doc-comment describes. A survey of this repository found fourteen such tests: a longest-prefix test with no shorter competing article, a retain test whose only candidate was filtered out upstream, an assertion about a variable nothing references. Each one was a mutation survivor, and each read as coverage.

Rewriting the existing test beats adding a second one beside it. A second test leaves the first as a decoy for the next reader.

A mutant that says nothing

Debug helpers, Display implementations, bin/ targets such as validate_annotations.rs, WASM binding shims. Pinning these down with assertions adds maintenance and no safety.

Record the decision so it stays recorded:

#[mutants::skip]  // Debug-only; asserting on the format pins nothing that matters.
fn render_internal(&self) -> String { … }

or, for a whole area, an exclude_re entry in packages/.cargo/mutants.toml with a comment saying why. This is a valid outcome, not a way out. The count drops because the question was answered, and the next reader inherits the answer instead of the question.

Dead code

Delete it. Do not write a test around code that nothing calls. Check first that it is genuinely unreachable and not merely unused by tests.

Limits

The engine decides legal outcomes: benefit entitlements, allowances, tax calculations. These limits follow from that.

• Never write an assertion that pins an arbitrary implementation detail just to kill a mutant. A test describes behaviour that is wanted. If you cannot say which behaviour a test protects, it is the wrong test.
• If killing a mutant means changing behaviour rather than coverage, stop and raise it. A surviving mutant can be the first sign that the code is wrong, and that is a decision for a human, not a cleanup.
• Never weaken or delete an existing test to make something pass.
• REVIEW.md in the repository root holds the project's review dimensions and applies here too.

Step 4: Verify

just mutants-diff        # mutates only your own changed lines, minutes not hours
just mutants             # the full set, over five hours; CI shards it instead

just mutants-diff must be clean before committing. It runs the same check as the pull-request gate, so a clean result here means a green gate there.

Exit code 2 means a mutant survived in your own changed lines, which is the gate doing its job. Exit code 3 means a test ran out of time under mutation.

Commit per file, in small commits, conventional-commit subjects in Dutch.

What the numbers do next

The weekly run compares against the previous week and reports what is new, what was solved, and what moved between caught and timeout. As long as more than 25 mutants survive, the timeout multiplier stays fixed so the weeks stay comparable. Below that it starts rotating over 3, 2 and 5, which exposes mutants that only die because a test ran out of time rather than because an assertion caught them. Those are the next layer of the same question, and the same three classifications apply to them.