Systematic Debugging

Core principle: Find root cause before attempting any fix. Symptom patches are failure.

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes. Apply this skill to ALL technical issues - especially "quick fixes" and emergencies.

---

The Feedback Loop Rule

Before reading code to build a theory, establish a tight command that goes red on the exact symptom and green when the bug is fixed. Tight = fast, deterministic, agent-runnable, specific.

When a clean repro is hard, spend disproportionate effort building the loop. Guessing without a red-capable loop is the failure mode this skill exists to prevent.

---

Phase 1: Root Cause Investigation

1. Read Error Messages Carefully

Read stack traces completely - line numbers, file paths, error codes. Don't skip warnings.

# View recent logs
shell_exec("tail -100 logs/app.log")

# Search error string in codebase
shell_exec("grep -rn 'ErrorString' src/")

Use file_read on relevant source files. To trace the error across the tree, use file_search (path and pattern, plus content to match inside files) or run grep through shell_exec.

2. Build a Tight Feedback Loop

Can you trigger the exact symptom with one command? Pick the lowest-cost loop type:

1. Failing test at the seam that reaches the bug (unit, integration, e2e)
2. HTTP/curl against a running dev server
3. CLI invocation with fixture input, diff stdout against expected
4. Headless browser asserting on DOM, console, or network
5. Replay captured trace - HAR, request payload, event log, webhook body
6. Throwaway harness booting the smallest useful slice of the system
7. Property/fuzz loop for intermittent wrong output over broad input space
8. git bisect run harness when the bug appeared between two known states
9. Differential loop - old vs new version, two configs, two providers, two datasets
10. Human-in-the-loop script (last resort) - structure the human steps, capture the result

Tighten the loop:

• Faster: cache setup, narrow scope, skip unrelated initialization
• Sharper: assert the exact symptom, not generic success
• More deterministic: pin time, seed randomness, isolate filesystem, freeze network

For non-deterministic bugs, raise reproduction rate before analyzing. Run 100×, parallelize, add stress, narrow timing windows. A 50% flake is debuggable; 1% usually is not.

shell_exec("pytest tests/test_module.py::test_name -v")
shell_exec("for i in $(seq 1 100); do pytest tests/test_flake.py::test_name -q || break; done")

3. Check Recent Changes

shell_exec("git log --oneline -10")
shell_exec("git diff")
shell_exec("git log -p --follow src/problematic_file.py | head -100")

4. Gather Evidence in Multi-Component Systems

For each component boundary (API → service → database, CI → build → deploy):

• Log what data enters and exits
• Verify environment/config propagation
• Check state at each layer

Run once to gather evidence → identify WHERE it breaks → investigate that component.

5. Trace Data Flow

Trace bad values upstream to their origin. Fix at the source, not at the symptom.

shell_exec("grep -rn 'function_name(' src/")
shell_exec("grep -rn 'variable_name\s*=' src/")

Phase 1 Checklist

• Error messages fully read and understood
• Tight loop command exists and is red on the exact symptom
• Loop is deterministic, or flaky bug has high enough reproduction rate
• Recent changes identified and reviewed
• Evidence gathered (logs, state, data flow)
• Root cause hypotheses can be stated and tested

STOP: Do not proceed to Phase 2 until you understand WHY it's happening.

---

Phase 2: Pattern Analysis

1. Minimize the Reproduction

Shrink the repro to the smallest scenario still going red. Remove inputs, callers, config, steps one at a time - re-running the loop after each cut. Done when removing anything more makes the loop go green.

2. Find Working Examples

shell_exec("grep -rn 'similar_pattern' src/")

3. Compare Against References

Read the reference implementation completely - every line. Understand the pattern fully before applying.

4. Identify Differences

List every difference between working and broken code, however small. Don't assume "that can't matter."

5. Understand Dependencies

What config, environment, or assumptions does this component require?

---

Phase 3: Hypothesis and Testing

1. Form Ranked Falsifiable Hypotheses

Generate 3–5 plausible hypotheses before testing any. Rank by likelihood and cheapness to falsify. State the prediction each makes: "If X is the cause, then observing/changing Y should produce Z." Discard hypotheses that don't make testable predictions.

Show the ranked list to the user if present - they may have domain knowledge that re-ranks it instantly.

2. Test Minimally

Test the top hypothesis with the smallest possible probe. Change one variable at a time. Never fix multiple things at once. Prefer debugger/REPL inspection - one breakpoint beats ten logs.

If adding temporary logs, tag every line with a unique prefix (e.g., [DEBUG-a4f2]) so cleanup is a single search.

3. Verify Before Continuing

• Loop goes green → Phase 4
• Loop still red → form a NEW hypothesis; do NOT layer more fixes on top
• Don't know → say so explicitly; use web_search/read_extract to research

---

Phase 4: Implementation

1. Write a Regression Test First

Write the simplest automated test that reproduces the bug and is currently red.

2. Implement a Single Fix

Address the root cause only. One change at a time. No "while I'm here" improvements. No bundled refactoring.

3. Verify the Fix

shell_exec("pytest tests/test_module.py::test_regression -v")
shell_exec("pytest tests/ -q")

4. Rule of Three

• Fix #1 failed → return to Phase 1 with new information
• Fix #2 failed → return to Phase 1, look harder
• Fix #3 failed → STOP. Question the architecture.

3+ failed fixes = architectural problem, not a bug:

• Each fix reveals new shared state/coupling elsewhere
• Fixes require massive refactoring
• Each fix creates symptoms in a different place

Discuss with the user before attempting another fix. The pattern itself may be wrong.

---

Red Flags - Stop and Return to Phase 1

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• "It's probably X, let me fix that" (without tracing data flow)
• Making multiple changes at once to narrow it down
• "I don't fully understand but this might work"
• Proposing solutions before reproducing the bug
• "One more fix attempt" when 2+ have already failed

---

Quick Reference

Phase	Key Activity	Done When
1. Root Cause	Read errors, build loop, check changes, gather evidence	Know WHY
2. Pattern	Minimize repro, find working examples, diff	Know WHAT differs
3. Hypothesis	Rank hypotheses, test one at a time	Confirmed cause
4. Implementation	Write regression test, fix root cause, verify	All tests pass