Debugging Mastery

Source: "Debugging" (David Agans) + Google/DeepMind/ByteDance production debugging experience + years of midnight production incidents Core Philosophy: "Debugging is the art of systematically testing hypotheses until the root cause is found. It is NOT randomly changing things hoping."

The Nine Indispensable Rules

From David Agans' "Debugging" — the debugging bible:

  1. Understand the system
  2. Make it fail
  3. Quit thinking and look
  4. Divide and conquer
  5. Change one thing at a time
  6. Keep an audit trail
  7. Check the plug
  8. Get a fresh view
  9. If you didn't fix it, it ain't fixed

---

1. The Debugging Workflow

1.1 Systematic RCA Flow

  ┌──────────────────────────────────────────────────────────────┐
  │  Step 1: REPRODUCE                                            │
  │     Can you make it happen?                                   │
  │     If not: this is the first problem to solve                │
  │                                                                │
  │  Step 2: ISOLATE                                               │
  │     Binary search through the system to find the component    │
  │     Remove variables: isolate the MINIMAL reproducing case    │
  │                                                                │
  │  Step 3: MEASURE                                               │
  │     Log everything. Add MORE logging if needed.               │
  │     "The resolution of your debugging is limited by the       │
  │      resolution of your instrumentation"                      │
  │                                                                │
  │  Step 4: HYPOTHESIZE                                           │
  │     Form a specific, testable hypothesis                      │
  │     Bad: "Maybe there's a memory issue"                       │
  │     Good: "The cache eviction runs on the wrong goroutine"    │
  │                                                                │
  │  Step 5: TEST THE HYPOTHESIS                                   │
  │     If hypothesis is wrong → return to Step 3                  │
  │     If hypothesis is right → FIX it                           │
  │                                                                │
  │  Step 6: VERIFY THE FIX                                        │
  │     Run the reproduction case again (it should pass)          │
  │     Run the full test suite                                    │
  │     Add the reproduction case as a REGRESSION TEST             │
  └──────────────────────────────────────────────────────────────┘

1.2 The Binary Search Debugging Technique

  Step through the system components:

  ┌──────────────────────────────────────────────────────────────┐
  │  Input → [A] → [B] → [C] → [D] → [E] → Output              │
  │                                                               │
  │  Check output at C:                                           │
  │  ✅ C passes → bug is in D or E                              │
  │  ❌ C fails  → bug is in A, B, or C                         │
  │                                                               │
  │  Repeat: check at B (or D, depending on result)              │
  │  Continue until you've isolated the single component         │
  │                                                               │
  │  This is O(log n) — exponentially faster than linear scan    │
  └──────────────────────────────────────────────────────────────┘

---

2. Log-Based Debugging

2.1 Structured Logging for Debuggability

// ❌ Bad: unstructured, no context
console.log('Order processed');
console.log(`Error: ${err}`);

// ✅ Good: structured, searchable, contextual
logger.info('order.processed', {
    orderId: order.id,
    userId: order.userId,
    duration: Date.now() - start,
    items: order.items.length,
    total: order.total,
});

logger.error('order.processing_failed', {
    orderId,
    errorCode: err.code,
    errorMessage: err.message,
    stackTrace: err.stack,
    // Include relevant state at failure point
    currentState: { status, step, retryCount },
});

2.2 Log Levels for Debugging

  ┌──────────┬────────────────────────────────────────────────────┐
  │ TRACE    │ Every function entry/exit (noisy, turn off by     │
  │          │ default)                                           │
  ├──────────┼────────────────────────────────────────────────────┤
  │ DEBUG    │ Detailed state dumps, intermediate values          │
  │          │ Enable when investigating specific issue           │
  ├──────────┼────────────────────────────────────────────────────┤
  │ INFO     │ Normal operations, key business events             │
  │          │ "Order created", "Payment confirmed"               │
  ├──────────┼────────────────────────────────────────────────────┤
  │ WARN     │ Anomalous but non-critical                         │
  │          │ "Retry attempt 2/3", "Cache miss"                  │
  ├──────────┼────────────────────────────────────────────────────┤
  │ ERROR    │ Something is broken — needs investigation          │
  │          │ "Database connection failed", "Timeout exceeded"   │
  └──────────┴────────────────────────────────────────────────────┘

2.3 Finding the Needle in the Haystack

# Find all ERROR lines in last hour
journalctl -u your-service --since "1 hour ago" | grep ERROR

# Follow logs in real-time for a specific request
tail -f /var/log/app.log | grep "order-123"

# Extract structured logs for a trace ID
cat app.log | grep "trace_id=abc123" | jq .

# Show how long something took (timing from structured logs)
cat app.log | grep "order.processed" | awk '{print $2}' | sort | uniq -c

# Count error types
grep ERROR app.log | grep -oP '"errorCode":"[^"]*"' | sort | uniq -c | sort -rn

# Find slow operations (duration > 1000ms)
cat app.log | jq 'select(.duration > 1000)' | jq -r '.message + ": " + (.duration|tostring)'

---

3. Production Debugging

3.1 The Production Debugging Checklist

  □ Is it happening NOW?
     → Check monitoring dashboard (latency, error rate, saturation)
     → Check alerting for related incidents
     → Check recent deployments/configuration changes
  
  □ Is it a known issue?
     → Search internal KB / runbooks / postmortems
     → Search Slack history for similar symptoms
     → Check GitHub issues / bug tracker
  
  □ Can we observe the problem?
     → Check logs (see section 2)
     → Check metrics (CPU, memory, disk, network, GC)
     → Check distributed traces (Jaeger/Tempo)
  
  □ Can we reproduce it in staging?
     → Same deployment version
     → Same data or data pattern
     → Same traffic pattern
     → If not reproducible → add more instrumentation

3.2 Safe Production Debugging

# IMPORTANT: Never SSH into production unless absolutely necessary
# Try these first:

# 1. Check monitoring (Grafana, Datadog, etc.)
# 2. Check logs (Loki, Splunk, ELK)
# 3. Check traces (Jaeger, Tempo)
# 4. Check health endpoints
curl -s http://service:8080/health | jq .
curl -s http://service:8080/metrics | head -50
curl -s http://service:8080/debug/vars  # Go expvar

# 5. Only if above fails → limited safe access
kubectl exec -it pod/app -- /bin/sh -c "curl localhost:8080/debug/pprof/heap"

# NEVER modify production data
# NEVER kill/restart services without understanding the impact
# ALWAYS have a rollback plan before making any change

3.3 Postmortem-Driven Debugging

When you encounter a bug, ask:

  ❓ "What type of bug is this?"
     - Logic bug (wrong condition, missing case)
     - Concurrency bug (race, deadlock, stale data)
     - Data bug (corruption, encoding, validation)
     - Configuration bug (wrong env, wrong feature flag)
     - Dependency bug (upstream change, API drift)
     - Resource bug (memory, file handles, connections)

  ❓ "Does this bug belong to a known category?"
     If yes → apply known pattern fix
     If no  → write a new postmortem entry about this pattern

---

4. Debugging by Bug Type

4.1 Concurrency / Race Conditions

// Symptom: intermittent crashes, wrong values, data corruption
// Root cause detection:

// Go race detector
go test -race ./...
go run -race ./...

// The output pinpoints the exact lines where race occurs:
// WARNING: DATA RACE
// Read at 0x00c00028c010 by goroutine 7:
//   main.updateCounter()
//       main.go:42 +0x39
// Previous write at 0x00c00028c010 by goroutine 5:
//   main.incrementCounter()
//       main.go:38 +0x58

// Node.js: debug async flow
// Use --async-stack-traces (Node 14+)
node --async-stack-traces app.js

// Track promise chains
Promise.config({
    warnings: true,
    longStackTraces: true
});

4.2 Memory Leaks

  Symptoms: RAM grows over time, GC overhead increases, OOM crashes

  Debug flow:
  1. Take a heap snapshot (time 0)
  2. Run operation n times
  3. Take another heap snapshot
  4. Compare: what grows?
     - Growing maps (map without cleanup) ← common
     - Growing slices (append without limit) ← common
     - Event listeners without removal
     - Closed-over variables in callbacks
     - Cached objects with no eviction

4.3 Deadlocks

// Go: detect deadlocks in tests
go test -v -timeout=5s ./...

// Go race detector catches lock ordering issues
go test -race ./...

// Debug deadlock: get all goroutine stacks
import "net/http/pprof"

func main() {
    // ...
}

// Then get stacks:
curl http://localhost:6060/debug/pprof/goroutine?debug=2

// Look for:
// goroutine 1 [chan receive]:
//     main.waitForResult()
//         main.go:25 +0x45
// goroutine 2 [chan receive]:
//     main.waitForResult()
//         main.go:25 +0x45
// → Two goroutines waiting on each other = deadlock

4.4 Heisenbugs (Bugs that Disappear When You Look)

  Symptoms:
  - Adding a log line "fixes" the bug
  - Debugger breakpoints "fix" the bug
  - Bug only happens in production, never staging

  Causes:
  - Timing-dependent bugs (race, channel, timeout)
  - Buffer flush / log delay timing
  - Heisenberg uncertainty principle of debugging:
    "The act of observing changes the behavior"

  Debug strategies:
  1. Use structured logging (less I/O impact than console.log)
  2. Use tcpdump / strace (observe without modifying)
  3. Add counters instead of log lines:
     metrics.counter('bug_scenario.hit').inc()
  4. Capture state, don't log it:
     Take periodic snapshots → analyze offline

4.5 Non-Deterministic Bugs

  Symptoms:
  - "Sometimes it works, sometimes it doesn't"
  - "I can't reproduce it"
  - "It only happens on Tuesdays"

  The 5 most common causes of non-determinism:
  1. Uninitialized memory → read before write
  2. Map iteration order (random in many languages)
  3. Goroutine/thread scheduling order
  4. Network timing / retry interactions
  5. Hash collision / random seed

---

5. Tools Arsenal

5.1 Quick Reference by Problem

  ┌────────────────────┬──────────────────────────────┐
  │ Problem            │ Tool                         │
  ├────────────────────┼──────────────────────────────┤
  │ CPU spike          │ pprof, perf top, top -H      │
  │ Memory leak        │ pprof heap, heap dump        │
  │ Deadlock           │ goroutine stack dump, lsof    │
  │ Race condition     │ race detector, tsan          │
  │ Slow DB query      │ EXPLAIN ANALYZE, pg_stat_activity│
  │ High GC            │ gc tracer, allocation profiler│
  │ Network issue      │ tcpdump, strace, ss, iperf   │
  │ Disk I/O           │ iostat, iotop, fio           │
  │ File handle leak   │ lsof -p PID, /proc/PID/fd    │
  │ Config wrong       │ diff config files, env vars  │
  │ SSL/TLS            │ openssl s_client, ssllabs     │
  └────────────────────┴──────────────────────────────┘

5.2 Universal Debugging Commands

# Process status
top -H -p <PID>           # Show per-thread CPU
strace -p <PID> -e trace=network  # Trace syscalls
lsof -p <PID>             # Open file descriptors
ls /proc/<PID>/fd/ | wc -l       # Count open files

# Network
ss -tulpn                  # List listening ports
tcpdump -i eth0 port 8080 -w capture.pcap  # Packet capture

# Disk
iostat -x 1                # Disk I/O stats
df -h                      # Disk space
du -sh /path               # Directory size

# System
dmesg | tail -20           # Kernel messages (OOM kills!)
free -m                    # Memory
ulimit -a                  # Resource limits

---

6. The Debugging Mindset

6.1 What Great Debuggers Do Differently

  As a junior dev, I'd panic and change random things.
  As a senior engineer, I use the scientific method.

  ┌──────────────────────────────────────────────────────────────┐
  │  Junior Engineer:                                            │
  │  "Oh no! The database is failing! Let me restart it!"       │
  │  → Restarts the DB → "It works now!"                        │
  │  → Same bug happens tomorrow                                 │
  │                                                               │
  │  Senior Engineer:                                            │
  │  "The database is failing. Let me check the logs."          │
  │  → Finds "disk space 100% full"                             │
  │  → Cleans up old data, sets up disk alert                   │
  │  → Bug never comes back                                      │
  └──────────────────────────────────────────────────────────────┘

6.2 The Two-Day Rule

  If you can't fix a bug after 2 hours of focused debugging:
  
  1. Step away from the keyboard (5-10 minutes)
     → Fresh perspective is the #1 debugging tool
  
  2. Explain the bug to someone else (rubber duck debugging)
     → Saying it out loud forces clarity
  
  3. Write down what you KNOW vs what you ASSUME
     → Most stuck bugs come from a wrong assumption
  
  4. Ask yourself:
     "What would have to be TRUE for this bug to reproduce?"
     "What evidence DISPROVES my current hypothesis?"
  
  5. If still stuck after 2 more hours:
     → Escalate or pair with someone who hasn't seen the issue

6.3 The Rubber Duck Debugging

# Pseudocode execution:
def rubber_duck_debug(code_bug):
    """
    Explain the code line by line to a rubber duck.
    The duck doesn't know anything, so you need to
    be precise enough that a complete beginner would
    understand.
    
    90% of the time, you find the bug mid-explanation.
    """
    while not bug_found:
        for line in code_bug:
            explain_out_loud(line)

---

7. Debugging Anti-Patterns

  ❌ "Let me just try restarting it"
     → You learned nothing. Same bug will return.

  ❌ "Let me change this randomly and see if it helps"
     → If it "fixes" the bug, you still don't know WHY.
     → The "fix" might have introduced a different, worse bug.

  ❌ "I checked the code, it looks fine"
     → The code is NOT fine — the bug proves it.
     → The bug is in the gap between "what you think the code does"
       and "what the code actually does."

  ❌ "It must be a compiler/interpreter bug"
     → It's NEVER the compiler. (Google's postmortem: zero compiler bugs)
     → The compiler is better tested than your code.

  ❌ "This worked yesterday, nothing changed!"
     → Something ALWAYS changed. Find it.
     → Deployment, config, data, traffic pattern, time of day.

---

References

• references/debugging-recipes.md — Language-specific debugging recipes (Go/Node/Python/Rust)
• references/logging-standards.md — Structured logging patterns for debuggability
• references/post-deploy-checks.md — What to check immediately after a deployment