Kernel debugging

Contract

Field	Bound contract
Trigger	A kernel panic or oops with an unclear trace, live debugging of a module with kgdb or kdb, tracing a function without recompiling, enabling driver debug output, or analyzing a vmcore.
Authority	Read-only. Emits analysis and commands for the operator to run on the target; no file writes, no rollback needed. No remote mutation.
Side effect	Diagnostic commands and an evidence-backed verdict in chat. Nothing is written.
Done	The failing path is named with evidence from a trace, backtrace, or crash session, or the next diagnostic step is stated with the reason.

Inputs

1. Symptom (required): panic, oops, hang, lockup, or wrong behavior.
2. Target access (optional): serial console, root shell, mounted debugfs, or a vmcore file.
3. Kernel build (optional): vmlinux with debug info for the running kernel; mainline 7.2 or LTS 6.18 assumed when not stated.

Procedure

1. Read the ring buffer before anything else.

   dmesg -T -l err,warn              # filter by level
   dmesg -w                          # follow live
   cat /proc/sys/kernel/printk       # current default minimum boot-default
   echo 8 > /proc/sys/kernel/printk  # raise verbosity
   

   Driver dev_dbg() output stays hidden until dynamic debug or a DEBUG define turns it on. Done when: the level is raised and the first fault line is captured.

2. Break into kgdb when the target is alive. Boot with kgdboc=ttyS0,115200 kgdbwait, or attach at runtime and trigger the break from sysrq.

   echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc
   echo g > /proc/sysrq-trigger
   

   gdb vmlinux
   (gdb) set serial baud 115200
   (gdb) target remote /dev/ttyUSB0
   (gdb) bt
   

   A USB serial adapter shows up as ttyUSB0 and works the same as a built-in port. Done when: gdb reports a backtrace on the stopped target.

3. Use kdb when no host gdb exists. CONFIG_KGDB_KDB gives an in-kernel shell at the sysrq break (echo k > /proc/sysrq-trigger). Commands: bt backtrace, ps process list, lsmod modules, md <addr> memory, rd registers, id <addr> disassembly, cpu <n> switch CPU, go continue. Done when: the faulting frame is visible without a host debugger.

4. Trace without recompiling: ftrace.

   cd /sys/kernel/debug/tracing
   echo function > current_tracer
   echo schedule > set_ftrace_filter
   echo '*probe*' > set_ftrace_notrace
   echo 1 > tracing_on
   cat trace_pipe
   echo 0 > tracing_on && echo nop > current_tracer
   

   trace-cmd record -p function -l schedule,do_page_fault
   trace-cmd report          # kernelshark reads the same trace.dat
   

   Requires debugfs mounted and root. Done when: the trace shows the calls in question with the filters narrowing it.

5. Probe one suspect function with kprobes.

   echo 'p:myprobe do_sys_open $arg1 $arg2' > /sys/kernel/debug/tracing/kprobe_events
   echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
   cat /sys/kernel/debug/tracing/trace
   echo 'r:myret do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events
   # cleanup
   echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events
   

   An in-kernel probe registers a struct kprobe with .symbol_name and a .pre_handler through register_kprobe(). Done when: the probe fires and the fetch arguments read sanely.

6. Turn on driver debug prints with dyndbg.

   echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
   echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control
   grep mydriver /sys/kernel/debug/dynamic_debug/control
   

   Boot time: dyndbg="module mydriver +p" on the kernel command line. Needs CONFIG_DYNAMIC_DEBUG. Done when: the driver's debug lines appear in the log.

7. Analyze the crash dump. Reserve memory with crashkernel=256M, then use the distro tool (kdump-config show on Ubuntu, kdumpctl status on RHEL). After a panic the vmcore lands in /var/crash/.

   crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore
   crash> bt
   crash> log
   crash> ps
   crash> kmem -i
   crash> mod
   

   Compress with makedumpfile -c -d 31 /proc/vmcore /tmp/vmcore; the dump level is a number after -d. The vmlinux must carry debug info for the exact running kernel, usually the distro linux-image-*-dbg package. Done when: the crash session yields the panic backtrace.

8. Tune the target for reproduction.

   sysctl kernel.panic_on_oops=1      # stop at the first oops, in a VM
   sysctl kernel.softlockup_panic=1
   sysctl kernel.nmi_watchdog=1
   # boot: slub_debug=P,pagealloc     # poison slab for corruption hunts
   

   Done when: the target halts at the defect instead of limping on.

9. Triage by issue class.

   Panic or oops        -> dmesg, then crash on the vmcore
   Driver logic bug     -> dyndbg, then ftrace function_graph
   Latency regression   -> perf record -g -a, trace-cmd
   Intermittent         -> kprobe the suspect path
   Module crash         -> kgdb, audit module refcounts
   

   Done when: the next tool follows from the class.

Failure and recovery

Symptom	Cause	Fix
kgdb will not connect	Wrong tty or baud	Match kgdboc to the adapter, check both ends
Trace stays empty	Tracer not set or tracing off	echo function > current_tracer, echo 1 > tracing_on
kprobe registration fails	Inlined symbol or CONFIG_KPROBES off	Pick a symbol present in /proc/kallsyms, or use a tracepoint
dyndbg has no effect	CONFIG_DYNAMIC_DEBUG disabled	Rebuild the kernel with the option
crash rejects the vmcore	Wrong vmlinux debug symbols	Install the dbg package matching uname -r
sysrq dead	kernel.sysrq is 0	echo 1 > /proc/sys/kernel/sysrq

Failure class	Behavior
Break hangs the target	Fall back to kdb in-kernel or to the kdump path; do not retry the same break without changing the transport.
kprobe floods the trace	Narrow to one symbol, or switch to a kretprobe that only logs $retval.
crash session mismatches the kernel	Refuse analysis; a mismatched vmlinux produces plausible garbage. Fetch the matching debug package first.
Debug controls write nothing	debugfs is not mounted: mount -t debugfs none /sys/kernel/debug.

Output

1. The faulting function or subsystem named with captured evidence.
2. The command transcript used.
3. The next diagnostic step when the cause is still open.