Kernel internals

Contract

Field	Bound contract
Trigger	A diagnosis or explanation inside the scheduler, the memory allocators, the VFS, the page cache, or the OOM killer; also reading kernel source in kernel/sched/, mm/, or fs/.
Authority	Read-only. Analysis and measurements stay in chat; no file writes, no rollback needed. No remote mutation.
Side effect	A subsystem diagnosis grounded in the running kernel's version and measurements. Nothing is written.
Done	The observed behavior is explained by one subsystem mechanism with the measurement that confirms it, or the mismatch between the claim and the kernel is named.

Inputs

1. The question or symptom (required): latency, allocation failure, cache behavior, memory pressure, OOM victim.
2. Target access (optional): /proc, /sys, and perf on the running machine.
3. Kernel version (optional): mainline 7.2 or LTS 6.18 assumed when not stated.

Procedure

1. Pin the kernel and the scheduler it runs. uname -r first. EEVDF (Earliest Eligible Virtual Deadline First) replaced CFS: merged as an option in 6.6, the transition completed in 6.12, so mainline 7.2 and LTS 6.18 run EEVDF. On 6.6 through 6.11, check which one is active before applying CFS reasoning. Each CPU carries a runqueue with three classes:

   Per-CPU runqueue (struct rq)
   +-- cfs_rq  fair-class tasks ordered by eligibility and deadline
   +-- rt_rq   real-time tasks (FIFO/RR)
   +-- dl_rq   SCHED_DEADLINE tasks
   

   Done when: the kernel version and the active fair scheduler are recorded.

2. Measure scheduling before touching anything. chrt -p <pid> reads the class, /proc/<pid>/sched the per-task counters, /sys/kernel/debug/sched/debug the runqueue state, and perf sched record with perf sched latency the delay picture. The old CFS sysctls (kernel.sched_latency_ns, kernel.sched_min_granularity_ns, kernel.sched_wakeup_granularity_ns) were removed in 6.12 with EEVDF; they exist only on pre-6.12 kernels. vruntime is virtual runtime: CPU time consumed, normalized by weight. Under EEVDF a task with positive lag is eligible and the earliest eligible virtual deadline wins. Done when: one measurement names the unfair or slow path.

3. Read the buddy allocator from /proc. Physical pages are handed out in power-of-two orders; order 0 is one page, 4 KiB on default x86-64 and arm64 configs. /proc/buddyinfo shows free blocks per order and zone, /proc/zoneinfo the per-zone counts. ZONE_HIGHMEM exists on 32-bit and selected configs only; on 64-bit x86-64 and arm64 essentially all RAM sits in ZONE_NORMAL. Done when: the fragmentation picture comes from buddyinfo, not from a guess.

4. Pick the allocation API by the constraint that binds.

   API	Use when
   kmalloc(size, GFP_KERNEL)	Physically contiguous, size bounded by KMALLOC_MAX_SIZE (arch dependent)
   kzalloc(size, flags)	Zeroed kmalloc
   vmalloc(size)	Virtually contiguous, large, may be physically fragmented
   __get_free_pages(gfp, order)	Direct page-granular allocation

   void *buf = kmalloc(4096, GFP_KERNEL);
   if (!buf)
       return -ENOMEM;
   kfree(buf);
   

   Done when: the chosen API matches the contiguity and size bound the caller actually needs.

5. Read VFS pressure through its caches. A path lookup walks the dentry cache, the inode carries metadata and ops, and struct file holds per-open state. /proc/mounts lists mounts, vm.vfs_cache_pressure sets how eagerly the dentry and inode caches are reclaimed (higher reclaims sooner), and /proc/<pid>/fd counts open files. Done when: cache pressure is distinguished from file-leak pressure.

6. Judge the page cache with the writeback state. Cached in meminfo is page cache and tmpfs; Dirty pages wait for writeback. blockdev --getra reads the readahead window and --setra changes it per device. echo 3 > /proc/sys/vm/drop_caches after sync drops caches for benchmarking only; it is destructive to performance and proves nothing in steady state. Done when: cache and dirty state are read together.

7. Interpret /proc/meminfo field by field.

   Field	Meaning
   MemTotal	Total usable RAM
   MemFree	Completely unused pages
   MemAvailable	Estimated allocatable memory including reclaimable cache
   Cached	Page cache and tmpfs
   Buffers	Block device metadata cache
   SwapTotal / SwapFree	Swap space
   Dirty	Pages pending writeback
   AnonPages	Anonymous heap and stack pages
   Slab	Kernel object cache
   SReclaimable / SUnreclaim	Slab split by reclaimability

   MemAvailable low + Cached high -> reclaim page cache, pressure is not fatal
   AnonPages high + SwapFree low  -> OOM risk
   Slab huge                      -> kernel object leak, check /proc/slabinfo
   Dirty high                     -> writeback lag, check the I/O scheduler
   

   Done when: the pressure verdict comes from two or more fields read together.

8. Explain the OOM pick, then move the pick. The killer scores candidates from memory use, child processes, and oom_score_adj (-1000 to 1000). Protect a daemon by lowering its adj; cap a service with a cgroup memory.max instead of tuning adj upward for everyone else. Events land in dmesg and journalctl -k. Done when: the victim choice is explained and the correction is in place for the next episode.

Failure and recovery

Symptom	Cause	Fix
kmalloc: allocation failed	Fragmentation or size over KMALLOC_MAX_SIZE	Use vmalloc, reduce pressure, reserve GFP_ATOMIC for non-sleeping paths
High iowait with low MemAvailable	Page cache thrashing	Add RAM or cut the working set, tune vfs_cache_pressure last
OOM kills the wrong process	Big user with a neutral adj	Set oom_score_adj, cap with memory.max
Unfair scheduling	Real-time starving the fair class	chrt audit, isolcpus to split CPUs
SLUB corruption	Use-after-free in a module	KASAN, slub_debug=P
Slow file reads	Readahead window too small	Raise readahead, check the backing device

Failure class	Behavior
The claim contradicts the running kernel	Trust the kernel: re-read the sysctl or counter, and restate the claim with the version attached.
A tunable is missing	The version removed it (EEVDF removed the CFS sysctls in 6.12); do not advise it for newer kernels.
Two metrics disagree	Measure again under the same load before explaining; never average across episodes.

Output

1. The mechanism that explains the observation.
2. The measurements that confirm it, with the kernel version attached.
3. The specific correction to apply.