Linux perf and PMU counters

perf ships in the kernel source tree under tools/perf, so its feature set follows the running kernel line (mainline 7.2, LTS 6.18). This skill covers the three jobs perf does for a CPU-bound program: count hardware events with perf stat, sample where cycles go with perf record and perf report, and attribute samples to instructions with perf annotate. It also covers PAPI region instrumentation, counter-derived metrics, memory-bandwidth counters, and raw PMU event lookup.

Contract

Field	Bound contract
Trigger	A Linux program is slow or CPU-bound and the user wants hot functions, counter totals, or instruction-level attribution from perf, or asks for PAPI instrumentation, IPC, miss rates, MPKI, or counter-based memory bandwidth. Also fires on [unknown] frames, Permission denied from perf, or a request to feed perf data to a flamegraph pipeline.
Authority	Reversible local: writes only perf.data files (or the -o target), compiled or PAPI-instrumented test binaries, and PMU tool checkouts in the working directory; rollback is deleting those files. sysctl and MSR module changes are proposed to the user, never applied. No remote mutation.
Side effect	Profile and counter data on disk. Counting mode (perf stat) adds negligible overhead; PAPI adds instrumentation overhead around the measured region; sampling mode slows the program by a small factor that depends on sample frequency and call-graph mode.
Done	The report names the hot symbols with their sample share, every counter metric carries its formula, raw counts, workload, microarchitecture, and comparison it supports, and any unresolved frame is explained with its fix.

Inputs

• Target program and a representative workload (required). Short programs need a longer input or a higher -F.
• Debug symbols (required for symbol names and source lines): build with -g. For frame-pointer call graphs add -fno-omit-frame-pointer; otherwise use --call-graph dwarf.
• CPU model (lscpu) and whether the host is virtualized: a VM often exposes no hardware events.
• Value of /proc/sys/kernel/perf_event_paranoid (gathered by the skill). Meaning: 2 allows user-space measurement of the caller's own processes only, 1 adds kernel measurement, 0 adds CPU-wide events, -1 removes all restrictions. Values above 2 block unprivileged perf entirely.
• A question: which function is hot, which counter or derived metric is high, which source line causes misses, or which instruction stalls. The question picks the subcommand or counter mode.
• For PAPI: libpapi headers and library (-lpapi), PAPI 7.3.0 at grounding.

Procedure

1. Check permissions. Read cat /proc/sys/kernel/perf_event_paranoid. If the value blocks the intended measurement, propose sudo sysctl -w kernel.perf_event_paranoid=1 for the session, or the persistent form echo 'kernel.perf_event_paranoid=1' | sudo tee /etc/sysctl.d/99-perf.conf followed by sudo sysctl -p /etc/sysctl.d/99-perf.conf. Wait for the user to apply it. Done when: perf can open the events the plan needs, or the user has declined and the plan is narrowed to what the current level allows.

2. Build with symbols: gcc -g -O2 -fno-omit-frame-pointer -o prog main.c. Keep the optimization level of the real build, because a -O0 profile points at code that does not exist in production. Done when: nm prog | head shows symbols and the binary runs the workload.

3. Counter mode. Confirm what the CPU exposes with perf list hw, perf list cache, and perf list pmu; the first prints generic hardware events, the second cache events, and the third CPU-specific names. If perf list hw is empty, the host is virtualized or the PMU is locked and only software events remain. Run perf stat ./prog for the default set, then perf stat -e cycles,instructions,cache-references,cache-misses,branches,branch-misses ./prog for named events. For cache and TLB pairs, use perf stat -e L1-dcache-loads,L1-dcache-load-misses,LLC-loads,LLC-load-misses ./prog and perf stat -e dTLB-loads,dTLB-load-misses,iTLB-loads,iTLB-load-misses ./prog. Use perf stat -r 5 -e instructions,cycles ./prog to average five runs and print the spread; attach to a live process with perf stat -p <pid> sleep 10. Group events that must be measured together in one -e list; when more events are requested than counters exist, perf multiplexes and scales them, and prints the percentage of time each was counted. Done when: the events for the chosen metric appear in perf list (or software-only operation is recorded), and raw counts for every event are recorded with the run spread.

   Read derived metrics as conditional signals, not verdicts:

   Metric	Formula	Reading
   IPC	instructions / cycles	Ceiling is the core's issue width. Compare two builds on the same core; a memory-bound or pointer-chasing loop has a low IPC by nature, and a lower IPC after a change is the signal, not the level
   L1 miss rate	L1-dcache-load-misses / L1-dcache-loads	Meaningful with the absolute miss count beside it; streaming through a large array raises it by design
   LLC miss rate	LLC-load-misses / LLC-loads	High rate plus high absolute count means DRAM traffic; check bandwidth below
   Branch miss rate	branch-misses / branches	Data-dependent branches drive it; if sorting the input lowers it, the branches are the cost
   MPKI	misses / (instructions / 1000)	Normalizes misses to work done; use it to compare builds with different instruction counts

   Done when: each metric carries its formula, raw counts, workload, CPU model, and the build comparison it supports.

   When whole-program totals are too coarse, use the PAPI region mode. The low-level API (the old PAPI_start_counters and PAPI_stop_counters are gone from PAPI 7.x) is:

   #include <papi.h>
   #include <stdio.h>
   
   int main(void) {
       int events[] = { PAPI_TOT_INS, PAPI_TOT_CYC, PAPI_L2_TCM, PAPI_BR_MSP };
       long long values[4];
       int set = PAPI_NULL;
   
       if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) return 1;
       if (PAPI_create_eventset(&set) != PAPI_OK) return 1;
       for (int i = 0; i < 4; i++)
           if (PAPI_add_event(set, events[i]) != PAPI_OK) return 1;
   
       PAPI_start(set);
       do_work();
       PAPI_stop(set, values);
   
       printf("IPC %.2f  L2 misses %lld  branch mispredicts %lld\n",
              (double)values[0] / values[1], values[2], values[3]);
       return 0;
   }
   

   Build with gcc -O2 -g -o prog prog.c -lpapi. The high-level API wraps the same region in PAPI_hl_region_begin("name") and PAPI_hl_region_end("name") with events chosen through PAPI_EVENTS. papi_avail -a lists the presets this CPU supports (PAPI_TOT_INS, PAPI_TOT_CYC, PAPI_L1_DCM, PAPI_L2_TCM, PAPI_L3_TCM, PAPI_BR_MSP, PAPI_TLB_DM, PAPI_FP_INS, PAPI_VEC_INS); papi_native_avail lists native events. A preset absent from papi_avail -a fails in PAPI_add_event. Done when: the region's counts print and the preset list confirms each event.

   When LLC misses are high, measure memory bandwidth. Uncore IMC events count DRAM transactions: perf stat -e uncore_imc/cas_count_read/,uncore_imc/cas_count_write/ -a ./prog on Intel hosts that expose the uncore_imc PMU in perf list pmu; system-wide -a needs paranoid level 0 or root. Intel PCM (https://github.com/intel/pcm, built with cmake, binaries in build/bin) reports socket bandwidth with pcm-memory 1 and core metrics with pcm 1, both with a -csv mode. PCM reads MSRs and needs root or CAP_SYS_RAWIO, or its daemon mode for unprivileged readers. Done when: achieved bandwidth is recorded next to the platform peak.

   When generic names do not cover the question, reach raw events: perf list pmu prints the CPU's named events, and the raw form is perf stat -e cpu/event=0x..,umask=0x../. pmu-tools (git clone https://github.com/andikleen/pmu-tools, run ./ocperf.py from the checkout) translates vendor event names to raw codes; showevtinfo from libpfm4 lists what the library knows. Done when: the raw event's name and code are recorded together.

4. Sample with perf record. Use perf record -F 999 -g ./prog for a frame-pointer call graph at 999 Hz (an odd frequency avoids lock-step with periodic timers). Use perf record -F 999 --call-graph dwarf ./prog when the binary or its libraries lack frame pointers; cap the copied stack with --call-graph dwarf,4096 if the data file grows too fast. Sample a specific event with -e cache-misses; for miss attribution use -e LLC-load-misses and add the :p or :pp suffix when the PMU supports precise sampling. Attach with -p <pid> sleep 30, sample all CPUs with -a, and name the output with -o app.perf.data. Done when: perf record prints the sample count and the data file exists.

5. Read the report. perf report opens the TUI on perf.data; perf report -i app.perf.data reads a named file; perf report --stdio prints text; perf report --no-children shows self time instead of inclusive time; perf report --sort comm,dso,sym groups by process, library, and symbol. In the TUI, Enter expands a symbol, a annotates it, d filters by DSO, t filters by thread, and ? lists keys. Done when: the top symbols by self time are listed with their percentages.

6. Annotate the hot symbol. perf annotate -i perf.data --symbol=<name> --stdio prints the disassembly with per-instruction sample shares; s in the TUI toggles source when debug info is present. A high share on a load instruction (mov, vmovdqa) marks a stall on that load, usually a cache miss. For miss attribution, the percentage next to a source line is the share of miss samples landing there; a loop body with a strided access pattern shows the load instruction at the top. Done when: the hottest instructions and source lines (or, for miss attribution, the top source lines by miss samples) are named.

7. Watch live when the workload is long-running: sudo perf top -g or sudo perf top -p <pid>. Done when: the live view confirms or refutes the recorded hotspot.

8. Export for a flamegraph: perf script -i perf.data > out.perf, then hand out.perf to flamegraphs (stackcollapse-perf.pl and flamegraph.pl). Done when: out.perf exists.

9. Resolve any bad frames or empty output using the failure table below before reporting. Done when: no [unknown] frame remains in the top entries, or its cause is stated.

Event names to reach for: hardware cycles, instructions, cache-references, cache-misses, branches, branch-misses, stalled-cycles-frontend, stalled-cycles-backend; cache L1-dcache-loads, L1-dcache-load-misses, LLC-loads, LLC-load-misses, dTLB-load-misses; software context-switches, cpu-migrations, page-faults, major-faults; tracepoints such as sched:sched_switch (need paranoid level 0 or root). perf list prints what this CPU exposes; perf list hw, perf list cache, perf list sw, and perf list tracepoint filter by class. Raw PMU codes are CPU-specific, and :u, :k, :p, and :pp modifiers restrict counting or request precise sampling. Full event notes live in references/events.md.

Failure and recovery

Failure	Cause	Fix
Permission denied or Error: Access to performance monitoring and observability operations is not permitted	perf_event_paranoid above the needed level	Propose the sysctl change from step 1, or run the measurement with sudo
[unknown] frames	No frame pointers or no debug info	Rebuild with -g -fno-omit-frame-pointer, or record with --call-graph dwarf
Kernel frames show as addresses	Kernel symbols hidden	Record with sudo; kptr_restrict at 0 (/proc/sys/kernel/kptr_restrict) exposes /proc/kallsyms; install the kernel debug symbols package for the running kernel
Empty report for a short program	Too few samples	Raise -F toward the value in kernel.perf_event_max_sample_rate, or lengthen the workload
DWARF unwinding is slow or the data file is huge	Whole user stack copied per sample	--call-graph dwarf,4096 or a smaller size, or add frame pointers and use -g
Counter shows <not supported>	Event absent on this CPU or virtualized	Pick a name from perf list; in a VM only software events may exist
Counter shows <not counted> or a low multiplex percentage	More events than counters	Split the events across runs, or group the ones that must be read together
Miss rate looks alarming	Absolute count is small	Report the count beside the rate; a rate on few references is noise
PAPI_add_event fails	Preset not available or counter conflict	Check papi_avail -a; reduce the event set
Uncore events are missing	PMU not exposed or paranoid level too high	Propose paranoid level 0 or root for -a; use PCM or the vendor profiler (intel-vtune-amd-uprof)
Numbers differ run to run	Frequency scaling, SMT sibling, or cache state	Use -r, pin with taskset, and report the spread

A partial profile or counter report is reported as partial: the report states which steps ran and which measurement is missing.

Output

A profile report containing the counter table from step 3 with formulas, raw counts, run spread, workload, CPU model, and the comparison it supports; the top symbols by self time from step 5 with percentages; the hot instructions and source lines from step 6; the bandwidth figure and platform peak when measured; the PAPI region output when instrumented; the path of every data file written; and any unresolved frame with its cause and fix.