Build & Test - imp

Hard rules

#	Rule	Detail
1	No CUDA toolkit on the host	Everything runs in Docker. Toolchain nvidia/cuda:13.4.2-devel-ubuntu26.04, GCC 15.2, C++23 for host AND device (CMAKE_CUDA23_STANDARD_COMPILE_OPTION shim at the top of CMakeLists.txt).
2	build/ and build-dev/ are root-owned	make dev-clean, or docker run --rm -v $PWD:/src -w /src ubuntu rm -rf build. Never sudo.
3	No --mount=type=cache on the build dir in the Dockerfile	03a2cc19: ninja reused stale objects. The ccache mount (id=imp-ccache) is content-addressed and stays; docker builder prune empties it, which only costs one cold build.
4	models/ in the repo is a symlink farm	Custom docker run mounts $HOME/models:/models; Makefile targets already do.
5	Dependency pins live ONLY in cmake/imp-deps.cmake	CUTLASS, GTest, httplib, nlohmann/json, each a TAG plus the SHA the build fetches. make build injects both via scripts/dep_build_args.sh. check_dep_pins.sh runs offline in the deps gate (drift, missing SHA, unpinned uses:) and --online in Lint (tag must still resolve to the pinned commit).
6	CI has no GPU runner	Test job skipped unless repo var HAS_GPU_RUNNER=true. GPU correctness and perf are gated LOCALLY, by hand or with IMP_HOOK_GPU=1 on the hooks: pre-commit (make build + GPU modules or the full make test-gpu), pre-push (make verify-fast, plus the paired A/B make verify-ab on kernel diffs). The hook prints the skipped command; run it before the PR.
6a	git commit / git push are CPU jobs	Both hooks run make preflight only. With IMP_HOOK_GPU=1 they and make GPU targets refuse while another worktree holds scripts/gpu_lock.sh (scripts/require_free_gpu.sh, hook_gpu in scripts/hook_lib.sh), then check load. Holding the card across jobs: scripts/gpu_lock.sh acquire "<purpose>" in your worktree, release after, status shows the holder (lock /tmp/gpu-lock, shared with ~/.claude/skills/gpu-stats/gpu-lock.sh, keeper TTL 4 h; a live pid in the old /tmp/imp-gpu.lock also refuses). Still send the START/END handshake to other sessions. No --no-verify: the hooks touch no GPU by default. The skipped GPU tests are owed: run them and report the result in the PR (CI Test never runs: .github/workflows/ci.yml:999).
7	Required check = Build, ruleset 14716423	Static gates run as its FIRST step (scripts/ci_static_gates.sh, unfiltered) and block the merge since #1527. Rename the job without the ruleset and every PR sits at mergeStateStatus=BLOCKED.
8	main's CI status is stale by default	Auto-merge squashes with GITHUB_TOKEN, which starts no workflow run. Judge a fix by the PR run or the file; refresh with gh workflow run CI --ref main.

Command table

Task	Command	Time
Incremental build (inner loop)	make dev	2-14 s
Incremental build + CI unit lane	make dev-test (= ctest -L unit)	~3 s
Full image (the gate; anything you measure or push)	make build -> imp:test (main checkout) or imp:test-<dir>-<hash8> (worktree, scripts/image_tag.sh)	~6 min cold; 10-40 s from a warm ccache (595 TUs, 2026-09-11); 0 s when the image already carries the tree (imp.tree label)
CPU unit binary	make test-unit (imp-tests-unit)	<5 s
Full GPU suite	make test-gpu	4-10 min
E2E on real models	make test-e2e	Qwen3-4B-Instruct-2507-Q8_0, Qwen3.5-4B-mxfp4, gemma-4-26B-A4B Q4_K_M, MOE_MODEL (gpt-oss-20b-mxfp4), Qwen3-Coder-30B FP4, Nemotron-3-Nano NVFP4 (paths in Makefile)
Server e2e (the ONLY place handlers/batching run end to end)	make test-server	includes degen_suite.py and tests/test_server_tracing.py
Rerank / agents / external harnesses	make test-rerank, make test-agents, make test-agents-external	GPU + local model
NIAH retrieval gate, spec fidelity	make test-niah, make test-spec-fidelity	spec-fidelity needs ~26 GB free
Vision goldens	make test-vision (IMP_VISION_GOLDEN_DUMP=1 regenerates)
Chat-template goldens	make chat-goldens (network, writes tests/refs/chat_template_goldens.h)
Pre-push gate	make verify-fast	18 s without perf gate, 36 s with
Full / chunked / north-star gates	make verify, make verify-chunked, make verify-north-star	~5 min full
Kernel register/spill ratchet (CI kernels gate)	make kernel-resources, re-pin make kernel-resources-update	needs libimp.a
clang-format	make format-check; never bare make format	repo is not format-clean
clang-format, changed lines only (what CI checks)	per file: ranges from git diff -U0 main -- f, then docker run --rm --user $(id -u):$(id -g) -v $PWD:/work -w /work $CLANG_FORMAT_IMG clang-format --style=file -i --lines=a:b ... f (--dry-run to check)	image name: CLANG_FORMAT_IMG in Makefile
Host ASan/UBSan over test-core/test-text	make asan	works on WSL2
compute-sanitizer	make sanitize	does NOT work on WSL2 (WDDM)
Alloc-interpose census	make check-alloc-interpose (build-interpose/)	never benchmark it (reads ~3% low)
Server gcov	make coverage	GPU + model
API mock contract	pytest tests/api/ or tests/api/run_mock_tests.sh	no GPU; the mock lane also runs the nomodel tests
GPU free?	make check-gpu (scripts/require_free_gpu.sh: utilisation AND memory)

Filtered GPU run (GTEST_FILTER is not threaded through make test-gpu):

docker run --rm --gpus all -v $HOME/models:/models \
  -e IMP_TEST_MODEL=/models/Qwen3-8B-Q8_0.gguf \
  imp:test imp-tests --gtest_filter="DegenerationTest.*"

DetEvalE2ETest.* matches 0 tests and prints PASSED; the working filter is *DetEvalE2ETest*.

Binaries

Where	What
imp:test image	imp-server, imp-cli, imp-bench, imp-quantize, imp-tests (full GPU), imp-tests-unit (generated wrapper = ctest -L unit: test-core + test-text + e2e unit filter), modules test-core test-text test-compute test-attention test-quant test-kv test-moe-gdn test-e2e
build-dev/ (host, from make dev)	same binaries; run them inside imp:toolchain with -v $PWD:/src -w /src (that mount hides the image's own /src)

tests/test_ssm.cu -> test-moe-gdn. SamplingTest -> test-compute. A new CPU test belongs in test-core. Lane census: python3 tools/check_test_lanes.py --report (the Test lanes check fails on a test in no ctest entry or a unit-lane test demoted vs the base; a new GPU test in a gpu module needs no edit). The unit lane has no skips (guard_unit_skips in ctest -L unit fails on any): a test that needs a GPU or a model file goes to test-kv or to test-e2e outside _unit_e2e_filter; fixture files live in tests/fixtures/ (IMP_TEST_FIXTURES_DIR). A new tool binary needs a cp in the builder stage AND a COPY --from=builder line in the Dockerfile.

Gates and hooks

Gate group	Runs where	Content
filesize	Build, hooks	check_filesize.py ([allow] pin = ceiling, +25 slack, shrink free), check_determinism_sites.py, check_dead_inline_accessors.py, check_log_fatal.py
lanes	Build, hooks, own check Test lanes (#1770)	check_test_lanes.py --report (macros per lane, no-lane delta vs base); guard_unit_skips inside ctest -L unit fails on any runtime skip
entrypoint	Build, hooks	tests/test_entrypoint.sh drives docker-entrypoint.sh against a stub (25 assertions)
alloc	Build, hooks	check_alloc_sites.py + check_alloc_pairs.py (allowlist is two-way: removing a site needs the allowlist edit)
kernels	Build only (needs artifact)	make kernel-resources vs tools/kernel_resource_baseline.txt (REG >= 240 or non-zero local frame)
launchguards	Build, hooks	check_launch_guards.py
docs	Build, hooks, Docs job	sync_docs.py --check, docs_lint.py
citations	Build, hooks	check_doc_citations.py over all living docs (#1783)
hygiene	Build, Release hygiene job only	check-release.sh with SKIP_VERIFY=1 (also resolves relative links in docs)

Advisory jobs: Lint, clang-tidy, Mock API contract, Real API contract (model-less), Sanitizers, PTX fallback.

• Hooks: make install-hooks copies scripts/pre-commit.hook and scripts/pre-push.hook into git rev-parse --git-path hooks (shared by all worktrees). Never edit .git/hooks/*: guard_precommit_filter in the CPU lane diffs them against the tracked files.
• make build runs scripts/build_image.sh: skipped when the worktree image already carries the tree and build args (labels imp.tree = scripts/image_tag.sh tree, unstaged and untracked files included; imp.build; imp.worktree). IMP_FORCE_BUILD=1 builds anyway. Runners refuse an image whose imp.tree is not this tree (IMP_ALLOW_FOREIGN_IMAGE=1 overrides). Saves 3.5 min per hook stage on one tree.
• Pre-commit = make preflight (~20 s); GPU step with IMP_HOOK_GPU=1 only: make build, then only the test modules the staged diff reaches (modules_for in scripts/pre-commit.hook: src/memory|runtime -> test-core test-kv test-e2e, src/compute -> test-compute test-attention test-quant test-kv test-moe-gdn, src/exec -> test-e2e test-moe-gdn test-compute test-attention, src/model|quant -> test-quant test-text test-e2e test-core, a tests/ file -> its module); the full make test-gpu (3108 tests in imp-tests --gtest_list_tests, 2026-09-28) on include/ src/core src/api CMakeLists cmake/ changes or with IMP_PRECOMMIT_FULL=1. .md .py .hook skip.
• Pre-push = make preflight; GPU step with IMP_HOOK_GPU=1 only: require_free_gpu.sh, then make verify-fast; the perf gate runs only when the diff matches PERF_RE (src/{compute,exec,quant,runtime,model}/, any .cu/.cuh, cmake/, CMakeLists.txt, tests/perf_baseline*). On src/compute or src/exec diffs the paired A/B (make verify-ab, 2 %, ~5 min) replaces the single-arm perf gate; IMP_SKIP_AB=1 falls back to it. Skips: IMP_VERIFY_SKIP_PERF/_VRAM/_GRAPHS=1.
• Hooks run make preflight: every ci_static_gates.sh group, hygiene included.
• docs_lint.py writes docs/audit/docs-rewrite/STALE.md only with --write-stale; hooks and CI never touch it (#2194).

Traps

Symptom	Cause	Fix
Segfault / cudaFree on garbage / MoEExecutorTest 5/5 red after a header or class-layout edit	stale .o despite ninja	grep -rl exec/executor.h src tools tests | xargs touch && make dev (healed 3/3)
Link error __cudaRegisterLinkedBinary after make dev	same class	touch includers, rebuild
Gate result contradicts the tree	make build COPYs the tree at start (#1531); verify-fast never builds (SKIP build line)	rebuild after every edit or amend, then gate
Bench numbers move while a chain runs	make dev rebuilds build-dev/ under the running binary	bench from copies (-v dir:/bin_arm)
NvFP4SmallMTest.BandwidthAboveStarvationFloor red in the full suite	contention in the run (401 vs 537.6 GB/s threshold; isolated 590-622)	rerun isolated
NvFP4SmallMV2Test.BandwidthAboveStarvationFloor red in the pre-commit suite	same class: 39.68 % vs the 40 % floor, isolated 73-78 % (3/3, 2026-09-22)	rerun isolated (docker run --rm --gpus all imp:test test-quant --gtest_filter=...); only with IMP_HOOK_GPU=1 does the hook run it
ci_static_gates.sh in imp:toolchain: FAIL registers + local frame vs the pin, docker: command not found	the kernels group shells out to docker	run make kernel-resources on the host (reads build-dev/libimp.a: make dev first)
test-core fails to link in Sanitizers / make asan	sources inside the IMP_BUILD_SERVER block (#1821)	keep test-core's deps outside that block
--perplexity OOMs or pool floors on a head checkpoint	mtp_k=auto loads the MTP head (+0.79 GiB)	--set speculative.mtp_k=0 in PPL harnesses
PPL differs 0.35% between runs of the SAME binary	non-deterministic forward	--set runtime.deterministic=true on both arms (implies deterministic_gemm)
std::sort etc. "not declared"	libstdc++ 15 dropped transitive <algorithm> (#906)	add the include
nvcc arch error	target is compute_120a/sm_120a + optional compute_120f PTX (IMP_SM120_FLAGS)	never generic sm_120 (sm120-cuda-expert)
Compose/container ignores a config key	only IMP_CONFIG / IMP_SET reach every key (#1823); the 19 legacy IMP_* names in docker-entrypoint.sh are frozen	-e IMP_SET="key=v key2=v"; a bogus key fails at start (no such key)
Value looks like an unread env var	engine reads only IMP_CONFIG IMP_DETERMINISTIC IMP_FMHA_FA2 IMP_WORKER_TIMING IMP_SPEC_TRACE IMP_JUMP_TRACE IMP_PPL_DUMP; the entrypoint translates the rest	grep docker-entrypoint.sh too
git worktree remove fails	root-owned build-dev/ inside it	rm -rf via container, git worktree prune; a worktree also needs the models/ symlink dir or verify-fast fails in the hook
FAIL model directory not found: /src/models from make verify-fast in a git worktree	models/ symlink farm exists only in the main checkout (scripts/verify.sh:192)	IMP_VERIFY_MODELS=$HOME/models make verify-fast (or ln -s $HOME/models models)
A gitignored ./imp.conf in the repo root	dev builds with -w /src load it, images do not (#1784 fixed the build context)	read the imp.conf loaded from line in every log

Determinism and quality

• --set runtime.deterministic=true: temp=0 reproducibility incl. MoE routing atomics and top-k races; implies runtime.deterministic_gemm; promoted process-wide since #542; default OFF. Qwen3.6-35B stays non-deterministic at temp=0 (routing flips); never assert exact output there.
• 7 DISABLED_ tests with reasons: 2 determinism boundaries (cross-context GDN, FMHA smem), the rest benches without assertions. Do not re-enable.
• imp-cli --perplexity <file> is teacher-forced prefill (chunk-aware since #553); use tools/analysis/ppl_corpus_45k.txt (untracked, rebuild with tools/analysis/make_ppl_corpus.sh), not the 199-token ppl_corpus.txt.
• imp-cli --json: one JSON document on stdout, logs on stderr (#1715).

PR / merge

Ship flow, auto-merge race, release cut: skill shipping-prs.