Intel canary skill

Pulp ships macOS arm64 as usable and macOS x86_64 / universal as experimental. This skill owns the machinery that keeps the Intel story from silently regressing. The full design, tiering, and the honest catch/miss list live in docs/guides/intel-support.md — read it first.

The Tier-1 Rosetta lane measures logic, not speed

intel-portability.yml runs x86_64 binaries under Rosetta on an arm64 runner, at roughly a third of native speed. Anything asserting a wall-clock budget therefore reports the emulator.

That lane went red for seven consecutive runs on exactly three of 16,314 tests, all timing-bound: heritage-performance (labels performance quality-lab), a CompiledTempoMap randomized-map Timeout, and process-deadline-selftest. None was an x86_64 defect.

The cause was a drifted exclusion list: build.yml excludes performance|bench|quality-lab on PR runs, this lane excluded only validation|slow. When adding a timing-sensitive label or a deadline test, update both — and remember --repeat until-pass:2 only absorbs a single flake, so a consistently slow test fails twice and reds the lane.

A red here is advisory and blocks nothing, which is precisely why it can sit red for days while burning ~52 minutes of hosted macOS per triggering PR.

What this skill covers

• tools/scripts/intel_canary_lint.py — the Tier-0 static lint (5 classes).
• tools/scripts/intel_canary_allowlist.txt — its exemption list.
• tools/scripts/test_intel_canary_lint.py — the lint's self-test.
• The PULP_INTEL_CANARY option in the root CMakeLists.txt.
• The Tier 0-4 workflows: build.yml (canary step), intel-portability.yml (Tier 1 advisory PR lane), nightly-intel.yml (Tier 2), and the universal-crosscheck job in nightly-intel.yml (Tier 3, nightly), plus the dedicated physical Intel selector and supervisor (Tier 4, advisory).

Note: the Tier-3 universal check only validates a universal build — it publishes nothing, and the release ships THIN per-arch binaries. It lives in nightly-intel.yml's universal-crosscheck job (nightly), NOT on the release path: a redundant canary for an artifact we do not ship must never be able to block or starve a release. intel-portability.yml covers Intel at PR time.

Before Tier 3 accepts the universal build, its m153 capability step must produce one source/asset/generation-bound receipt containing both an arm64 native compile/link/run and an explicitly Rosetta-run x86_64 compile/link/run. The final Compute result step consumes that status alongside build, lipo, and dual-arch auval; an absent or partial capability receipt is a failure, not an advisory skip.

Its auval step must never be written as auval | tee /dev/stderr | grep -q PASS. Under set -o pipefail, grep -q exits on its first match and SIGPIPEs tee, failing the step even though auval printed "AU VALIDATION SUCCEEDED" — and grep for bare PASS matches a per-subtest line that prints even on an overall failure. Capture to a file, then assert on AU VALIDATION SUCCEEDED. That false failure is what got auval misdiagnosed as flaky ("Bad Max Frames") and as "unreliable on hosted VMs"; it passes fine there.

The installable Intel artifact is a separate concern — a REQUIRED darwin-x64 build+smoke leg in release-cli.yml (os: macos-15-xcompile) that ships pulp-darwin-x64.tar.gz + pulp-sdk-darwin-x64.tar.gz in every release. It is CROSS-COMPILED on the healthy Apple-Silicon runner (-DCMAKE_OSX_ARCHITECTURES=x86_64

• -DPULP_RUST_CLI_TARGET=x86_64-apple-darwin), NOT the flaky native macos-15-intel image — that native leg CPU-pegged and never shipped an artifact; its timeout cancellation (unabsorbed by continue-on-error) turned build-cli's aggregate cancelled and skipped the whole release. So BOTH "Intel is validated" (the Tier-3 gate, now advisory) and "Intel ships via release-cli" (this cross-compiled leg) now hold. macos-15-intel survives only as the Tier-2 nightly's native-silicon signal. See docs/guides/intel-support.md → "Shipped Intel artifacts".

Whether the darwin-x64 leg runs at all is now governed by active_platforms in tools/scripts/release_product_matrix.json (leg config in release_build_matrix.py). When the field lists a subset without darwin-x64, the Intel release leg is paused: no artifact ships AND no Intel release compile happens, so an Intel regression lands unnoticed for the duration. The pause is deliberately time-boxed — release-platform-subset-check.yml opens a tracking issue after 7 days of subset. When judging "why did no Intel artifact ship for vX.Y.Z", check that field for the era before suspecting the cross-compile leg.

The five lint classes (and why they are scoped the way they are)

1. Raw NEON intrinsics / arm_neon.h outside an __aarch64__/__ARM_NEON guard — core/ only.
2. An ARM-gated SIMD #if chain with no __x86_64__/__SSE* sibling branch AND no #else fallback — core/ only.
3. Hardcoded darwin-arm64 / mac-arm64 / aarch64 — tools/cmake/** + tools/scripts/fetch_*, allowlist-driven.
4. CMAKE_SYSTEM_PROCESSOR (host arch) used for an Apple target decision in a file that never consults CMAKE_OSX_ARCHITECTURES — tools/cmake/**, file-scoped.
5. Hardcoded CMAKE_OSX_ARCHITECTURES=arm64 (arm64-only) — tools/cmake/**.

The GPU-audio exact-provider helper authenticates a thin Apple-Silicon asset. Its three ARM literals are intentional only behind the target-architecture check; ordinary Intel and universal builds keep exact proof disabled. The allowlist uses narrow substrings, with a mutated unrelated ARM asset as a negative control. When changing that helper, run both test_gpu_audio_provider_identity.py and test_intel_canary_lint.py; a clean lint alone does not establish Intel configuration or native execution.

Non-obvious gotchas (learned building this)

• vst3 is NOT a NEON intrinsic. The NEON store intrinsics are vst1..vst4, so a naive \bvst[1-4] regex flags the VST3 plugin format, its vst1/vst2 variants, every #include ".../vst3_*.hpp", and the pulp::format::vst3 namespace — 60+ false positives. Class-1/2 detection therefore requires the canonical NEON lane-type suffix (_f32, _s16, _u8, _p64, …) or the vldN/vstN load/store shape. If you widen the NEON token set, keep the suffix requirement or the VST3 tree lights up red.
• The lint MUST stay clean on a healthy tree. --mode=tree returning a finding is a real portability regression to fix at the source — do NOT weaken the lint or blanket-allowlist to make it pass. The allowlist is for genuinely-arch-aware occurrences ONLY, each with a stated reason.
• Allowlist entries are path :: substring, not line numbers. Substrings survive line moves and force each exemption to name the exact construct. A coarse whole-file exemption would hide a newly-added hardcoded arm asset.
• Class 3 skips comment lines (^\s*#, ^\s*//) but Python docstrings are not # comments — the fetch_* docstrings that list darwin-arm64, darwin-x64, … are allowlisted explicitly.
• Class 4 is file-scoped by design. Detecting "used for an Apple target decision" precisely in CMake needs semantic analysis; instead it flags a CMAKE_SYSTEM_PROCESSOR read in an Apple-aware file that never mentions CMAKE_OSX_ARCHITECTURES. Correct files (they consult the target arch) are clean; a new file that forgets it is caught. This is documented as a MISSES limitation in the guide.
• macos-15-intel is real but flaky (linker-image errors, CPU-peg). It is the fallback for Tier-2 job A, which has an infra-vs-product watchdog classifier. The physical Tier-4 runner uses only pulp-intel-native,pulp-host-macmini; never give it the required Studio pool labels and never route Intel work to Namespace.
• A universal wgpu dylib must be re-signed after lipo (G3 lesson): a raw fat dylib fails codesign --verify and the arm64 slice is killed at load. The Tier-2/3 check_bundle_architectures.py --strict assertions verify BOTH lipo -archs and codesign --verify on every embedded dylib for this reason.
• A JIT workspace is not a credential boundary. The physical Tier-4 host keeps gh/ghapp and runner-group administration in the login-account controller, but executes run.sh and workflow steps as the non-admin pulp-ci account through a root-owned, fixed-operation worker shim. Never install controller credentials in the build account or make the shim/job checkout writable by it. Every cycle starts from a root-owned runner/tool golden and deletes the complete job root plus leftover account processes; cleaning _work alone permits runner persistence. Activation requires the fixed hidden uid-499 account, Xcode, immutable read-only warm cache, shim, and narrow sudoers setup documented in docs/guides/local-ci.md. The worker gives each job a private ephemeral home and temp root, then kills and removes all uid-owned residue before serving the next job. Its protected warm cache is read-only to jobs and sets CCACHE_NODEPEND=1.

Editing checklist

When a new packaging contract intentionally names thin architecture artifacts, allowlist both the architecture-selection predicate and any literal fixture slug. Keep the entries narrow and explain why each apparent arm64 assumption is portable; never blanket-exempt the containing Rack or packaging file. The control-shipping artifact scanner is one such contract: its non-Apple validation accepts both x86 and ARM spellings emitted by native inspection tools, so its paired ARM predicates are narrowly allowlisted.

When you change the lint, the allowlist, or an Intel workflow:

1. python3 tools/scripts/test_intel_canary_lint.py — self-test must pass.
2. python3 tools/scripts/intel_canary_lint.py --mode=tree — must be clean.
3. yamllint --no-warnings -d relaxed .github/workflows/ and actionlint -shellcheck= -pyflakes= .github/workflows/<file> for any YAML.
4. If you add/remove a lint class or change scope, update docs/guides/intel-support.md (catch/miss list) and this skill.
5. tools/scripts/skill_path_map.json maps the lint + allowlist + self-test to this skill — a diff touching them without updating this SKILL.md is rejected by skill_sync_check.py.

Never let a flaky advisory leg decide a run's conclusion

nightly-intel concluded cancelled on every scheduled run for a long time, which reads as "this workflow produces no Intel coverage". That reading is wrong, and the trap is worth internalising:

Universal + lipo + dual-arch auval (macos-15) : SUCCESS    <- every night
Native Intel build + test (macos-15-intel)    : cancelled  <- its 120m job TIMEOUT
Intel nightly watchdog                        : SUCCESS

universal-crosscheck — the arm64+Rosetta lipo + dual-arch auval signal that release-cli.yml relies on after the per-tag universal gate was removed — succeeds every night. The Intel signal was there the whole time, buried under a run-level conclusion poisoned by a different leg.

native-intel on macos-15-intel had never once completed: that image CPU-pegs, so the job hit its 120-minute limit every run. GitHub reports a job timeout as cancelled, and a cancelled job cancels the whole RUN. So a leg producing zero signal was deciding the conclusion of the leg producing the real one — while burning two hours of a scarce Intel runner nightly.

Fix pattern: bound the work in the STEP, not with the job timeout.

if python3 tools/ci/run_with_timeout.py 4500 \
    cmake --build "$BUILD_DIR" -- -k 0 2>&1 | tee build.log; then
  echo "status=pass" >> "$GITHUB_OUTPUT"
elif [ "${PIPESTATUS[0]}" = "124" ]; then     # WE killed it, not GitHub
  echo "::warning::runner pegged — INFRA, not a product failure"
  echo "status=infra-timeout" >> "$GITHUB_OUTPUT"
fi

Use the repository helper rather than GNU timeout: macOS does not provide the latter, and the helper terminates the whole compiler process group before returning the same exit-124 contract.

The job then finishes normally with a loud, explicit infra-skip, and the run can reach a conclusive success/failure. Do not reach for job-level continue-on-error here: GitHub documents it for a job that fails, and a timeout is reported as cancelled — whether it covers that is a semantics gamble. Bounding the step is correct by construction.

A silent cancel is indistinguishable from "this workflow does nothing" — which is exactly how working coverage got written off as absent.

The Intel lanes run a broad ctest, so they need the history hydration step

intel-portability.yml and nightly-intel.yml both run a full ctest pass, which selects the GPU provenance selftests. Those read real per-path Git history, and actions/checkout gives the lane a shallow clone, so the suite fails on the shape of the checkout rather than on any Intel portability defect. Both lanes therefore carry the hydration step immediately after checkout:

      - name: Hydrate bounded GPU provenance commits
        shell: bash
        run: python3 tools/scripts/hydrate_gpu_provenance_commits.py

Adding another Intel leg that runs a broad ctest means adding this step too. tools/scripts/test_gpu_provenance_ci_wiring.py is the cover that fails when a lane is missing it, so the omission surfaces locally rather than as a red Intel lane whose message points at neither Intel nor portability. It runs in workflow-lint.yml, not as a ctest: it parses the workflow YAML, and the required macOS CTest hosts carry no PyYAML.