Ship Skill — Signing, Packaging, and Distribution

Overview

The pulp ship command handles the full distribution pipeline: code signing, Apple notarization, platform-specific packaging, update feed generation, and Android APK/AAB builds.

For the end-to-end release pipeline that turns a merged PR into a published GitHub Release (auto-release.yml → sign-and-release.yml → release-cli.yml, the 11 published assets, and the failure-mode triage by which asset is missing), see docs/guides/release-pipeline.md. Cross-reference comments at the top of release-cli.yml and sign-and-release.yml point back to that doc — keep them in sync when either workflow changes ownership of appcast generation, draft creation, or the matrix-tarball upload.

There is no Intel/universal gate on the release path. A universal-arch-gate job used to build PulpGain universal and run dual-arch auval on every tag. It was removed: it is redundant with nightly-intel.yml's universal-crosscheck (same check) and intel-portability.yml (Intel at PR time), and — worse — it pinned itself to the GitHub-hosted macos-15 pool for ~2h per tag, which is the SAME scarce pool the release's own required darwin-x64 legs need. At ~14 tags/day it queued ~28 macOS-hours/day of hosted work ahead of the leg that actually gates publication, while the self-hosted Studios sat idle. Never put an advisory check in front of the release on a pool the release itself competes for. (Universal-build gotcha worth keeping: a raw lipo'd wgpu dylib fails codesign --verify and the arm64 slice is SIGKILL'd at load — always re-sign after lipo.) See docs/guides/intel-support.md.

Which platforms a release ships is active_platforms in release_product_matrix.json — one field consumed by the build/smoke matrix (release_build_matrix.py), the publish-time content verification, and the --exact-required asset list, so the built legs and the demanded assets cannot desync. Absent = full inventory; a subset (e.g. ["darwin-arm64"]) pauses the other platforms' legs AND drops their archives from the asset contract in one edit. The publish step reads it from the default branch, so the flip governs re-dispatches and backfills too. Paused platforms are time-boxed by release-platform-subset-check.yml (7 days → tracking issue). The per-release asset table in docs/guides/release-pipeline.md describes the full inventory; a subset release legitimately carries fewer rows, and installers for paused platforms serve users from the last release that carried them. The narrowing is permanent per release: a published GitHub release is immutable, so every tag published while the field is narrow ships without the paused platforms' assets forever — widening the field later cannot repair them; only a new tag can. The 7-day check catches a subset that outlives its purpose, NOT a release that shipped incomplete inside the window — that one is already immutable by the time the issue opens.

Intel-Mac release slice — CROSS-COMPILED on Apple Silicon, required. The darwin-x64 build+smoke rows (os: macos-15-xcompile) cross-compile the x86_64 CLI+SDK on the healthy arm64 runner via -DCMAKE_OSX_ARCHITECTURES=x86_64 (C++) + -DPULP_RUST_CLI_TARGET=x86_64-apple-darwin (Rust CLI). They prefer the per-leg override, then PULP_RELEASE_MACOS_RUNS_ON_JSON (the dedicated pulp-build-vm-release Tart pool), then the legacy PULP_INTEL_RELEASE_MACOS_RUNS_ON_JSON, and finally hosted macos-15. The native GitHub-hosted macos-15-intel image is deliberately avoided: it CPU-pegs on a full CLI+SDK build (observed: 71-min build cancelled at a 75-min cap, every run) and its timeout cancellation (not a clean failure) makes build-cli's aggregate cancelled and skips release — the earlier native leg never shipped an artifact for this reason. The native Mac Mini remains a separate advisory/nightly portability canary. The pair is REQUIRED (release-publish.yml lists it unconditionally). Two leg-specific gotchas: (a) the arm64 runner's bootstrap prefetches arm64 Skia, so the leg rm -rf external/skia-build/build before the x86_64 Skia fetch and asserts lipo -archs libskia.a == x86_64; (b) rustup target add x86_64-apple-darwin is required (the toolchain pins the channel but no targets). Load-bearing CI gotchas that caused a multi-hour release stall:

• continue-on-error on a matrix leg masks a clean failure (leg finishes non-zero) but NOT a cancellation (timeout / stuck-queued / run-cancel). A cancelled advisory leg still turns the aggregate cancelled. If you ever reintroduce an advisory leg, wrap its long steps in a shell timeout so they exit non-zero (clean fail) before the job timeout-minutes cancels them.
• The release job's if: uses always() && needs.build-cli.result == 'success' && needs.smoke-cli.result == 'success' && needs.universal-arch-gate.result == 'success' (NOT the implicit success() over all needs). always() forces evaluation even if a needed job failed, so nothing silently skips publish; the explicit == 'success' checks are what enforce the requirement. All three are now required (the universal-arch-gate was re-required once its shell-bug false failure was fixed). A genuine build/smoke/gate failure blocks publish.
• To recover a stuck release when the tag already exists but no run published: the successful matrix legs' CLI+SDK artifacts persist on the (even cancelled) run — gh run download <run> -R Generous-Corp/pulp, then gh release create <tag> --latest --notes-file <composed> with compose_release_notes.py for the body. On workflow_dispatch the pipeline itself publishes directly (draft only on tag-push), so a hand-published backfill matches its semantics.
• sign-and-release.yml and release-cli.yml both fire on the SAME v* tag and race. sign-and-release notarizes, then its "Attach appcast.xml" step POLLS gh release view <tag> for the draft that release-cli's release job creates at the very end of its build chain. Since the Intel darwin-x64 cross-compile leg landed, that chain runs 60-90 min (the leg queues for a hosted macos-15 runner, then builds), so a too-short poll times out and the release ships without the Sparkle appcast → someone has to hand-publish. The poll window is bumped to 100 min (seq 1 300, 20s each). The deeper fix (not yet done — needs a real-tag validation): the poll runs ON the macOS notarize VM, holding a scarce self-hosted release VM for the whole wait, which also starves the lane; emit appcast.xml as an artifact and move the wait+attach to a cheap ubuntu job so the macOS VM frees right after notarization.

Release builds carry an explicit --parallel count — do not "simplify" it away. release-cli.yml's CLI and SDK build steps (and release-dry-run.yml's mirror) compute jobs from the runner's visible cores and pass cmake --build … --parallel "$jobs". Without it the default Unix Makefiles generator runs make with no -j — a strictly serial compile that cost ~50 min of every ~55 min release leg for months while build_parallelism_guard.py stayed green (it polices unbounded/whole-machine counts, not the serial opposite). Full visible cores is deliberate on this lane: release runners are ephemeral and single-tenant (GitHub-hosted images or dedicated release Tart VMs), so the shared-host governed-share rule does not apply; capacity is controlled by how many release VMs a host admits, not by throttling each build. test_release_workflow_test_step.py::ReleaseBuildParallelismExplicit enforces the shape; the steps also echo cores/RAM so an OOM or a small-VM allocation is diagnosable from the run log alone.

Pre-flight: plugin ↔ CLI skew check

Before running pulp ship ..., source the shared skew-check helper so a user on an outdated CLI sees a one-line hint (stderr, once per session) when the installed CLI is older than the plugin's declared min_cli_version:

source "$(git rev-parse --show-toplevel)/tools/scripts/cli_version_check.sh"
pulp_cli_version_check

Advisory only — never blocks. Full contract + override knobs live in the upgrade skill.

Subcommands

Command	Platform	What It Does
sign	macOS, Windows, Android	Sign plugin bundles or APK/AAB; --path <file> signs one explicit desktop artifact (macOS .app/.dmg/bundle, Windows .exe/bundle; not .pkg)
notarize	macOS only	Submit to Apple notarization, poll, staple; --path <file> notarizes one explicit .dmg/.pkg/.zip (repeatable), not a raw .app
package	All	Create .pkg/.dmg (macOS), NSIS (Windows), APK+AAB (Android)
release	macOS only	One command: sign → package → notarize + staple the .pkg/.dmg it builds → verify
share	macOS only	One-off: sign → wrap .app in DMG → notarize → staple → Gatekeeper-verify a single artifact for sharing
auv3-xcodeproj	macOS only	Generate an Xcode project for an AUv3 target
appcast	All	Generate Sparkle-compatible XML update feed
check	All	Verify signing status of built artifacts
doctor	macOS	Make signing+notarization non-interactive: self-heal the dedicated signing keychain and validate the .p8 notary key. No build dir required.
swap-pack	All (keychain macOS-only)	Sign a hot-reload UX bundle into a swap pack. No build dir required. Capabilities are inferred from the bundle's JS (--autocaps, no hand-maintained list); the Ed25519 signing key is reused from the macOS keychain (pulp.reload.signing.<plugin_id>, created + stored + screamed on first use, never silently regenerated, corrupt entry refused) or a --sign-key <file>. --backup-github [--repo owner/name] publishes the key as a GitHub secret in the plugin repo (refuses core Pulp). The key blob is piped to security/gh via a 0600 temp, never on argv. Thin assembly over the header-only reload building blocks (pack_build/pack_signing_ui/swap_pack serialize/key_store); the signing key is a distinct concern from the Developer-ID/notary identity the other subcommands use. See docs/guides/reload-trust.md.

Non-interactive signing (no keychain / 1Password prompt)

On a workstation, codesign and notarytool can pop a macOS keychain "allow access" dialog or a 1Password prompt — which silently wedges any headless, SSH, or CI sign. Two root causes, two durable fixes, both codified in pulp ship doctor (script: tools/scripts/ensure_signing_ready.sh):

1. Signing key in the login keychain → codesign/1Password prompt. Fix: sign from a dedicated keychain whose key is authorized for codesign via security set-key-partition-list. The doctor creates/unlocks it, imports the .p12, runs the partition-list step, and adds it to the search list — all idempotent.
2. notarytool driven by a keychain profile re-prompts when the login keychain locks (and the profile periodically vanishes on churny hosts). Fix: notarize from a file-based App Store Connect .p8 API key — no keychain at all. The doctor validates it; --check-online also re-mints the optional pulp-notary convenience profile from the same .p8.

pulp ship doctor                 # heal + timestamped signing probe; exit 0 = prompt-safe
pulp ship doctor --check-online  # also prove the .p8 against Apple (read-only) + refresh profile
pulp ship doctor --print-env     # emit resolved identity/keychain/keypath handles (no secrets) for eval

pulp ship sign and the combined-installer path run this doctor as a mandatory, quiet, fail-closed preflight so the hardened path is automatic. Readiness requires a dedicated keychain, the full apple-tool:,apple:,codesign: partition list, an unambiguous identity hash, and a real timestamped hardened-runtime signing probe. Any failure stops before production codesign; never retry through the login keychain and never ask the user for a password. The doctor itself NEVER prints secret values.

Installer signing is a SEPARATE key and a SEPARATE tool — probe it, never infer it from codesign. Bundle signing uses the Developer ID Application key through codesign; the .pkg uses the Developer ID Installer key through a different binary. A green codesign probe says nothing about the second one, so the doctor probes it independently (pkgbuild → productsign → pkgutil --check-signature) whenever PULP_SIGN_INSTALLER_HASH is configured, and refuses READY when it fails.

The concrete trap: the dedicated keychain authorizes the installer key for codesign/security/productsign only, so productbuild --sign is denied. Headless, the suppressed authorization dialog surfaces as CSSMERR_CSP_USER_CANCELED (-128) and Error signing data. — with no mention of a keychain, and only after every bundle has already been signed. Read that error as "this tool is not on the key's ACL", not as a broken certificate. build_combined_installer.sh therefore builds the product archive UNSIGNED and signs it with productsign, which is authorized; the two produce an equivalent archive. Keep it that way — reintroducing productbuild --sign re-breaks every headless package build (guarded by test_build_combined_installer.py::test_the_product_archive_is_signed_by_productsign_not_productbuild). The combined-installer recipe must pass --keychain "$PULP_SIGN_KEYCHAIN" to productsign explicitly. Search-list order is not a sufficient binding in every SSH or GUI session: bare productsign can fall through to a locked login keychain even after the doctor probe succeeds. The package test asserts this explicit keychain argument so a future recipe change cannot silently reopen that prompt-capable path. An ACL is baked in at security import time, so widening the doctor's -T list only helps keychains created after the change; an existing keychain keeps its old ACL.

The combined installer also supports repeatable --plugin-scripts <au|vst3|clap> <bundle name> <scripts directory> entries. Use this for format-specific postinstall work, such as refreshing the macOS Audio Unit registrar after replacing an AU. Match the bundle name exactly and attach the hook only to the format that needs it; attaching an AU refresh to an optional standalone app leaves plugin-only installs stale. Hooks must be executable, must resolve the active console user safely, and must treat a missing cache or already-stopped registrar as success.

Control-shipping sidecars are relocated for you. CMake emits *.inspector-capabilities.json / *.control-shipping*.json beside the target, which for a bundle target lands in Contents/MacOS — where Apple permits code only, so codesign fails with code object is not signed at all / In subcomponent: ….json. deep_sign() moves them to Contents/Resources/pulp-control-shipping-evidence before signing, preserving the receipts. A consumer package.sh that deletes them first is both redundant and lossy: it destroys build provenance the installer would have sealed into the bundle.

deep_sign() also walks Contents/Helpers before sealing a bundle. Products may carry nested helper apps there, including their own executables and *.dylib files. Those files must receive the same Developer ID identity, hardened runtime, and secure timestamp as the containing product; signing only Contents/Resources, Contents/MacOS, or Contents/Frameworks produces a package that builds locally but is rejected by Apple notarization.

Secrets live OUTSIDE the repo (never committed), in ~/.config/pulp/secrets/ (override dir with $PULP_SECRETS_DIR):

• keychain.env — PULP_SIGN_KEYCHAIN, PULP_SIGN_KEYCHAIN_PW, PULP_SIGN_P12, PULP_SIGN_P12_PW, PULP_SIGN_IDENTITY_HASH (+ optional …_INSTALLER_HASH)
• notary.env — PULP_NOTARY_KEY_PATH (.p8), PULP_NOTARY_KEY_ID, PULP_NOTARY_ISSUER_ID (the same trio notary_env.cpp resolves for notarize)

Each value may also come from the same-named environment variable; env wins over the file. If no dedicated keychain is configured, the doctor reports any login-keychain identity only as diagnostic evidence and exits nonzero. It never classifies a prompt-capable fallback as ready.

If the configured legacy dedicated keychain exists but its recorded password no longer unlocks it, the doctor preserves that file and creates/reuses a stable *-unattended.keychain-db sibling from the local P12. It then applies the same partition, search-list, identity-hash, and real-probe gates to the sibling. Do not delete or overwrite the legacy keychain and do not substitute login.

Configuration

Settings are resolved in order: CLI flag > environment variable > ~/.pulp/config.toml

Setup

pulp config init                    # Create config from template
pulp config set signing.apple.identity "Developer ID Application: Name (TEAMID)"
pulp config set signing.apple.team_id "ABCDE12345"
pulp config set signing.apple.apple_id "you@example.com"
pulp config set signing.android.keystore "~/keystores/release.jks"

Config file location

~/.pulp/config.toml (override with $PULP_HOME)

See config.example.toml in the repo root for all options with documentation.

Interactive safety contract

Before executing any signing, notarization, or packaging action, the /ship command uses AskUserQuestion to:

1. Show resolved config values (identity, keystore, credentials) and their source (CLI/env/config.toml)
2. Let the user review, edit, or cancel before proceeding
3. Offer to save new values with pulp config set

When invoked via skill trigger (not slash command), apply the same pattern: always show what will happen and confirm before executing.

PULP_TRACING ship guard

sign and package refuse an artifact built with PULP_TRACING=ON (the dev-only Perfetto tracing config — an ~80 MB in-memory ring + a .pftrace written inside the host). The runtime embeds a retained sentinel byte-string in any ON build; pulp ship scans the candidate bundle/binary for it and stops with an error naming the offending file. release inherits the guard through its package step. Override deliberately with --allow-tracing (prints a loud warning, then proceeds). If a ship step fails with "refusing to ship a binary built with PULP_TRACING=ON," the fix is to rebuild with tracing off (the default pulp build), not to reach for --allow-tracing. The scanner lives in tools/cli/ship_tracing_guard.hpp (unit-tested in test/test_ship_tracing_guard.cpp).

Development-SDK ship guard

package, notarize, release, and share refuse a project configured against a development Pulp SDK — the local-only SDK pulp sdk can build from a clean committed checkout so Forge can iterate against an unreleased Pulp. sign is deliberately NOT guarded: signing a dev build for local testing is fine; handing it to anyone else is not.

tools/cmake/PulpSdkProvenance.cmake writes PULP_SDK_DISTRIBUTION_ELIGIBLE (and PULP_SDK_DEVELOPMENT) into the SDK's CMake cache via CACHE INTERNAL, and pulp::cli::sdk_allows_distribution() (tools/cli/sdk_distribution_guard.cpp) reads that cache from the build dir. A development SDK fails with a message naming the reason; released SDKs are unaffected.

There is no override flag. If a ship step stops with "the configured Pulp SDK is development-only," the fix is to reconfigure against a released SDK — the whole point of the marker is that a locally-built SDK cannot be identified later by whoever receives the artifact. Note the guard is deliberately fail-open on a missing or unrecognised cache, so that SDKs predating the marker keep working; it is a tripwire against the development profile, not an attestation that any given SDK is releasable.

Unit-tested in both directions in test/test_sdk_distribution_guard.cpp, plus the CMake-level provenance contract in test/cmake/test_sdk_provenance_contract.cmake.

Workflows

One-off share (macOS): hand a build to a friend, properly signed

When a developer just wants to give someone a working .app/.dmg/.pkg — NOT cut a versioned GitHub release — reach for share. It is the opinionated one-command path and is fully separate from the repo release pipeline.

# .app → signs, wraps in a DMG, signs the DMG, notarizes, staples, then runs
# the exact `spctl -a -t open --context context:primary-signature` Gatekeeper
# check. Green = the recipient will NOT see "Unnotarized Developer ID".
pulp ship share MyApp.app --identity "Developer ID Application: Name (TEAMID)"
pulp ship share MyApp.app --identity "..." --output dist --entitlements entitlements.plist

pulp ship share MyApp.dmg            # already a DMG → skip the wrap step
pulp ship share Installer.pkg        # pkg assumed installer-signed → notarize only
pulp ship share MyApp.app --dry-run  # print the plan; sign/notarize nothing

Why this exists: a Developer-ID-signed-but-unnotarized DMG is rejected by Gatekeeper (source=Unnotarized Developer ID). The cert is fine; the gap is notarization. share closes that gap in one step. Credentials resolve through the same chain as pulp ship notarize (App Store Connect API key preferred); without creds, the notarize step fails loudly rather than shipping unnotarized. For .app inputs, --output <dir> chooses where the generated DMG lands and --entitlements <plist> overrides the default app-signing entitlements.

Signed + notarized is NOT the same as portable. A perfectly signed, notarized .app still degrades on another machine if it reads an asset (SVG / JSON / image) from an absolute build-tree path at runtime — that path exists only on the build box. TRIAZ hit this (2026-06-18): create_view() did std::ifstream(TRIAZ_FRAME_SVG) where the macro was "${CMAKE_CURRENT_SOURCE_DIR}/import/frames/mixer.frame.svg"; the shared .app found nothing and fell back to the generic auto-Parameters panel on every Mac but the build box. It built, signed, notarized, and ran fine locally — nothing flagged it. Two defenses now exist:

• pulp_assert_portable_bundle (PulpPortable.cmake) runs automatically on every standalone .app build and WARNs (or fails, with -DPULP_STRICT_PORTABLE=ON) if the binary bakes the source/build dir as a string. Set PULP_STRICT_PORTABLE=ON for release/ship builds so a non-portable binary can't ship.
• The fix for a finding: EMBED the asset — pulp_embed_files(target FILES …) → pulp::EmbeddedAsset::get("name"), or pulp_add_binary_data(...) — so it's compiled in. Never read an absolute build-tree path at runtime. (The faithful design-import codegen already embeds; only hand-rolled create_view asset loads bypass it.)

The composable primitives underneath:

• pulp ship sign --path <artifact> — sign exactly one desktop artifact (.pkg installers are signed by package).
• pulp ship notarize --path <dmg|pkg|zip> — notarize + staple one artifact (repeatable). Use share for a raw .app.

Do not confuse with the production release. share/release/sign/ notarize are local developer commands. The versioned GitHub Release (all example plugins, appcast, downloads) runs through CI (.github/workflows/sign-and-release.yml), which does its own codesign + pkgbuild/productbuild + xcrun notarytool submit + stapler staple and never calls these CLI subcommands. Changing the CLI ship subcommands cannot affect a regular release.

Combined installer is component-SELECTABLE by default

create_combined_pkg (ship/platform/mac/codesign_mac.mm) builds one component .pkg per format and combines them with productbuild --distribution, emitting a distribution document with one user-toggleable <choice> per InstallComponent (all start_selected, customize="allow"). So a multi-format installer always offers a Customize pane to install only AU / VST3 / CLAP as desired — do NOT drop back to a flat productbuild --package ... archive (that installs everything with no choice). Set InstallComponent::title for the choice label, or leave it empty to derive from the install location ("Components" → "Audio Unit (AU)", etc.). Verified by test_codesign.cpp ("component-selectable with a choice per format").

CLI default (macOS). pulp ship package DEFAULTS to this single component-selectable .pkg — it collects every built bundle (Standalone → /Applications, AU/VST3/CLAP → their system plug-in folders) into one create_combined_pkg call so the user is never forced to install every format. --separate restores the legacy behavior (one .pkg per format); --dmg produces disk images instead. The routing lives in the pulp ship package branch of tools/cli/cmd_ship.cpp.

macOS 27 pkgbuild --analyze omits BundleIsRelocatable. build_combined_installer.sh pins every app component non-relocatable, so it must Add :N:BundleIsRelocatable bool false when Set finds no entry — a bare Set aborts with Entry, ":0:BundleIsRelocatable", Does Not Exist (covered by test_build_combined_installer.py).

A packaging run that looks dead is almost always alive

build_combined_installer.sh spends 5 to 30 minutes inside xcrun notarytool submit --wait on every notarized release, and for that whole window the two instruments anyone reaches for both report "dead":

• The log goes dark. notarytool writes its entire poll phase as ONE unterminated line, flushed only when Apple returns a verdict, and stdio switches to full buffering the moment stdout is a file rather than a TTY. A watcher tailing the log sees nothing between Submission ID received and the verdict. Six status polls across half an hour can arrive as a single line.
• The process name disappears. The wrapper scripts (examples/*/package.sh) end by execing into the recipe, which replaces the process image. pgrep -f package.sh returns 0 for a live run, same PID, from a few seconds in. stdbuf / script do not help either instrument: the poll phase has no newlines to flush.

So the recipe emits its own heartbeat. Every PULP_NOTARIZE_HEARTBEAT_SECS (default 30) it prints

[heartbeat] notarization in progress (120s elapsed) — not hung

to the same stream the log captures. If you see that line, the run is alive — do nothing. It is implemented in tools/scripts/lib/heartbeat_wait.sh and covered by NotarizationHeartbeatTest in tools/scripts/test_build_combined_installer.py, which drives the real recipe with a stand-in xcrun.

If the heartbeat is absent, ask the PID, never the name:

ps -o pid=,etime=,command= -p "$PID"     # the wrapper exec'd; the name is gone
xcrun notarytool history --key "$PULP_NOTARY_KEY_PATH" \
  --key-id "$PULP_NOTARY_KEY_ID" --issuer "$PULP_NOTARY_ISSUER_ID" | head -20

Asking Apple costs nothing and touches no artifact. Do not staple a .pkg another process may still own. Two release packages once had their final bytes written by an ad-hoc recovery script seconds after the provenanced pipeline had already finished and validated them — the provenance guards are all on the recipe's inputs, and there is no custody on its output. Before concluding a recovery is needed, compare the .pkg's mtime against the recipe's own completion: a write after OK → came from something else.

The heartbeat deliberately does not change what the recipe guarantees. The wrapped command runs in the foreground and its exit status is returned unchanged, so under set -euo pipefail a failed notarization still aborts before stapler staple — an un-notarized .pkg can never reach the staple / validate / OK → path. That is asserted in both directions by test_a_failed_notarization_never_reaches_stapler_staple and its success-path counterpart; breaking the status propagation makes the failure test go red with an OK → line, which is exactly the incident shape.

Multi-product installers: group by product, ship the uninstaller

build_combined_installer.sh gained four things a multi-product installer needs. Both Forge installers use them.

Group by product, not by build target. Without --product-title, an installer carrying three products lists each one TWICE under two naming schemes: an expandable format group named from the bundle (PulpDesignSynth) and, separately at top level, its standalone app named from the title passed in (Kelvin (instrument)). Six rows for three products, with nothing on screen saying they are the same thing.

--product-title PulpDesignSynth "Kelvin — instrument" \
--app-for       PulpDesignSynth "Standalone app" "$B/…/PulpDesignSynth.app"

--app-for BUNDLE TITLE PATH nests the app inside that product's group instead of at top level, and marks the choice enabled="false" selected="true". Forced on deliberately: the standalone carries the uninstaller, so a user who deselects it installs plugins they cannot later remove. The row stays visible rather than hidden, which is honest about what is being installed.

The uninstaller is manifest-driven. --uninstaller-in BUNDLE ships pulp_uninstall.sh inside that app next to a manifest generated for the release. Two rules it exists to enforce:

• It removes what the MANIFEST says, never a glob. Globbing for likely-looking names under /Library/Audio/Plug-Ins deletes another vendor's plugin the day two products share a word.
• It ALSO removes the same bundle NAMES from the other standard plugin folder. The installer writes to /Library; a development build writes to ~/Library; every host scans both. Removing only the manifest's exact paths leaves a plugin the host keeps loading, so "uninstalled" would not mean gone. Names, never wildcards: Forge FX.component matches itself and not Forge FX Pro.component.

The manifest is computed BEFORE anything is signed, because a bundle modified after signing has a broken signature. Its receipt ids are derived independently of the loops that build the components and are then VERIFIED against REFS — the build fails if an id rule drifts, rather than leaving the uninstaller quietly unable to forget a receipt.

Panes. --welcome / --license / --readme / --conclusion. productbuild resolves these by BASENAME inside --resources, so each file is staged there under its own name. Passing a path in the XML shows a blank pane with no error.

Covered by test_build_combined_installer.py and test_pulp_uninstall.py (the pulp-uninstaller-contract ctest), which tests both failure directions: removing too little and removing too much.

VST3 bundles: moduleinfo.json must be in Contents/Resources/

A loose non-code file directly under Contents/ makes codesign treat it as an unsigned nested code object and refuse the whole bundle:

code object is not signed at all
In subcomponent: …/Contents/moduleinfo.json

An unsignable bundle cannot be notarized, so it cannot ship. Nothing before packaging notices, because the plugin builds, loads and validates happily unsigned — and core/host/src/scanner.cpp reads the file from Contents/Resources/, so a misplaced one is invisible to Pulp's own FUID lookup too. Guarded by the vst3-bundle-layout ctest (tools/cmake/scripts/check_vst3_bundle_layout.py).

The same Apple rule applies inside Contents/MacOS/: only code belongs there. Pulp's control-shipping JSON is intentionally emitted beside an unsigned build target so the scanner can bind its proof to that exact binary. Before Developer ID signing, build_combined_installer.sh preserves those manifests and reports under Contents/Resources/pulp-control-shipping-evidence/; leaving them beside the Mach-O makes codesign report the JSON as an unsigned subcomponent. The signer also resolves CFBundleExecutable from Info.plist for every bundle kind. Do not derive the main executable by stripping only .app: doing so double-signs the main binary in .component, .vst3, and .clap bundles and can trigger the same misleading subcomponent failure.

Every bundle carries pulp-build-info.json — read it before guessing

pulp_add_plugin() writes Contents/Resources/pulp-build-info.json (pulp.build-info.v1) into every bundle it makes, POST_BUILD, so the record is sealed by pulp ship sign like any other resource. It names the product version and source commit, build type/archs/min-OS, the SDK version, and embeds the SDK's share/pulp/runtime-pins.json (Skia release + commit + asset digest, Dawn commit, wgpu-native version, JS engine, min-OS floors). For "what is this user running?" read that file from the installed bundle instead of inferring from Info.plist (product version only) or strings over the binary. Full schema: docs/guides/shipping.md#build-identity-in-every-bundle.

Watch out for:

• Release builds should pass the commit explicitly — -DPULP_PRODUCT_GIT_SHA=<sha> -DPULP_PRODUCT_GIT_DIRTY=FALSE (or SOURCE_GIT_SHA on pulp_add_plugin). The default reads git at build time, which records "unknown" from a source archive and the wrong commit if a bundle did not relink after the last commit.
• Never edit the file after signing — it is a sealed resource; changing it invalidates the signature exactly like editing Info.plist.
• runtime_pins.skia.asset_in_manifest: false or null is a real finding: the build linked a Skia other than the manifest-pinned archive (typically an exported SKIA_DIR pointing at a stale checkout). dawn.commit is decoded from the linked headers, so it disagrees with the pin in that case too.
• An SDK older than the record yields runtime_pins: null; the rest of the file is still produced by the consumer's own CMake helpers.

macOS one-command pipeline: pulp ship release

pulp ship release --dmg --identity "Developer ID Application: ..."   # standalone app
pulp ship release --pkg --identity "..."                            # plugin installers

Runs sign → package → notarize → staple as one command. Unlike calling the stages by hand, the notarize stage targets the distributable .pkg/.dmg that packaging produced (selected by build time), so release --dmg leaves a Gatekeeper-ready disk image in artifacts/, not a signed-but-unnotarized one. --skip-sign / --skip-package / --skip-notarize gate stages for CI dry-runs.

Signing identities matter per artifact. A .pkg must be signed with a Developer ID Installer identity (distinct from the Developer ID Application identity that signs bundles/apps/dmgs) or notarization rejects it. Pass it via --installer-identity "Developer ID Installer: …" (or signing.apple.installer_identity in config); release also signs standalone .apps and the produced .dmg. As a safety net, the notarize stage verifies each artifact's signature and skips (does not submit) any unsigned .pkg/ .dmg — so a missing installer identity yields a clear "skipping unsigned artifact" warning instead of a failed notarytool submission. notarize --path likewise rejects a raw .app (notarytool needs a .dmg/.pkg/.zip container — use share for an app).

AUv3 Xcode handoff: pulp ship auv3-xcodeproj

pulp ship auv3-xcodeproj MyPlugin --sdk iphonesimulator --dry-run
pulp ship auv3-xcodeproj MyPlugin --sdk iphoneos --open

Generates a separate CMake Xcode build directory for a project containing an AUv3 target. --sdk accepts iphonesimulator, iphoneos, or macosx; the default output is build/xcode/<target>-<sdk>. The generated build hint targets <target>_AUv3. Use --dry-run to print the CMake invocation and build hint without requiring Xcode. For macOS AUv3 work, the generated project also contains the runnable containing-app target <target>_AUv3Host.

macOS: Build → Sign → Package → Notarize → Appcast

pulp build                                              # Must build first
pulp ship sign                                          # Uses identity from config
pulp ship package --version 1.0.0                       # Creates .pkg in artifacts/
pulp ship notarize --path artifacts/MyPlugin-1.0.0.pkg  # Submit the packaged artifact
pulp ship appcast --url https://example.com/Plugin.pkg --version 1.0.0
pulp ship appcast --url artifacts/Plugin.pkg --download-url https://example.com/Plugin.pkg --sign-key-file ~/.config/pulp/secrets/sparkle/plugin_ed25519

Notarization credentials (pulp ship notarize)

FIRST, on a set-up dev machine: check ~/.config/pulp/secrets/. It holds notary.env (the ASC API-key trio), the AuthKey_*.p8, keychain.env, and pulp-signing.p12 — everything needed to sign AND notarize. Source it and go: set -a; . ~/.config/pulp/secrets/notary.env; set +a → xcrun notarytool submit … --wait → xcrun stapler staple. Do NOT conclude "no credentials" without looking there — these are machine-local and absent from fresh checkouts, but present on a configured machine. (Mirrored in the CLAUDE.md "Local macOS signing & notarization credentials" note.)

Two lanes, resolved in this precedence (CLI > env > file > config.toml):

1. App Store Connect API key — preferred. xcrun notarytool submit --key <p8> --key-id <id> --issuer <uuid>. Maps to rcodesign --api-key-path on Linux too.

   # One-shot CLI form
   pulp ship notarize --path artifacts/MyPlugin-1.0.0.pkg \
                      --api-key ~/.config/pulp/secrets/AuthKey_XXX.p8 \
                      --api-key-id XXX --api-issuer <uuid>
   
   # Persisted form (recommended) — store once, reuse forever:
   # ~/.config/pulp/secrets/notary.env  (chmod 600)
   #   PULP_NOTARY_KEY_PATH="$HOME/.config/pulp/secrets/AuthKey_XXX.p8"
   #   PULP_NOTARY_KEY_ID="XXX"
   #   PULP_NOTARY_ISSUER_ID="<issuer-uuid>"
   pulp ship notarize --path artifacts/MyPlugin-1.0.0.pkg
   pulp ship notarize --path artifacts/MyPlugin-1.0.0.pkg --dry-run
   

   For the full reusable dev-signing stub (schema template + sourceable helper + per-plug-in setup), see docs/guides/ios-dev-signing.md.

2. Legacy Apple-ID + app-specific password — kept working as a fallback.

   pulp ship notarize --path artifacts/MyPlugin-1.0.0.pkg \
                      --apple-id you@example.com --team-id ABCDE12345
   # password defaults to @keychain:AC_PASSWORD — store via
   #   security add-generic-password -s AC_PASSWORD -a you@example.com -w
   

Override the env-file path with PULP_NOTARY_ENV=/some/path or pulp ship notarize --env-file /some/path (handy in CI sandboxes that don't share $HOME). --dry-run is the safest way to verify credential resolution — it prints which surface each value came from ((from cli) / (from env) / (from file)) and never contacts Apple.

When changing ship/include/pulp/ship/codesign.hpp, keep every platform implementation in lockstep. Linux intentionally returns no-op / false / std::nullopt for signing and notarization entry points, but it still must define every public overload, including App Store Connect notarize_submit_asc(...); otherwise Linux-only package tests are the first place the API parity break appears. Keep this covered in test/test_linux_packaging.cpp alongside the macOS parser tests in test/test_codesign.cpp.

macOS manual sign + notarize from a worktree (no pulp CLI built)

Feature worktrees usually build only the example targets (build-gpu/...), not the pulp CLI — so pulp ship sign/notarize isn't available. Sign and notarize with the raw Apple tools. This is the exact flow pulp ship automates; reach for it only when the CLI binary is absent.

One-time keychain authorization (THE blocker). A fresh login keychain does not let codesign use the Developer ID private key non-interactively, even when the key is present and the keychain is unlocked. codesign fails with errSecInternalComponent (and a half-written *.cstemp is left behind). The fix is to authorize the partition list once — it needs the login password, so the user runs it (suggest the ! prefix, or their own Terminal):

security set-key-partition-list -S apple-tool:,apple:,codesign: -s \
  -k "<login-password>" ~/Library/Keychains/login.keychain-db

Symptoms map cleanly: errSecInternalComponent → partition list not set; User interaction is not allowed → keychain locked (security unlock-keychain); no identity found → wrong/absent cert (security find-identity -v -p codesigning).

Signing over SSH (e.g. an agent on a remote Mac like macstudio). An SSH session is NOT the GUI Aqua session, so the login keychain is locked there even when it's unlocked on the console — security show-keychain-info login.keychain-db prints User interaction is not allowed, and codesign fails with errSecInternalComponent regardless of the partition list. The GitHub self-hosted runners sign fine only because they run inside the logged-in GUI session. For SSH signing you must unlock the keychain in-session first:

security unlock-keychain -p "<login-password>" ~/Library/Keychains/login.keychain-db
# then partition-list (once) + codesign as above

Notarization over SSH is unaffected — notarytool authenticates with the App Store Connect API key in ~/.config/pulp/secrets/notary.env, never the keychain. For unattended/turnkey remote signing, prefer a dedicated build keychain (import a .p12 of the Developer ID cert+key, give it its own password stored in ~/.config/pulp/secrets/, set-key-partition-list on it, and unlock-keychain it per session) so signing never depends on the interactive login password — the same pattern apple-actions/import-codesign-certs uses in CI.

Dedicated-keychain recipe that actually works when the login keychain ALSO has the cert (the macstudio case — the GUI host has the same Developer ID identity):

1. Export the identity to a .p12 once on a Mac where it works — the private-key export needs a GUI "Allow" click, so the user runs it (security export -k login.keychain-db -t identities -f pkcs12 -P <p12pw> -o key.p12); it can't be done headlessly.
2. On the remote Mac: security create-keychain -p <kcpw> pulp-signing.keychain-db; unlock-keychain; security import key.p12 -k <kc> -P <p12pw> -T /usr/bin/codesign -T /usr/bin/productbuild -T /usr/bin/pkgbuild; set-key-partition-list -S apple-tool:,apple:,codesign: -s -k <kcpw> <kc>. Store <kcpw> + the cert SHA-1 hashes in ~/.config/pulp/secrets/keychain.env.
3. Sign by SHA-1 hash, not by name. The identity now exists in BOTH the dedicated keychain and login keychain, so signing by name → ambiguous (matches ... in two keychains). Signing by the 40-char hash disambiguates.
4. The dedicated keychain MUST be in the search list for codesign to find the identity — --keychain <kc> alone does NOT restrict lookup (codesign still uses the search list and hits the locked login cert → errSecInternalComponent). Put it FRONT of the search list for the signing call, then restore: security list-keychains -d user -s <kc> <login> → sign → ... -s <login>. Safe on a CI-runner host because the Mac Studio runner only builds+tests; the signing/release lanes run on GitHub-hosted (it never signs, so a transient search-list change can't break the required macos gate).

~/.config/pulp/pulp-sign.sh wraps steps 3–4 (inner-out, by hash, restore) and falls back to login-keychain-by-name when no keychain.env exists. Deployed on both Daniel's Macs; notary + signing creds live only in ~/.config/pulp/secrets/ (chmod 600, never in the repo).

Sign inner-out. Pulp GPU bundles embed libwgpu_native.dylib in Contents/MacOS/. Sign every embedded dylib BEFORE the bundle, else you get invalid or unsupported format for signature / code has no resources but signature indicates they must be present. Remove any stale _CodeSignature first so the re-seal is clean. --deep is deprecated and misses nested Mach-Os — do the explicit inner-out walk:

ID="Developer ID Application: NAME (TEAMID)"
sign() { codesign --force --timestamp --options runtime --sign "$ID" "$@"; }
for b in build-gpu/AU/X.component build-gpu/VST3/X.vst3 build-gpu/CLAP/X.clap \
         build-gpu/examples/X/X.app; do
  rm -rf "$b/Contents/_CodeSignature"
  find "$b/Contents/MacOS" -name '*.dylib' -exec false {} + ; \
    while IFS= read -r d; do sign "$d"; done < <(find "$b/Contents/MacOS" -name '*.dylib')
  sign "$b"
  codesign --verify --deep --strict --verbose=2 "$b"   # expect "satisfies its Designated Requirement"
done

Build the signed installer:

pkgbuild --root <stage-dir> --identifier com.pulp.<name> --version X.Y.Z \
  --install-location / --sign "Developer ID Installer: NAME (TEAMID)" out.pkg
pkgutil --check-signature out.pkg   # "signed by a developer certificate ... for distribution"

Signed ≠ notarized. spctl --assess --type execute MyApp.app on a signed but un-notarized bundle prints rejected / source=Unnotarized Developer ID. That is expected and fine for the build machine; recipients on other Macs need notarization. Submit the installer (or app/dmg) with notarytool — it authenticates to Apple's cloud, so it needs App Store Connect API-key creds (the same ~/.config/pulp/secrets/notary.env trio that pulp ship notarize reads):

set -a; . ~/.config/pulp/secrets/notary.env; set +a   # PULP_NOTARY_KEY_PATH/_KEY_ID/_ISSUER_ID
xcrun notarytool submit out.pkg --key "$PULP_NOTARY_KEY_PATH" \
  --key-id "$PULP_NOTARY_KEY_ID" --issuer "$PULP_NOTARY_ISSUER_ID" --wait
xcrun stapler staple out.pkg                  # embed the ticket so it works offline
spctl --assess --type install -vv out.pkg     # now: accepted / source=Notarized Developer ID

If notarization is Invalid, fetch the per-file reasons: xcrun notarytool log <submission-id> --key ... --key-id ... --issuer ....

Linux: Build → Package (.deb / .tar.gz, no signing)

pulp build                                              # Must build first
pulp ship package --version 1.0.0                       # Creates a .deb (or .tar.gz)
# Standalone app → single-file AppImage (point --binary at the built executable):
pulp ship package --version 1.0.0 --format appimage \
    --binary build/examples/myapp/MyApp_Standalone [--icon myapp.png]

pulp ship package on Linux builds a Debian package from the plugin bundles in build/{VST3,CLAP,LV2} via pulp::ship::create_deb (ship/platform/linux/package_linux.cpp), installing them under /usr/lib/{vst3,clap,lv2}. When dpkg-deb is not on PATH it falls back to a .tar.gz. Linux has no signing requirement.

AppImage (standalone apps): --format appimage routes to pulp::ship::create_appimage, which wraps a single standalone executable (plugins ship as .deb/.tar.gz, not AppImage). It synthesizes an AppDir (AppRun launcher, <app>.desktop, an icon — caller --icon or a built-in 1×1 PNG placeholder, .DirIcon) and invokes appimagetool with ARCH=<appimage_arch()> (note: aarch64/x86_64/…, distinct from the Debian arch names). It honest-fails (returns false, no stray AppDir) when appimagetool is absent or the executable is missing — appimagetool is not vendored. The standalone binary must be passed explicitly via --binary; the plugin-centric build/ layout has no standardized standalone location to auto-discover. To run the produced AppImage at the end-user's side, FUSE (libfuse2) or APPIMAGE_EXTRACT_AND_RUN=1 is needed, same as any AppImage.

Routing invariant: the pulp ship package branch in tools/cli/cmd_ship.cpp keeps three mutually exclusive platform arms — #if defined(__linux__) (→ .deb via create_deb, .tar.gz fallback), #if defined(__APPLE__) (→ .pkg/.dmg), and an honest-fail #else for other Unixes. pkgbuild/hdiutil exist only on macOS, so the Linux arm must never fall through into them. Inside the Linux arm, --format appimage is an early sub-branch (standalone executable) that returns before the plugin-bundle .deb/.tar.gz path. If you touch that block, preserve the mutual exclusion and re-run test_cli_ship_shellout.cpp (the Linux routing regression guard) plus test_linux_packaging.cpp (the create_deb/create_tar_gz/create_appimage helper coverage; the AppImage real-build case runs only where appimagetool is installed — e.g. the tartci VM with libfuse2 — and verifies honest-fail otherwise).

The plugin-bundle Linux path must also fail when no actual .vst3, .clap, or .lv2 bundles exist under the build directory. Do not reuse the macOS "Created 0 .pkg and 0 .dmg" empty-artifact summary on Linux; report missing plugins and return a non-zero exit instead. The regression guard is test_cli_ship_shellout.cpp's "pulp ship package on Linux with no plugin bundles reports missing plugins" case.

Architecture field: create_deb stamps the .deb Architecture: from the compile-time debian_architecture() helper in installer.hpp, so a native build labels the package for its own arch (arm64/amd64/…) rather than a fixed value. Keep that helper and the package field in sync.

Android: Build → Package (includes Gradle build + optional signing) → Verify

pulp build --target android                             # Build native libs
pulp ship package --target android --keystore ~/key.jks # Gradle build + sign APK/AAB
pulp ship check --target android                        # Verify APK/AAB signatures

Note: pulp ship package --target android invokes Gradle which builds AND signs in one step when a keystore is provided. Use pulp ship sign --target android only to re-sign existing artifacts in artifacts/.

Windows: Build → Sign → Package

pulp build                                              # Must build first
pulp ship sign --identity "Your Company"                # Uses signtool
pulp ship package --version 1.0.0                       # Creates NSIS .exe installer

Windows does not notarize, but ship/platform/win/codesign_win.cpp must still define every public notarization symbol from ship/include/pulp/ship/codesign.hpp. Keep ASC-key notarize_submit_asc(...) as a fail-closed std::nullopt stub so pulp-test-codesign links on Windows and the cross-platform API surface stays in lockstep with macOS/Linux.

signtool failure contract. codesign() on Windows must never report success for an unusable signature: it rejects an empty identity/path up front (no signtool sign /n ""), and after signing it runs signtool verify /pa and returns false if verification fails — so a sign that exits 0 but leaves the artifact unverifiable still fails. The empty-input reject is covered by the [windows]-guarded case in test_codesign.cpp (runs without signtool present); the verify-after-sign path is compile-verified on the Windows lane (a real round-trip needs a cert).

NSIS installer (W7). generate_nsis_script() is a pure function — assert its output in test_nsis_installer.cpp (cross-platform, real verification, no NSIS/Windows needed). It places VST3/CLAP under $COMMONFILES\{VST3,CLAP} (or $LOCALAPPDATA\Programs\Common\... for --per-user), and standalone apps get a Start-menu shortcut under $SMPROGRAMS\<publisher> (removed on uninstall); plugins get none. When you touch the generator, add/extend the pure-output assertions.

Auto-update decision (W7): DEFERRED. No WinSparkle/auto-update is wired on Windows. Rationale: Pulp targets developers shipping their own apps/plugins, who pick their own update channel (DAW-scanned plugins don't self-update at all); pulling WinSparkle in would add a dependency for a story the SDK doesn't own. macOS Sparkle appcast generation stays available for those who want it. Revisit if a first-party standalone-app updater becomes a requirement.

Android-Specific

ABI Selection

--abi arm64-v8a     # Default — ARM64 phones + tablets
--abi x86_64        # Emulator / Chromebook
--abi all           # arm64-v8a + x86_64 + armeabi-v7a

Tablet Support

No special flags needed. ARM64 covers phones and tablets. AAB with split APKs handles screen density automatically.

Keystore Creation

keytool -genkey -v -keystore release.jks -keyalg RSA -keysize 2048 -validity 10000

Password Security

Never store passwords in plaintext config. Use environment variable references:

store_pass = "@env:ANDROID_STORE_PASS"

Common Issues

A tag did not publish and ios-compile-gate is red

release-cli.yml runs the full iOS compile gate (test/cmake/test_ios_compile_gate.sh) on the tag, on the darwin-arm64 leg's runner, and the release publish job requires it. Per-PR gates run the iOS gate only for iOS-only surfaces, so a tag can be the first place an iOS break in shared-looking code shows (the nightly ios-compile-gate-nightly.yml normally catches it first and opens a tracking issue). A 77 exit (iOS SDKs absent on the runner) is a failure here, not a skip. Fix forward and cut a new tag; re-running the job only helps for an environment failure such as a configure timeout.

hdiutil: create failed - Resource busy — a process is vetoing unmounts

hdiutil create -srcfolder builds a DMG by attaching a temporary volume, copying into it, and unmounting it. Any DiskArbitration client may veto that unmount, and hdiutil then reports the whole operation as Resource busy and exits non-zero. The message names neither the volume nor the vetoing process, and it looks exactly like a transient collision — it is not. The veto persists until the offending client is restarted, so retrying, waiting, or detaching stale images will not clear it.

Find the dissenter by unmounting any image and reading the error:

diskutil unmount /Volumes/SomeImage
# Dissenter parent PPID 1 (/sbin/launchd)
hdiutil detach /Volumes/SomeImage
# Simulators are still shutting down. Please try again in a few moments.

Trust the PID, not the message. diskutil unmount names the dissenting process by PID; the sentence printed beside it can come from an entirely different subsystem. A wedged Mac Studio on 2026-07-10 reported PID 1801 while printing CoreSimulator's "Simulators are still shutting down" with no booted simulators — PID 1801 was Console.app. Killing it cleared the veto instantly; sudo killall -9 simdiskimaged had done nothing. Identify the PID, then quit that process:

diskutil unmount /Volumes/AnyMountedVolume   # → "... failed to unmount: '...' (PID NNNN)"
ps -p NNNN -o pid,user,comm=                 # what is it, really?
kill NNNN                                    # user-owned: no sudo needed

Two consequences worth knowing. Every failed create leaks its temporary attachment, so failures compound and hdiutil info fills with orphaned images — that is a symptom, not the cause. And a self-hosted CI runner on a wedged host will fail create_dmg produces a file from valid source and pulp ship release --dmg on every PR while GitHub-hosted runners stay green, because they get a fresh VM.

create_dmg prints hdiutil's own output and this remedy on failure. It used to redirect hdiutil to /dev/null and return a bare false.

"No signing identity specified"

Run security find-identity -v -p codesigning (macOS) to find your identity, then:

pulp config set signing.apple.identity "Developer ID Application: ..."

codesign fails with errSecInternalComponent (or pops a GUI password dialog)

In a fresh agent / SSH / CI session this almost always means the dedicated signing keychain is locked and not on the search list, so codesign -s <hash> falls through to the (locked) login-keychain copy of the same Developer ID cert. The GUI dialog that appears is asking for the dedicated keychain's password (a stored secret, PULP_SIGN_KEYCHAIN_PW), not the user's login password — so a user typing their login password is correctly rejected. The non-interactive fix is to unlock the dedicated keychain from the secret and restrict the search list to ONLY it (so codesign can't reach the login keychain), then restore:

set -a; source ~/.config/pulp/secrets/keychain.env; set +a   # PULP_SIGN_KEYCHAIN, PULP_SIGN_KEYCHAIN_PW, …
security list-keychains -d user -s "$PULP_SIGN_KEYCHAIN"      # ONLY the dedicated keychain
security unlock-keychain -p "$PULP_SIGN_KEYCHAIN_PW" "$PULP_SIGN_KEYCHAIN"
# … sign / package …
security list-keychains -d user -s "$HOME/Library/Keychains/login.keychain-db" "$PULP_SIGN_KEYCHAIN"  # restore

tools/scripts/build_combined_installer.sh runs ensure_signing_ready.sh (the same pulp ship doctor preflight, single source of truth) before signing, so pulp ship package / a plugin's package.sh sign prompt-free out of the box — no manual pulp ship doctor needed first, and the fresh-machine case (keychain or .p12 not yet imported) is covered too. A failed preflight terminates the installer before its first codesign; there is no skip or warn-and-continue escape hatch for unattended packaging. (zsh trap: ${PIPESTATUS[0]} is empty in zsh — it's pipestatus, 1-indexed — so a codesign exit reads blank when it actually succeeded; verify with codesign --verify --strict or run the check under bash -c.) Full local sign+notarize recipe (inner-out dylib signing, the *.cstemp leftover, pkgbuild, notarytool, stapler) in macOS manual sign + notarize from a worktree above.

"Android SDK not found"

Install Android Studio or set ANDROID_HOME:

export ANDROID_HOME=~/Library/Android/sdk  # macOS

"Notarization failed" / "no notary credentials resolved"

• For the ASC-key flow: verify the .p8 is readable, the Key ID matches the filename suffix (AuthKey_<id>.p8), and the issuer UUID is from the same App Store Connect tenant. Use pulp ship notarize --dry-run to see which surface each value resolved from and confirm no fields are blank.
• For the legacy flow: regenerate the app-specific password at https://appleid.apple.com → Sign-In and Security.
• Ensure the bundle is properly signed with a Developer ID certificate (not just a development cert).
• Check the notarization log: xcrun notarytool log <UUID>.

sign-and-release.yml ships UNSIGNED unless its secrets exist — and says so

The repository holds none of the signing secrets (MACOS_CERTIFICATE, SIGNING_IDENTITY, APPLE_ID, …); real Developer ID signing happens on hosts from ~/.config/pulp/secrets. The workflow's Detect signing inputs step (id: signing) decides once and every signing step gates on steps.signing.outputs.sign / .notarize. Do not guard a step with if: env.X != '' where X comes from that step's own env: — a step's env: is not in scope for its own if:, so the guard is never true and signing is skipped silently (this shipped every release unsigned with no annotation). With no secrets the run is marked unsigned (warning, job summary, SIGNING-STATUS.txt, -UNSIGNED artifact suffix) and continues; vars.PULP_RELEASE_UNSIGNED_POLICY=fail makes that a failure. A partial secret set always fails. Once signing is attempted, a notarization that is not status: Accepted or a failed staple fails the job (notarytool --wait exits 0 on Invalid, so the status line is checked). Note that the packages are built with plain pkgbuild (no --sign); adding the secrets will surface that notarization rejects an unsigned installer package.

With no signing or notary secret at all, the macOS job does not run: the resolve-macos-runner job's preflight step (on the control-plane Linux runner, which can read secrets) outputs build=false and build-and-sign-macos is skipped with a No macOS signing configured notice. It used to build the whole tree on a release macOS runner, sign nothing, and upload a 278-byte status file on every tag. Any single secret, or PULP_RELEASE_UNSIGNED_POLICY=fail, runs the job so the in-job detector still fails a partial setup loudly. A job-level if: cannot read secrets; that is why the decision travels as a job output.

check_notarization and Gatekeeper-disabled CI environments

check_notarization(path) runs spctl --assess --type exec <path>. On a stock Mac, spctl --assess returns non-zero for an unsigned or nonexistent path. But CI base images (notably the cirruslabs macOS bases used by the Tart VM lane) ship with Gatekeeper assessment disabled (spctl --master-disable), in which case spctl --assess returns 0 for any argument — so it cannot distinguish a notarized binary from a bogus path. The fix (already applied in ship/platform/mac/codesign_mac.mm) is an explicit fs::exists short-circuit: a path that doesn't exist returns false before spctl is consulted, which is correct in both environments. If you add new spctl/codesign predicates, don't rely on spctl --assess alone for negative cases — guard with an existence / signature pre-check so they hold on Gatekeeper-disabled runners too.

"Gradle build failed"

• Run pulp doctor to check Android SDK/NDK/Java versions
• Ensure android/ project exists (pulp create --targets android)
• Check Gradle output in the terminal for specific errors

Appcast parsing must tolerate malformed metadata

pulp ship appcast feeds can be generated by older tools or edited by hand. When parsing existing Sparkle appcast XML, malformed optional enclosure metadata must fail soft. In particular, a non-numeric or overflowing length="..." attribute should leave the item's file size at 0 and keep parsing the item instead of throwing out of Appcast::from_xml. Keep this behavior covered in test/test_appcast.cpp when changing ship/src/appcast.cpp.

Appcast output paths

pulp ship appcast --output appcast.xml must work from a project root. When creating the output directory, guard empty parent_path() values before calling std::filesystem::create_directories; otherwise a bare filename can throw instead of writing the feed. Keep this covered in test/test_cli_ship_shellout.cpp.

When signing appcast entries, --url must be the local artifact path used for file size and Ed25519 signing. Use --download-url to write the public Sparkle enclosure URL into the feed. The hosted file must be byte-identical to the local artifact passed as --url; otherwise Sparkle rejects the update because the feed length and signature describe different bytes.

Sparkle in a standalone app (pulp_add_sparkle)

• App only, never plug-ins. pulp_add_sparkle() refuses a non-.app target. The update payload is the whole installer .pkg, so updating the app updates every bundle the package installs.
• The SDK never links Sparkle. pulp-standalone looks up SPUStandardUpdaterController through the Objective-C runtime and only accepts a class whose bundle lives in the app's own Contents/Frameworks. The app links the framework with -needed_framework so the load command survives even though no Sparkle symbol is referenced.
• Scheduled checks start only in a Developer-ID-signed build (Team ID on the main executable). Dev/CI launches show the menu item but never prompt or reach the network on their own; PULP_STANDALONE_UPDATER=on|off overrides.
• Two surfaces, one service. The standalone installs the backend as the process-wide pulp::format::AppUpdateService (app_updates.hpp) BEFORE the editor opens. The app menu (About, then Check for Updates… via MenuCommand::AppMenuSection::after_about), Pulp's Settings Updates tab and a custom editor's pulp_updates_* EditorBridge messages (add_app_update_handlers(), client useAppUpdates() in @pulp/react) all read and drive it. Plug-ins install no service, so the shared editor code sees available: false and renders nothing — never gate update UI on a compile-time "is standalone" flag the plug-in build also sets.
• The Settings note is derived, not written. It comes from RELEASES_URL, INSTALLER package|app and AUTOMATIC_INSTALL (default OFF → SUAllowsAutomaticUpdates=false). Declare them instead of hand-writing "requires an administrator password" — an app-bundle update needs none.
• Dev builds without Sparkle show nothing. PULP_STANDALONE_UPDATER=stub wires the menu item and Settings controls to a stub that never contacts a feed, so the placement can be exercised without a release feed.
• Status before the controller exists is read from Sparkle's own persistence: user defaults SUEnableAutomaticChecks (falling back to the Info.plist key) and SULastCheckTime. Writing the toggle there before Sparkle starts is what SPUUpdater's setter does; once the controller exists, go through -[SPUUpdater setAutomaticallyChecksForUpdates:] so it reschedules.
• Signing order is load-bearing. codesign without --deep never reaches inside a framework, and notarization rejects Sparkle's upstream ad hoc signatures. build_combined_installer.sh signs XPCServices/*.xpc (--preserve-metadata=entitlements), nested *.app (Updater.app), helper executables (Autoupdate), then the framework, then the app. Versions/Current is a symlink: walk it with find -H, or the helpers are silently skipped and only notarization notices.
• Feed metadata. A .pkg enclosure needs sparkle:installationType="package" (pulp ship appcast adds it). Sparkle compares sparkle:version (--build-number) with CFBundleVersion, so it must rise every release; the CLI refuses an older build in the same channel. Release notes: prefer inline --notes-html-file over --release-notes-url when the notes would be a GitHub release asset — those are served with Content-Disposition: attachment, not as a page.
• Sparkle refuses file:// feeds at run time ("The download request URL must use http or https"), so a local rehearsal serves the feed and package from loopback (python3 -m http.server --bind 127.0.0.1) and pulp_add_sparkle rejects file:// at configure time. Measured on Sparkle 2.10.0: the updater started and checked, then failed on the scheme.
• Keys. Pass --sign-key-file, never --sign-key <key>, so the private key stays off argv. Sparkle cannot rotate keys for copies already installed: back the private key up before the first release that embeds it.

Android package tests fail only on Windows

Pulp executes Android package helpers through cmd.exe /c on Windows. Android SDK tools and Gradle wrappers may resolve to .bat files there, so command strings that begin with a quoted batch path need an outer command quote so cmd.exe does not strip the executable quote while preserving the remaining quoted arguments. For Gradle and bundletool name=value parameters that contain paths or passwords, quote the whole name=value token rather than only the value ("--output=C:\path\file.apks", not --output="C:\path\file.apks"), because Windows batch %~1/shift parsing does not normalize embedded quotes the same way POSIX shells do. Gradle wrapper invocations should use the ChildProcess working_directory option instead of inlining cd ... && gradlew into the shell string, so fake wrappers and real Gradle builds write artifacts relative to the project root consistently. Keep this in mind when touching ship/platform/android/package_android.cpp.

Released CLI tarball crashes on user machines

Symptom: dyld: Library not loaded: @rpath/libwgpu_native.dylib or error while loading shared libraries: libwgpu_native.so after the user extracts pulp-<platform>.tar.gz from a GitHub Release.

Root cause: release-cli.yml historically uploaded the bare build/tools/cli/pulp binary, which carries an LC_RPATH / DT_RUNPATH pointing at the build runner's home directory (e.g. /Users/runner/Library/Caches/Pulp/... on macOS, the GitHub-hosted-runner workspace on Linux). That path doesn't exist on user machines.

Fix (active since v0.20.x): tools/scripts/package_cli.py is invoked by release-cli.yml to:

1. Copy libwgpu_native.{dylib,so,dll} next to the binary.
2. macOS: install_name_tool -delete_rpath every absolute LC_RPATH and add @loader_path.
3. Linux: patchelf --set-rpath '$ORIGIN'.
4. Windows: no rewrite needed — DLLs resolve from the binary's directory automatically.

The portable-binary smoke gate in release-cli.yml runs the produced artifact on a clean runner that did not build it, catching the bug class before tagging. If you change rpath logic, run the smoke job locally first or it will fail in CI for everyone else.

A flat unpack is not the installed layout. v0.876.1's archive was correct and passed the flat smoke, yet install.sh left libwgpu_native.dylib out of ~/.pulp/bin whenever the optional broker failed, and pulp-cpp died in dyld on every host. The Unix smoke legs therefore also install the artifact through the default branch's install.sh (PULP_INSTALL_ARCHIVE, no download, a scratch HOME and install root where the broker is refused) and run tools/scripts/check_installed_rpaths.py on the installed tree. To reproduce locally: PULP_INSTALL_ARCHIVE=pulp-darwin-arm64.tar.gz PULP_INSTALL_DIR=/tmp/i/bin PULP_NO_MODIFY_PATH=1 PULP_SKIP_SDK_INSTALL=1 bash tools/install/install.sh && python3 tools/scripts/check_installed_rpaths.py /tmp/i/bin.

Phase 8 CLI release artifacts are dual-binary

After the Rust CLI flip, release artifacts must contain both pulp and pulp-cpp in the same archive. pulp is the user-facing Rust CLI; pulp-cpp is the C++ delegate used for fallthrough commands that still link framework libraries. Do not ship pulp-rs as the public binary name, and do not drop pulp-cpp from tarballs or zips.

Smoke both paths when touching .github/workflows/release-cli.yml or tools/scripts/package_cli.py: run pulp version --json against the Rust binary, then exercise a C++-owned command through PULP_RS_CPP_BINARY=/path/to/pulp-cpp pulp ... or invoke pulp-cpp directly. This preserves rollback/debug workflows such as PULP_USE_CPP=1 pulp <args> for existing Pulp projects.

CLI tarball also ships pulp-mcp

The release tarball is three binaries — pulp, pulp-cpp, and pulp-mcp. pulp-mcp is the Claude Code plugin's MCP server. The plugin ships no binaries; its .mcp.json invokes the full path in PULP_MCP_BINARY, which the CLI installers persist without depending on Claude Code's inherited $PATH. If the CLI tarball is missing pulp-mcp, every fresh claude plugin install pulp produces a /mcp panel reporting a missing server even though the plugin itself installed cleanly.

Contract that must hold across release lanes:

• tools/scripts/package_cli.py is called with --mcp-binary build/tools/mcp/pulp-mcp (Unix) or build/tools/mcp/Release/pulp-mcp.exe (Windows).
• .github/workflows/release-cli.yml strips pulp-mcp and adds it to the smoke matrix as pulp-mcp --version (its JSON-RPC stdin loop is not meaningfully testable from CI; the flag short-circuits before reading stdin).
• tools/install/install.sh / install.ps1 configure PULP_MCP_BINARY for ~/.pulp/bin/ so users see the binary that the plugin will use.
• pulp upgrade --install extracts the whole tarball, so refreshing the CLI also refreshes pulp-mcp.

On control-enabled macOS release lanes, the CLI archive is also an atomic three-file broker installation: pulp-control-broker, its sibling pulp-control-standalone-host, and that host's exact manifest. Strip, sign, package, install, upgrade, and roll back that companion set together. A release at or above the control-host compatibility floor must fail if any member is missing; shipping only the broker leaves ordinary author Standalone capability declarations impossible to launch securely.

Document and enforce install order in user-facing docs: curl install.sh | sh BEFORE claude plugin install pulp. The reverse order leaves /mcp broken until the user upgrades the CLI.

Do not lock-step plugin and pulp-mcp versions. The plugin and pulp-mcp are installed via separate channels and a project may pin an older SDK. pulp-mcp advertises its real PROJECT_VERSION in serverInfo.version, but the launcher must remain version-tolerant — backwards compat is the MCP server's responsibility (per-tool feature detection), not the launcher's. pulp doctor surfaces drift advisory- only.

Per-tool min_sdk_version floors live in tools/mcp/pulp_mcp.cpp::TOOL_MIN_SDK_TABLE. When adding a new MCP tool that depends on a specific SDK API, add a row to that table with the SDK version that API landed in. The tools/call dispatcher reads the active project's pinned SDK (from pulp.toml sdk_version first, then CMakeLists.txt project(... VERSION ...)) on every call and returns an isError:true content payload with actionable upgrade guidance when the project SDK is too old. The pulp_compat introspection tool exposes the full matrix (pulp_mcp_version, mcp_protocol_version, project_sdk, tool_min_sdk) so plugin authors can pre-filter their visible tool list at startup. Leaving a tool out of the table = no floor = runs on any project (matches pre-feature-detection behavior). Never replace this with a launcher-side hard gate.

Released SDK is missing WebView symbols

Symptom: a consumer links against a downloaded pulp-sdk-<platform> release artifact and fails to resolve pulp::view::WebViewPanel::create or pulp::view::make_webview_embedded_resource_fetcher.

Root cause: core/view/CMakeLists.txt defaults PULP_BUILD_WEBVIEW=OFF, so any release SDK build that forgets to opt in will ship a libpulp-view-core / pulp-view-core.lib without the native WebView objects.

Keep the release SDK path aligned with the GitHub release workflow:

1. Configure release SDK builds with -DPULP_BUILD_WEBVIEW=ON.
2. On Linux, use .github/actions/install-linux-build-deps with the native-webview capability profile. Shared release headers belong in tools/ci/linux_build_deps.json; do not copy an apt list into release-cli.yml or its PR gate. Manual backfills can check out a tag that predates the local action, so the workflow materializes the action, installer, and manifest from the immutable workflow revision when they are absent before invoking it.
3. Before packaging, verify the staged SDK view-core archive still contains WebViewPanel and make_webview_embedded_resource_fetcher.

This applies to both .github/workflows/release-cli.yml and the local helper tools/scripts/release-cli-local.sh. If one changes without the other, GitHub releases and local release drills diverge.

Release/SDK builds pass -DPULP_ENABLE_AUDIO_PROBES=OFF so SDK and standalone artifacts do not ship the dev audio-probe surface. Starting at the product matrix's inspector_sdk_floor, they also pass -DPULP_ENABLE_INSPECTOR=ON because the matrix promises the optional inspector SDK archive family; older marker-era backfills keep it OFF. That build-time component does not enable runtime inspector endpoints; those remain off by default. Keep .github/workflows/release-cli.yml, .github/workflows/sign-and-release.yml, and tools/scripts/release-cli-local.sh in sync when changing release configure flags.

Starting at tools/scripts/release_product_matrix.json's sdk_provenance_floor, the SDK tarballs also require sdk-provenance.json: a positive official-release marker bound to the exact release tag commit and archive platform, with a clean Release source, audio probes disabled, and the inspector SDK component set according to inspector_sdk_floor. release-cli.yml stamps the selected install prefix, and its downloaded-asset finalizer re-verifies the exact tag SHA/platform and parses the installed include/pulp/runtime/build_info.hpp before publication. Dirty/non-Release build metadata, or a build-info version/short-SHA inconsistent with the marker, fails closed. Untracked configure/build inputs are excluded from the dirty bit; tracked source changes are not. Manual-backfill compatibility helpers must stay under RUNNER_TEMP, outside historical tagged checkouts whose older probes count untracked files as dirt. Checkout-relative Linux dependency-action files are recorded when materialized and removed immediately after use, before configure. Marker-era manual backfills must build the tag itself; source_ref substitution is reserved for pre-marker history.

Starting at capability_handoff_floor, official SDK tarballs additionally require share/pulp/agent-capability-handoff.json and its installed schema. The provenance stamper emits the handoff only after cmake --install, binding the exact release source SHA and platform to the SHA-256 of the installed platform-specific bin/pulp-import-design executable and to both the exact installed agent-capabilities.json content and its byte hash. The native archive check and downloaded-draft finalizer revalidate the handoff, both installed schemas, the importer bytes, and the capability bytes before publication. Keep sdk_capability_handoff.py, json_schema_lite.py, and the authoritative capability floor in the manual-backfill helper overlay; otherwise current releases fail closed or historical releases are incorrectly forced to synthesize a contract they predate.

Shipyard pin: one source of truth

tools/shipyard.toml is the only Shipyard pin. Local installs, shipyard pr, and every workflow that needs the CLI (including version-at-land.yml, which renders CHANGELOG.md in the bump commit) install through tools/install-shipyard.sh, which reads it. Do not add an inline SHIPYARD_VERSION to a workflow; it drifts from the pin unnoticed. Use shipyard update --check --json to report local drift and shipyard update --to <pin> before cutting or debugging release jobs.

VST3 SDK tag drift in sign-and-release.yml

Pulp's notarized macOS release workflow clones the Steinberg VST3 SDK directly inside .github/workflows/sign-and-release.yml. Keep that workflow pinned to the same upstream tag used everywhere else in the repo: v3.8.0_build_66. The shortened v3.8.0 ref does not exist on Steinberg's repo and causes the tag-triggered macOS release job to fail immediately at Clone VST3 SDK, before configure, build, or signing begin.

Never run validation ctest tests in sign-and-release.yml

The Test step in .github/workflows/sign-and-release.yml MUST pass -LE validation to ctest. Without that flag, the suite includes auval-Pulp* tests that copy a freshly-built .component to $HOME/Library/Audio/Plug-Ins/Components/ and immediately invoke auval -v aufx <code> Pulp. On hosted GitHub macOS runners the AudioComponentRegistrar does not pick up the new bundle reliably, so auval emits:

ERROR: Cannot get Component's Name strings
ERROR: Error from retrieving Component Version: -50
FATAL ERROR: didn't find the component

The Test step then exits non-zero and the entire sign / notarize / publish pipeline silently fails. This failure mode can block many consecutive tag-triggered release runs before anyone notices.

The validation gates already run in .github/workflows/validate.yml on PR with the documented codesigning caveat. Re-running them in the release workflow on a runner that cannot satisfy the prereqs adds zero protection and only adds a silent-failure surface.

tools/scripts/test_release_workflow_test_step.py is the regression test that asserts -LE validation stays in the workflow; it is wired into .github/workflows/workflow-lint.yml so any future PR touching .github/workflows/** runs it automatically.

sign-and-release.yml must declare contents: write

Every release workflow that uses softprops/action-gh-release@v2 with generate_release_notes: true — or that otherwise PATCHes the release entry — needs an explicit job-level permissions: contents: write block. Without it the job inherits a read-only token on push: tags events in many repo configurations, and the final Create GitHub Release step fails with:

Skip retry — your GitHub token/PAT does not have the required
permission to create a release
##[error]Resource not accessible by integration

Everything up to that point — checkout, VST3 SDK clone, configure, build, codesign, notarize, artifact upload — succeeds, and the pipeline still exits non-zero. macOS-signed artifacts never land on the release. Classic silent-release-failure pattern.

The fix is a four-line addition at the job header:

jobs:
  build-and-sign-macos:
    runs-on: macos-14
    permissions:
      contents: write

Cross-reference: release-cli.yml already sets this on its release-creating job (line ~360). If you add a new release-time workflow, do the same. The regression test tools/scripts/test_release_workflow_test_step.py now includes SignAndReleaseContentsWriteTest to block reintroduction.

Skia-builder zip layout drift breaks the release matrix

release-cli.yml fetches prebuilt Skia binaries via tools/scripts/fetch_skia_for_release.py after setup.sh --deps-only. The upstream skia-builder zip layout is not stable — older series shipped libs flat under build/<plat>-gpu/lib/Release/libskia.a, but the chrome/m144 series moved them one directory deeper under an arch subdir: build/mac-gpu/lib/Release/arm64/libskia.a, similarly linux-gpu/lib/Release/x64/. Every release after v0.94.0 (v0.95.0.. v0.97.0) failed silently on this for four days, with release-guard.yml opening a per-tag tracking issue but no published GitHub Release.

The fetch script now flattens any single-arch subdir back up into Release/ after unpack, so FindSkia.cmake's layout probe keeps working unchanged. Regression coverage lives in tools/scripts/test_fetch_skia_for_release.py (covers flat layout, arch-subdir flatten for both mac-arm64 and linux-x64, missing-lib, sha256 mismatch). Wired into workflow-lint.yml.

When tools/deps/manifest.json bumps Skia to a new skia-builder release tag, eyeball the zip layout once:

curl -sL <new asset URL> -o /tmp/skia.zip
unzip -l /tmp/skia.zip | grep -oE '^.*/lib/Release/[^/]+/' | sort -u

If the lib path has a NEW level (not arch — e.g. a config subdir like Release/optimized/), the flatten heuristic needs extending. The flat

• single-arch-subdir layouts are the only two seen to date.

Also eyeball exposed symbols. Some Skia static lib builds re-expose fontconfig symbols (FcInitLoadConfigAndFonts, FcConfigGetSysRoot, FcPatternGetString et al.) that the previous release kept private. core/canvas/CMakeLists.txt already has the pkg_check_modules(FONTCONFIG fontconfig) block, but the runner needs libfontconfig1-dev installed for pkg_check_modules to find the library. Both release-cli.yml and build.yml Linux deps steps now include it. When bumping Skia, run nm -D on the new libskia.a and grep for Fc[A-Z]\|Hb[a-z]\|FT_ — any new symbol class means a matching system package needs to be added to the apt step.

A new fork build lane must be wired into BOTH the matrix AND the release upload — and Pulp must require GPU only where an asset exists. The chrome/m150 incident (v0.395.0 stuck as a draft for days): the skia-builder fork's linux-arm64 build lane was added (634672f) ~20h after m150 was already cut, and that commit wired the build matrix but not the create-release files: list — so the slice was built every run, uploaded as a workflow artifact, and silently dropped from the published release. Meanwhile release-cli.yml set PULP_REQUIRE_GPU_FOR_SDK=ON for linux-arm64 while tools/deps/manifest.json had no linux-arm64 asset, so the release leg FATALed with "Skia not found" and never promoted the draft. Two guards now catch this class:

• skia-builder tools/check_release_coverage.py (+ lint.yml): every active build-matrix lane must appear in the create-release upload list. Catches "built but not released."
• Pulp tools/scripts/test_release_cli_gpu_asset_coverage.py (wired into workflow-lint.yml): every release-cli platform marked PULP_REQUIRE_GPU_FOR_SDK=ON must have a release_assets entry in manifest.json (via fetch_skia_for_release.py's MATRIX_TO_MANIFEST). Catches "required but not provided" — the contradiction that test_skia_linux_arm64_asset.py deliberately tolerated. When a fork release drops or adds a slice, update the manifest release_assets + tools/harness/visual/pins.py in lockstep, and only keep a platform in the REQUIRE_GPU=ON case blocks while its asset is actually published.

A dispatch-published release is unverifiable by a provenance-pinning consumer

Publishing an old tag by workflow_dispatch from main works, and the bytes are fine, but the SLSA provenance names main's HEAD rather than the tag commit — GitHub derives it from the run's OIDC context, not from the checkout. A consumer running gh attestation verify --source-digest <tag commit> then refuses the artifact, and it is right to. Supersede a broken tag with a new one cut from the fixed default branch instead; never relax the consumer's check to make it green. See the ci skill for the full write-up and the guard that now enforces it.

A Windows leg failing an archive digest is CRLF, and it blocks the whole release

The archive verifier hashes raw tar/zip member bytes. Git for Windows ships core.autocrlf=true, so a checkout rewrites LF to CRLF and any digest-pinned text file hashes to bytes no pin can describe — on Windows only, while Linux passes on the identical commit. Because release-cli.yml's publish job gates on the all-platform build-cli matrix, that one leg keeps the release object from ever being created, so the symptom you see is "tag exists, no release."

Force upstream LF bytes before the build-cli checkout (a git config afterwards is too late), and never relax the digest instead — the SDK contract is that every platform embeds identical catalog bytes:

- name: Check out release sources with upstream LF bytes
  shell: bash
  run: |
    git config --global core.autocrlf false
    git config --global core.eol lf

Diagnosis is arithmetic, not inspection: a CRLF'd text member's size equals its LF size plus its newline count (NOTICE.md 79246 + 1892 = 81138). See the ci skill for the full write-up and the regression tests.

Backfilling a stuck release tag

auto-release.yml creates the tag immediately on merge, but release-cli.yml only publishes after the matrix is green. If matrix fails, the tag exists with no Release — and workflow_dispatch against that tag re-runs the BROKEN workflow file from the tag's source. Two safe options:

1. Main workflow + blank source_ref: Run the fixed workflow from main, pass the tag as version, and leave source_ref blank. That checks out the tag's source while enabling the backfill overlay step:

   gh workflow run release-cli.yml --ref main \
       -f version=v0.97.0
   

   The build-cli job overlays safe release-pipeline helper files from main automatically, so a backfill picks up post-tag fetch-script, packaging, manifest, and targeted CMake fixes even though the tag's tree predates them. It also overlays the current release_artifact_contents.py compatibility engine while retaining the tag's own product matrix. CLI packaging and smoke checks include the import-design binary/runtime only when that checkout built them; the verifier uses the release version to distinguish the original three-binary contract from the transitional v0.764.0 import-design payload, before newer matrices declare CLI members directly. Leave make_latest false for old-tag backfills; set it true only when backfilling the current newest tag after the automatic tag-triggered run failed.

2. Cherry-pick fix + retag: Only if the build itself needs to change. Pulp doesn't retag immutable releases — use option 1 unless the broken release artifacts would have been wrong even with a green build.

Doctor Checks

pulp doctor validates Android toolchain:

• Android SDK location
• NDK version (r26+ required for C++20)
• Java version (17+ required for AGP 8+)
• Build-tools availability (apksigner, zipalign)

ONE job publishes the release, end to end

release-cli.yml's release job is the sole writer of the GitHub Release. It downloads the 6 platform pairs, writes appcast.xml itself, attaches all 13 assets, verifies the EXACT required asset set, generates SHA256SUMS, and publishes — all inside one job. It sets the title (the bare tag, e.g. v0.659.0) and the body (humanized highlights from compose_release_notes.py --footer).

sign-and-release.yml signs and notarizes the example-plugin .pkg bundles and keeps them as workflow artifacts. It holds contents: read and cannot touch a release. It may fail, hang, or be cancelled without affecting whether the SDK ships.

Do not re-couple publication to the signing leg. It used to be a handshake across three workflows (release-cli drafted → sign-and-release polled for that draft to attach appcast.xml → a coordinator flipped the draft once BOTH legs succeeded). That made the release only as reliable as its worst leg and produced a 7% first-attempt success rate — 11 of 18 consecutive tags never published, several with all six platform binaries built green. The poll was the killer: it waited a fixed window for a draft that does not appear until the full build matrix finishes (70-165+ min, because the macOS legs queue for hours on the shared hosted pool), so every slow release timed it out.

If you find yourself widening that timeout, stop — that is treating the symptom, and it burns a macOS runner doing nothing but waiting. appcast.xml is a pure function of the tag name and the date (no enclosure, no signature, no dependency on a notarized artifact), which is why it can be, and is, written by the release job itself.

Guarantees are structural so they cannot quietly regress, and each is pinned by a test in tools/scripts/test_release_workflow_test_step.py:

Invariant	Enforced by
sign-and-release cannot write a release	it holds contents: read
auto-release cannot cancel a release run	it has no actions scope
the advisory universal gate cannot block a release	it is not in release's needs
recovery can only drive a release forward	release_reconcile.py has no cancel/delete path

A published GitHub release is IMMUTABLE. Assets can only be attached while it is still a draft, which is why the release job creates a draft and publishes it in the same job (the draft lives for seconds and is never observable from outside). It also means a release that publishes with a missing asset cannot be repaired — it needs a new patch tag.

Deleting a once-published release permanently BURNS its tag name. GitHub reserves an immutable release's tag_name forever, even after deletion: every later publish attempt for that tag fails with HTTP 422: tag_name was used by an immutable release, and no re-dispatch can ever succeed (v0.807.0 and v0.808.0 were burned this way on 2026-08-16). Deleting a draft destroys its attached assets — though the binaries survive as the building run's workflow artifacts for ~90 days (gh run download <run-id>). So: NEVER delete a release or draft; recovery from any failed publish is a NEW patch tag. The publish step classifies the burned-tag 422 explicitly, and release-deleted-tripwire.yml files a tracking issue the minute any release is deleted.

Slow is survivable; racy is not. A three-hour release still publishes. Never add automation that cancels or deletes an in-flight release because a newer tag appeared: releases legitimately complete OUT OF ORDER now (the pipeline outlasts the ~100-min gap between tags), and the "supersede reaper" that assumed otherwise is what destroyed most of July 2026's releases.

A stuck release repairs itself: release-reconcile.yml re-dispatches any recent tag that never published (max 3 attempts), leaves any tag with a LIVE run alone at any age, and keeps exactly one incident issue. Don't hand-backfill unless it escalates.

Both macOS legs prefer PULP_RELEASE_MACOS_RUNS_ON_JSON. The signing leg then uses optional Namespace or GitHub-hosted macos-15; it deliberately does not fall back to the shared local PR pool because it imports a Developer ID private key.

NEVER set run-name: on release-cli.yml (it stops all releases)

GitHub returns a workflow's run-name as workflow_run.name in the REST API — it REPLACES the workflow name, it does not sit alongside it. The self-hosted tartci supervisor that provisions release-cli's macOS VMs picks up its work with:

select(.name == "Release CLI")

So the moment a run-name is set, every release run becomes invisible to the supervisor. Its log reads queued=0 running_macos_vms=0/2 while release runs sit with their required darwin-arm64 leg queued forever. No macOS VM is booted, no runner appears, and no release can ever build — silently, for every future tag.

This shipped once (a run-name was added so release-reconcile.yml could attribute its repair runs, since a workflow_dispatch run's head_branch is the ref it was dispatched FROM, not the tag it builds). The tag now travels in a job name (resolve-macos-runner), which the API exposes as a separate field and the supervisor does not key on. tools/scripts/test_release_workflow_test_step.py asserts run-name never returns.

The general rule: a workflow's public identity (name, and therefore run-name) is an interface that self-hosted infrastructure keys on. Renaming a run is not cosmetic. Before changing it, grep the tartci supervisor config (TARTCI_RUNNER_WORKFLOW_NAME) for anything matching on it.

One installer for a plugin: standalone + plugins + diagnostics (don't ship them separately)

A user wants ONE installer, not a per-format .pkg plus a per-app .dmg. Do NOT reach for pulp ship package (emits a separate .pkg per format) and pulp ship share (a separate .dmg per app) piecemeal — that was the early SuperConvolver mistake. Use the canonical recipe:

tools/scripts/build_combined_installer.sh \
  --name MyPlugin --version X.Y.Z \
  --sign-identity <Developer ID Application hash> \
  --installer-identity <Developer ID Installer hash> \
  --out DIR \
  --plugin au   build/AU/MyPlugin.component \
  --plugin vst3 build/VST3/MyPlugin.vst3 \
  --plugin clap build/CLAP/MyPlugin.clap \
  --app "Standalone app" build/examples/myplugin/MyPlugin.app \
  --app "Diagnostics helper" "/path/MyPlugin Diagnostics.app" /path/Kit.entitlements \
  --content "Sample models" "Example .nam captures" \
           "/Library/Application Support/MyPlugin/models" build/models

It produces ONE component-selectable, notarized installer (Customize pane picks AU/VST3/CLAP/Standalone/Diagnostics). It deep-signs every bundle (inner dylibs first → @loader_path) and runs check_bundle_relocatable.py --strict on each, so a build that only works on the build machine never ships. Identities are the security find-identity hashes from the dedicated pulp-signing keychain (NOT the ambiguous name). examples/super-convolver/package.sh is a thin wrapper — copy it for a new plugin. Intended to graduate into pulp ship package --combined.

Packaging a MULTI-plugin product, and packaging from OUTSIDE the Pulp tree

Two facts that are easy to guess wrong, and both were verified by reading the recipe rather than assumed:

--plugin accumulates. The parser is --plugin) P_KIND+=("$2"); P_PATH+=("$3"); shift 3;; — arrays, not scalars. So a product with several plugins passes one --plugin per artifact per format (e.g. 3 AU + 2 VST3 + 3 CLAP = eight flags) and gets one installer with a nested per-product Customize tree. Do NOT conclude the tool is single-plugin-only from the single-plugin example above, and do NOT build one installer per plugin.

The recipe is NOT installed into the SDK prefix. An installed SDK carries bin external include lib only — no tools/. A consumer project (a separate repo building against the SDK, e.g. Forge) therefore cannot reach build_combined_installer.sh from the SDK and must point at a Pulp source checkout:

PULP_ROOT=/path/to/pulp ./package.sh   # package.sh resolves $PULP_ROOT/tools/scripts/...

pulp ship has the same limitation from the other direction — cmd_ship.cpp resolves a project root via find_project_root(), which requires core/, and root is then used ~45 times for Pulp-tree-internal assets (root/tools/scripts/ensure_signing_ready.sh, root/ship/templates/entitlements.plist, …), so swapping the resolver is not a one-liner. Tracked in #6714. Until it lands, a consumer project's package.sh is the supported path — keep it to inputs only and exec the shared recipe, so signing/notarizing logic never forks per product.

Two things such a wrapper must do, because both failures are silent:

• Resolve identities by name → hash at runtime (security find-identity) rather than pinning a hash, or the script only works on one machine.
• Assert every expected artifact exists before invoking the recipe. A Customize pane that quietly lost a format still looks like a successful build.

Standalone apps must be non-relocatable package components. pkgbuild otherwise marks an app bundle relocatable, and Installer may overwrite any Launch-Services-known development copy with the same bundle ID instead of placing the selected app in /Applications. A successful Installer summary and receipt do not prove the destination. The recipe analyzes each staged app, sets BundleIsRelocatable=false for every discovered bundle, and passes the result through --component-plist. Keep the app-relocation assertion in test_build_combined_installer.py, and on a real release verify the final paths under /Applications rather than accepting receipts alone.

The script also accepts bundles from multiple products in one installer. Every component package and pkg-ref is keyed by the plugin's deterministic first-seen index plus format (plugin-0-au, plugin-1-au, ...), never by format alone or by a lossy slug of the display name. Multi-plugin installers nest each product's formats beneath a product choice; single-plugin installers retain the flat format list. Keep tools/scripts/test_build_combined_installer.py green when changing this graph: it uses fake package/signing tools, sets PULP_SKIP_SIGNING_PREFLIGHT=1 so it cannot inspect or mutate a developer's keychains, and asserts that colliding-looking names remain distinct. Its HOME is intentionally empty: optional keychain.env and notary.env files must be guarded with -f before source, because macOS Bash can terminate a set -e script at a missing sourced file before any package tool runs.

--content "Title" "Desc" DEST SRCDIR (repeatable) adds a selectable component that installs the contents of SRCDIR to an absolute DEST (e.g. sample models / IRs into /Library/Application Support/<Plugin>/...), rather than a plugin bundle or /Applications app. Use it to ship bundled demo content that a plugin loads at runtime. Titles/descriptions are XML-escaped before they go into the distribution XML, so &/</>/quotes in a human-readable title no longer corrupt the installer; the staging tree is removed via an EXIT trap on any exit (success, error, or signal).

Notarize works without pulp-cpp (standalone / submodule consumers). The script prefers the in-tree pulp-cpp ship notarize when it exists, but a plugin repo that vendors Pulp as a submodule never builds pulp-cpp (the CLI is gated to top-level Pulp builds). In that case the notarize step falls back to xcrun notarytool submit --wait + stapler staple using the file-based .p8 key from ~/.config/pulp/secrets/notary.env (PULP_NOTARY_KEY_ID / PULP_NOTARY_ISSUER_ID / PULP_NOTARY_KEY_PATH — the same trio pulp ship doctor provisions). If neither pulp-cpp nor a notary key is available the step fails loudly (never ships a signed-but-unnotarized .pkg); pass --no-notarize to opt out deliberately. Always stapler validate + spctl --assess --type install the finished .pkg regardless of path.

Release legs are individually routable (local pool / one machine / GitHub)

release-cli.yml resolves each leg's runner from a platform -> runs-on map (tools/scripts/resolve_release_runners.py), driven by per-leg repo variables. Moving a release build is a variable, never a code change:

tools/scripts/release_routing.sh show
tools/scripts/release_routing.sh local  linux-arm64      # -> local VM pool
tools/scripts/release_routing.sh pin    linux-arm64 m5   # -> that ONE machine
tools/scripts/release_routing.sh github linux-arm64      # -> revert, next tag

Fluidity invariant: every variable unset == today's GitHub-hosted routing. If the local pool is down, github <leg> is a full revert in one command.

The lightweight resolver jobs for release-cli.yml and sign-and-release.yml may use the always-on trusted MacPro Linux/X64 pool without moving artifact builds or publication there. Their selector priority is PULP_RELEASE_CONTROL_LINUX_RUNS_ON_JSON, then the existing PULP_LOCAL_LINUX_RUNS_ON_JSON, then ubuntu-latest. Keep this routing limited to tag-push or maintainer-dispatch workflows, and keep resolver policy checkouts pinned to the repository default branch; never expose the persistent pool to pull_request or merge_group code through this fallback.

Release class labels are opt-in (PULP_RELEASE_CLASS_TOKENS). Exactly 1 or true appends pulp-release-tagged (release-cli darwin legs, sign-and-release) or pulp-release-pr-gate (release-path-pr-gate) to a self-hosted selector, dropping pulp-gate-fast exactly as build.yml does for its event classes; any other value is ignored with a ::notice::, and unset is byte-identical routing. One implementation, resolve_release_runners.py --apply-class-label, serves all three workflows. Gotcha: never enable it before tartci's hosts advertise these classes, because GitHub matches only runners carrying EVERY label, so a class-labelled job with no serving registration queues forever. Unsetting is the rollback. The two shell resolvers sparse-checkout that script, so they fail at once if it is renamed.

Facts worth keeping (measured):

• The local macOS VM built darwin-arm64 in 6.4 min after a 0.8 min wait. GitHub-hosted legs took 39-72 min to EXECUTE, and darwin-x64 sat 127 min in the hosted queue. The queue, not the compile, was the release's long pole.
• darwin-x64 needs no Intel machine — it is already a cross-compile (-DCMAKE_OSX_ARCHITECTURES=x86_64), and the release golden has Rosetta, so the smoke leg can RUN the x86_64 binary it just built. Verified by booting the golden.
• The x64 Linux/Windows legs have no local preset. Both VMs are ARM64 guests, so x86_64 there means emulation, and these are shipped artifacts. Their hosted queues are ~0 — the pain was never there.
• Do NOT batch both arches into one job to "reuse the warm VM". The host ccache is already mounted into every ephemeral VM (that is why the build was 6.4 min), so a warm cache needs no long-lived VM.
• Parallelism is bounded by the tartci GOVERNOR, not Apple's limit. TARTCI_MACOS_HARD_MAX=2 is Apple's guest cap, but the governor reserves cores for the required macos PR gate (total_cores 14, reserved_gate_cores 8), leaving 6 — exactly ONE release VM per host. The darwin legs SERIALIZE on a host, and local release builds compete with PR validations for those same non-gate cores.
• GitHub Actions has NO runner priority. A pool labelset does not give M3-then-M5-then-M1 ordering; the job goes to whichever matching runner registers first. Ordering belongs on the tartci supervisor side. pin is the deterministic lever today.
• Host-label hygiene matters for pinning. A supervisor advertising two host labels (e.g. both pulp-host-m5 and pulp-host-macstudio) makes pin land somewhere you did not choose. Check the labels before trusting a pin.

cmd_ship shells out with shell_quote(), never concatenated quotes

Every shell-out in tools/cli/cmd_ship.cpp goes through shell_quote() from cli_common.hpp. It previously mixed three idioms — the shared helper, a local sh_quote lambda, and ~18 naive '"' + path + '"' splices — including on the signing-keychain reload and gh secret set paths, so identically-shaped paths behaved differently per call site. Adding a subcommand: quote with shell_quote(), do not copy whichever idiom is nearest.

A required component, and consent before installing

Use --app-for BUNDLE "Title" PATH when a standalone application belongs to a product group and cannot be deselected. The parser records that app's choice id in REQUIRED_APP_IDS, and distribution generation emits enabled="false" selected="true" on the choice. For Forge Modular the app is required because the Rack modules and uninstaller live inside its bundle; an install that skipped it would produce plugins with no generator and no removal path. add_ref() itself only creates the choice and package reference.

Pass --license FILE to put a consent pane in front of the install (or set the helper's LICENSE_FILE shell variable before argument parsing). The helper stages the basename under productbuild --resources and emits the matching <license> element in the distribution XML. A product wrapper may use its own environment variable, but it must translate it into --license; exporting an unknown variable does nothing.

Do not hard-wrap the licence text. macOS rewraps it to the pane width, so pre-wrapped lines come out ragged and broken-looking. Write each paragraph as one long line and let the installer wrap it; keep blank lines between paragraphs, and keep indented list items indented, since those are preserved.

Verify a built PKG by component payload size, never by signature. A staging directory assembled with symlinks instead of real copies produces a 292-byte package that signs, notarizes, staples and passes Gatekeeper while containing nothing. pkgutil --payload-files does NOT enumerate nested component payloads either — pkgutil --expand and check each component's size. Use ditto for staging copies.

Bind product-specific installers to rebuilt inputs, not recognizable ones. Size thresholds, stable strings, and matching manifests still accept an older binary built from the same metadata. Rebuild every named target from its pinned source in Release before staging; refuse tracked changes plus untracked or ignored files reachable by source/resource globs and copied roots; record SHA-256 identities for the source tree, external SDK, manifests, archives, and every payload binary. Verification expands the finished installer and compares those exact identities. An SDK content digest computed during packaging is only an observation, not trust: require an expected digest from the release manifest (or an explicitly computed local-development pin) and reject any mismatch. Mach-O payload identities may exclude the replaceable code signature only when they also canonicalize the signature-dependent __LINKEDIT sizes; prove this against a real codesign --force -s - replacement. Archive-producing CMake helpers must clear their stage on every package invocation: persistent POST_BUILD copy directories retain resources that were removed from source.

• Design-import release packages can have a helper runtime dependency outside bin/browser_capture-v1: the capture script imports the shared materialized binding contract from bin/jsx-runtime. Any release or SDK staging change must include that sibling directory, hash it in provenance, and add a missing-sibling negative control; a browser runtime-only smoke test is not sufficient.