daw-smoke — functional reload/editor verification in a real DAW (REAPER)

Use this when you need proof that a reload / editor / format-adapter behavior actually works inside a host, beyond what auval / pluginval / clap-validator give (those prove scan+load, not functional behavior).

Harness: tools/testing/daw-smoke/reaper_smoke.py (+ insert_and_float.lua and sequence_loop_seek.lua). The scraper + mode dispatch are unit-tested with no REAPER in tools/testing/daw-smoke/test_reaper_smoke.py. Full rules: docs/guides/daw-smoke.md. CLAUDE.md has the one-paragraph policy.

Modes (--mode, default reload)

• reload — the original flow. Seed watched DSP variant A, insert+float the FX, copy variant B over the watched path, scrape swapped DSP / reload rejected.
• live-plugin-swap — a live plugin-INSTANCE swap inside a Pulp host that hosts another plugin in a SignalGraph. Insert+float the host FX, write a swap request to --watched-swap-request, then scrape [live-swap] committed (PASS) vs live plugin swap refused (FAIL). The success marker is NOT logged by the core swap path — SignalGraph::prepare_swap logs ONLY refusals. The host plugin must emit [live-swap] committed itself from its NodeLiveSwapPolicy::on_instance_swapped observer (that is the seam the smoke scrapes and the headless test exercises). The DAW-free, CI-runnable mirror is test/test_signal_graph_live_swap_continuity.cpp, which drives the same stage + prepare_swap commit through process() and asserts sample continuity (no dropout/xrun) across the swap block for every hosted format (VST3/AU/CLAP/ LV2). Use the REAPER mode as the local-only in-host confirmation.
• sequence-loop-seek — set a loop region on the REAPER timeline, play, and perform scripted seeks (into/out of the loop and across the wrap) against a plugin that embeds a sequence. sequence_loop_seek.lua drives the transport via a deferred pump (GetSet_LoopTimeRange + GetSetRepeat + OnPlayButton + SetEditCurPos seeks), handshaking FX_SHOWN→SEEKS_DONE through the status file. The plugin emits per-block markers [seq-loop] blk host_qn=.. seq_qn=.. active=.. jump=.. dropout=..; the scraper (analyze_seq_loop_log) asserts the sequence read position TRACKED the host playhead within --pos-tolerance-qn on every block — a free-running counter that ignores the host jump is caught as drift → FAIL. PASS requires >=1 wrap AND >=1 seek, note activity, and no dropout; otherwise INCONCLUSIVE (a SKIP/INCONCLUSIVE is never a PASS). This is the harness half of Phase-2 DoD Proof #2 — the full proof also needs the embedded-sequence plugin (Gate-5/6) and a real REAPER run. The scraper is unit-tested with synthetic logs (incl. a negative drift/dropout case), so the parse logic is proven without REAPER.
• editor-open — insert the plugin, open its editor, confirm it rendered. For a product whose editor is the product, neither a hot-swap nor a transport scenario applies; the only question is whether it loads in a real host and draws. This is precisely the check the format validators cannot make: auval and clap-validator prove a plugin scans and instantiates, never that its window comes up. After FX_SHOWN it waits before tearing down, because an editor that opens and then throws reports itself in the seconds after the window appears.
• editor-build — generate a patch from inside the host. editor-open proves the window comes up; it does not prove the product behind it works. The generator is spawned by the plugin, and an editor that draws perfectly can still fail to reach it — the standalone once did exactly that, because an app launched from Finder inherits no PATH. Nothing short of triggering a real build inside the host tests that path. The trigger is a file the shell reads (FORGE_MODULAR_TEST_PROMPT), not synthetic clicks — see the gotcha below. The verdict is the generator's own success line plus the file it names, scraped from the plugin's log; never "the click appeared to land."
• keyboard-routing — press keys AT a focused plugin editor and ask the HOST what it received. Keyboard routing has two halves, and only one of them can be tested from inside the plugin: that the editor consumes its own shortcuts. The half that breaks is the other one — an editor that also swallows the keys it did NOT handle looks identical from every vantage point in the plugin, and the first report is a musician saying the spacebar stopped working. keyboard_routing.lua journals GetPlayState + Undo_CanUndo2 to PULP_DAW_SMOKE_JOURNAL on a defer pump, so the verdict is the HOST's own state: Space at a focused editor must start the transport and a second Space must stop it, and a ⌘Z the editor owns must NOT move REAPER's undo history. Refuses to post a synthetic key until _reaper_editor_is_front proves REAPER is the frontmost process AND its front window is the plugin editor — a global key event posted at a guess lands in somebody else's terminal on a shared machine. A macOS Accessibility refusal is a SKIP with that reason, never a PASS. Its decision logic is unit-tested against a synthetic journal (including the swallowed-Space FAIL), so the scoring is proven without REAPER. The DAW-free mirrors are test/test_plugin_key_routing.cpp (the shared policy) and the [host-forward] case in test_plugin_view_host_key_focus.mm (the real PulpPluginView seam, counting what reaches the host view).
• All modes share the same REAPER lifecycle (ReaperSession): fresh portable dir, temp scan path, pre-warm scan, scripted insert+float, guaranteed teardown. They differ only in what they SEED, the TRIGGER they fire once the FX is shown, and how they VERIFY the captured log. Add a mode by writing seed/trigger/verify around ReaperSession, not by duplicating the launch plumbing.

When to reach for it

• A headless / standalone / render_to_png capture PASSES but you're not sure it works in a host. It usually doesn't tell the whole story — see the gotcha below.
• Reload/editor/format-adapter change, especially before asking a human to test.
• Opt-in (default OFF), enabled per-machine in ~/.config/pulp/daw-smoke.toml; a ship gate only when enabled + gate + the diff hits the allowlist. NOT every build. DAW-Smoke: skip reason="..." trailer bypasses a single commit.

Gotchas (the expensive lessons)

Copy a bundle with ditto, never shutil.copytree

A macOS code signature lives partly in extended attributes, and copytree does not carry them. The copy the harness handed REAPER verified as "code object is not signed at all" while the original was clean under --deep --strict. Use ditto, and verify the COPY rather than the source:

ditto "$SRC" "$DST" && codesign --verify --deep --strict "$DST"

(The failure named a subcomponent — a non-executable .json in Contents/MacOS/, where bundle convention expects only executables. Worth knowing when reading such an error; --deep --strict treats everything in MacOS/ as a subcomponent.)

A custom scan path does not publish on REAPER 7.78 — the editor modes cannot prove anything there

editor-open (and any mode needing the FX inserted) currently reports INCONCLUSIVE on REAPER 7.78 with:

pre-warm scan did not publish the target into REAPER's plugin cache before timeout
REAPER did not open the FX within timeout (flaky launch) — INCONCLUSIVE

Reproduced four times at 90s, 150s and 240s timeouts, before AND after the ditto fix, and with the plugin also present at the standard VST3 path — the harness redirects REAPER to its temp scan_dir regardless, so installing normally does not change the outcome. The user's own REAPER cache DOES contain these plugins, so REAPER scans them fine through its ordinary paths; it is the harness's vstpath_arm64=<temp> redirection that is not taking effect on 7.78.

So a SKIP or INCONCLUSIVE here is not a PASS, and right now it is the expected result rather than evidence about the plugin. Note these modes also exit 0 while reporting INCONCLUSIVE, so anything reading exit codes scores them as passes. Until the scan-path redirection is fixed, prove editors another way and say plainly that this rung is unproven.

• Headless capture hides adapter-only bugs. The format adapter injects state no headless path has. 2026-07-04: it synthesizes a Bypass parameter in a host, so the reload param-contract gate rejected every in-DAW reload (parameter contract differs) while every headless capture passed. Only REAPER surfaced it (fixed 4a6e048f4: the contract gate now excludes an adapter-synthesized bypass). If you change reload/editor/adapter code, a real-DAW smoke is the check that catches this.
• screencapture returns BLACK frames in an agent/SSH context (no Screen- Recording TCC grant). Do NOT verify visually via screencapture. Verify by scraping the plugin reload log from REAPER's captured stdout (swapped DSP = applied, reload rejected + reject-diff = fail, loaded initial logic = it loaded). Use render_to_png (e.g. examples/hot-reload-morph capture) for human-viewable visuals.
• REAPER launch is flaky (scan dialogs, stuck instances, first-run per portable dir). Mitigate: fresh temp portable dir per run, a pre-warm scan launch you kill, timeouts, and an INCONCLUSIVE outcome — never a spurious FAIL. A STATUS: None/ no-FX-load run is INCONCLUSIVE, not PASS.
• Only ever kill the REAPER you started. pkill -9 -x REAPER to clear "stragglers" kills every REAPER on the machine — including the one the person at the keyboard is working in, with their unsaved project. It was reported as "REAPER keeps restarting"; it was being killed and relaunched underneath them. Keep the owned child as a Popen object and stop it through poll / terminate / bounded wait / kill / wait; never send a later raw signal to its saved numeric PID, which may have been recycled after the child exited. PULP_DAW_SMOKE_KILL_ALL=1 restores the blunt behaviour for a dedicated box with nobody at it.
• Seed the portable config FROM the user's real one — never from empty. The throwaway reaper.ini is what keeps a run from polluting somebody's install, but an empty one makes REAPER come up as a fresh install on every launch: licence prompt, first-run preferences, audio-hardware setup dialog, over and over. Copy the real config and licence in, override only the plugin-scan paths, and never write back. Two specifics that bite:
  • Force lastproject=/loadlastproj=0. Seeding carries lastproject, so the "isolated" REAPER opens the user's actual session — and if that session references a plugin the smoke did not install, REAPER sits on a modal Project Load Warning forever. The scripted insert never runs and the smoke reports INCONCLUSIVE naming neither the dialog nor the project, while a stranger's work is on screen.
  • Strip [Recent]. It is not needed to run a smoke, and copying it puts the user's project filenames into a temp dir and into REAPER's menus for the run.
• AU cannot be proven over SSH — and the failure is silent. An AU is found through the system AudioComponent registry, which is not visible to a process outside a GUI login session. Over SSH the component installs correctly and is completely invisible: auval -a lists nothing, REAPER's scan finds nothing, and the run ends INCONCLUSIVE — which reads as flakiness and sends the next person hunting a scan bug that does not exist. Detect SSH_CONNECTION/SSH_TTY and SKIP with that reason. VST3 and CLAP are unaffected: they are scanned from a directory.
• An already-installed AU is the normal case, not a collision. Refusing to run when ~/Library/Audio/Plug-Ins/Components/<name> exists makes the AU leg unrunnable on every machine anyone would actually test on. If the installed component resolves to the same path as the one under test, use it in place and leave the "we installed it" flag unset so teardown does not uninstall something this run did not install. Only a different component at that path is a real collision.
• Qualify the FX name by format, or the run proves the wrong plugin. TrackFX_AddByName with a bare name lets REAPER pick whichever format it finds first, so --format clap inserted the installed AU and the run reported a CLAP pass for an AU — caught only by a crash report naming the .au bundle during a clap run. Prefix the name with VST3: / CLAP: / AU:. Three legs that all load the same plugin are one leg wearing three hats.
• Prefer a trigger FILE over driving the screen — this is the settled answer. Synthetic clicks and keystrokes typed a generation prompt into somebody's terminal twice, because every guess about what is on screen finds a new way to be wrong. The claim editor-build proves — that the generator runs when the plugin spawns it, inheriting the host's environment — is just as true asked through a file as asked through a button, so editor-build asks through a file and touches nothing. If you are about to add screen-driving to a mode, reach for a file handshake first. The two traps below are why, and they still apply to any mode that genuinely must drive the UI:
  • Never trust "frontmost"; ask the window server who owns the pixel. REAPER reports itself frontmost while its editor sits under a remote-desktop session or a terminal full of live agent sessions, so clicks and typed text land there. On a shared machine that is not a failed test — it is typing into somebody's work. Probe the actual owner at the exact points you are about to click. A pixel/brightness heuristic is worse in the other direction: it refused a plainly visible editor because the panels were lighter than the surface it expected.
  • The hosted editor is not the host window. REAPER wraps a plugin editor in a preset strip and pads it, so window-relative fractions land in the chrome — one Build click hit the Module tab and started a module build nobody asked for. Measure the plugin's surface inside the window rather than assuming an offset.
• Dismiss screencaptureui, not "Screenshot". screencapture leaves the capture UI running with a floating window that then sits over the plugin editor, so a run that failed and grabbed a screenshot arms the obstruction that blocks the next run. Quitting the Screenshot app does not touch it — pkill -x screencaptureui does.
• Never pollute the user's machine. VST3/CLAP scan from a TEMP path (portable reaper.ini vstpath/clap_path), never ~/Library/Audio/Plug-Ins. AU can't be scanned from a custom path, so --format au installs to the real Components folder and MUST uninstall on exit. Cleanup ALWAYS runs (finally): kill REAPER, rm temp dirs, uninstall AU.
• The temp-path scan does not actually work — --format au is the only lane that tests anything. Measured 2026-07-30 on REAPER/macOS arm64: REAPER ignores the vstpath/clap_path keys written into the portable reaper.ini. It scanned the real plugin folders instead (124 CLAP + 104 VST3 entries in the portable caches) and never looked at the staged copy, so the bundle under test is absent from the scan cache and the FX is never found. Because a not-found FX is INCONCLUSIVE rather than FAIL, this degrades silently: the run reports "flaky launch" forever while testing nothing. Confirm a CLAP/VST3 run really scanned your plugin by grepping the portable reaper-clap-*.ini / reaper-vstplugins_*.ini for its name before believing any verdict. Until the key names are fixed, use --format au.
• SKIP is never PASS. Exit codes: 0 PASS / 1 FAIL / 2 SKIP (REAPER absent) / 3 INCONCLUSIVE. A gate must treat SKIP/INCONCLUSIVE as not-passed.
• A4 DPR v2 evidence requires all three format-qualified editor passes for every Forge DAW cell: AUv2 in Logic, plus VST3 and CLAP in REAPER. Bind each subreceipt to the exact A3 lifecycle, plugin binary, host, and aggregate DPR matrix cell. All three must agree on terminal verdict, gates, and identity; their metrics remain independently retained. SKIP/INCONCLUSIVE, absence from the portable scan cache, or a different format being inserted leaves the aggregate cell incomplete. Keep the file-handshake rule above; DPR evidence does not authorize screen-driving. Drive or ingest that cell through tools/scripts/gpu_dpr_runner.py; the DAW adapter must emit the exact format, binary, scan-cache, raw-sample, artifact, and machine receipts. Metrics must retain measured/derived/unavailable provenance, and the input expectation must come from the frozen scenario—not from the DAW producer that reports the observed event/hit. Two same-content image artifacts and numeric fidelity oracles replace producer-authored pass booleans. Timeout is INCONCLUSIVE and durable for resumption. In v2, the runner snapshots the AUv2/VST3/CLAP product executable and all eight cell artifacts before acceptance, binds the actual adapter producer PID plus product/fresh-process IDs, and rejects reused paths/inodes. A DAW receipt outside the runner root, a script presented as the product, or an identity document that merely claims the expected host/format cannot complete a cell. Finalization accepts only the runner directory and fixed terminal A2T/A3 receipts; follow docs/validation/gpu-dpr/README.md.
• REAPER license is a secret (~/.config/pulp/secrets/reaper-license.txt, personal non-commercial) — never commit, echo, or bake into a TartCI golden. Local Macs (m1/m3/m5) are already serialized; local is the primary lane.

GPU-health DAW evidence

For a product declaring gpu.health.read, bind the response to the exact DAW instance/publication and preserve the returned snapshot. A valid response is not automatically a passing startup trial. A visible DAW campaign must declare native-compositor-presentation and bind an independent compositor timestamp; capture completion cannot substitute. Missing source/shader or stage timings are instrumentation coverage rather than event loss and remain nullable only under the closed A3 verifier's passing no-change or failing investigation rules. Preserve the correlated A2T trace. Produce the seeded blank receipt with the focused real-Standalone test command in docs/validation/gpu-first-visible-a3-acceptance.md; merely setting PULP_GPU_HEALTH_SEED_BLANK_FRAME=1 does not bind a control artifact. Keep SKIP/INCONCLUSIVE and absent format scans as pending, never PASS.

Use reaper_smoke.py --mode editor-open --reaper-bin /exact/REAPER as the exact-format prerequisite for a DAW A3 adapter, then execute that adapter through gpu_first_visible_a3_campaign.py run-role --role daw. The smoke result cannot be copied into the campaign: the adapter still owes 10 real cold and 10 real warm editor lifecycles, independent native presentation timing, exact product and host artifacts, the same-instance trace, and durable cache provenance. Translate REAPER's --format au identity to the receipt's auv2; VST3 and CLAP retain their names. An adapter timeout, SKIP, or INCONCLUSIVE produces a nonterminal run.json, never a passing campaign.

Use the checked-in gpu_first_visible_a3_external_adapter.py as the immutable envelope and select gpu_first_visible_a3_reaper_producer.py. Configure its exact source-bound smoke/Lua helpers, REAPER, plugin binary/bundle, role driver, checked-in source-bound trace analyzer, embedded-build verifier, and digest-bound product build attestation/receipt. Also configure a reviewed source-build driver from the exact clean revision; it must reproduce the measured plugin executable and complete bundle in a fresh output without being given the measured path. The analyzer is a sealed prepared binary, and every REAPER lifecycle answers the producer's live-host nonce with exact executable and start identity; the driver must resolve to the reviewed file at the requested clean Pulp revision. The producer runs the qualified-format preflight and then requires the driver to perform all 20 editor opens, bind the trace to one challenged live REAPER PID, and terminate every reported REAPER PID. It retains and rehashes those inputs with the exact driver request/receipt in the host evidence tar. Missing automation, build provenance, analyzer replay, or native-present truth is nonterminal; the producer cannot promote the editor-open smoke or a screenshot completion into campaign evidence.

The measured plugin executable must carry the canonical embedded build marker, and the source-bound analyzer must preserve structural unverified separately from the campaign budget verdict. Before terminal A3, also run the four-state product trace-producer overhead protocol in the acceptance guide. Its exact pre-change, compile-out, compiled-in idle, and active measurements require zero xruns/audio-thread trace events; the DAW role campaign alone does not waive it.

For the DAW-selected overhead row, run the same reviewed REAPER lifecycle driver through gpu_first_visible_a3_trace_producer_overhead.py collect-state; do not time the smoke wrapper or relabel its editor-open result. Every one of the 55 samples/state must answer the collector while the challenged REAPER PID and exact plug-in executable are alive, then terminate cleanly. Active replay must find the health producer and bind the complete b4ba exact 20-signature state/render/js package from the A3 acceptance guide. gpu_acquire/gpu_submit/gpu_present are mandatory per sample; every other signature remains counted and a zero count is reported not-covered, not zero-cost. Require zero xruns and zero producer events on the audio TIDs. Pin the candidate-relative state_build_driver too. The collector exports and default-deny rebuilds the exact source row without the measured plug-in, ambient build tree, or network, then requires identical executable bytes and the requested tracing sentinel state. Keep its source/build/toolchain artifacts with the campaign; an already-built bundle or receipt assertion is not proof.

A3 v2 additionally binds each DAW campaign's sample provenance and exact trace bytes, then reruns the prepared analyzer and checks campaign, instance, process, GPU-evidence, trace-evidence, category, and completeness identity. A JSON sidecar alone is never terminal proof.