Remote compute over SSH

Run heavy work on the user's own machines over SSH with their own keys or their own interactive sign-in — you never install anything remote and never handle credentials yourself. A machine may be a plain server (CPU or GPU, no scheduler) or a Slurm cluster.

0 · Always run ssh through the app's config

Every ssh, scp and rsync below passes -F "$OPENSCIENCE_SSH_CONFIG". Write it as $SSHCFG after defining it once per command you send:

SSHCFG="${OPENSCIENCE_SSH_CONFIG:+-F $OPENSCIENCE_SSH_CONFIG}"

That config makes your connection reuse the one the app already authenticated, which is what lets clusters with two-factor authentication work at all: the user signs in once, and every command after that needs no password or code. Without it, such a host refuses every command you send.

When a host asks for credentials

Many institutional clusters require a password or a one-time code on EVERY new connection. If a command fails with a permission or authentication error (Permission denied, no supported authentication methods), do NOT retry, and do not report the failure yet: call the ssh_connect tool with that host. It asks the desktop UI to prompt the user, and returns once the shared connection is up — then retry your command unchanged.

Two errors that sign-in cannot fix, so do not call the tool for them: a host-key error (Host key verification failed — the user must verify the fingerprint in their own terminal once) and a network error (unreachable, timed out, unknown host).

1 · Pick the machine

1. cat .openscience/compute.json in the workspace (the app keeps this file in sync from the user's settings — read it directly; the directory is hidden). It looks like: {"machines":[{"host":"home-3090","label":"8x3090", "caps":{"cores":16,"mem_total_bytes":...,"gpus":["RTX 3090",...],"slurm":null}}]} The directory is hidden — read the file directly.
2. If the file is missing or has no machines, ask the user to add one in Settings → Remote compute, or give you a user@host. Do not guess.
3. Choose by the task's needs and each machine's caps: a GPU job → a machine whose caps.gpus is non-empty; a CPU job → any reachable machine. If several fit, or none clearly does, ask the user which to use.
4. Confirm it's reachable and check live headroom before launching: ssh $SSHCFG -o BatchMode=yes -o ConnectTimeout=8 <host> "nproc; free -h; nvidia-smi 2>/dev/null | head -15". On "Permission denied", the host wants credentials interactively: call the ssh_connect tool for it (see §0), then retry. Never send a password yourself and never disable host-key checking.

If the chosen machine's caps.slurm is set, use §2-Slurm. Otherwise use §2-Direct.

2-Direct · Run on a plain server (no Slurm)

Long jobs must outlive the SSH connection. Use a per-job dir + a fully detached process, mirroring how the app tracks runs.

1. Pick a job name and build the remote dir path (remember the literal string — shell variables do not survive between separate ssh calls): REMOTE=openscience/jobs/<name>-<YYYYmmdd-HHMMSS>
2. Create it and copy inputs (confirm with the user before copying > ~100 MB):

   ssh $SSHCFG -o BatchMode=yes <host> "mkdir -p <remote-dir>"
   scp $SSHCFG -o BatchMode=yes run.sh <input files> <host>:<remote-dir>/
   

   Write run.sh in the workspace first (so it is versioned in provenance); it should cd into the job dir and run the actual commands, e.g. use CUDA_VISIBLE_DEVICES to select GPUs. On a plain box the software environment is ambient (whatever is installed) — not declared anywhere — so pin it at run time by having run.sh write a manifest as its first step (it is fetched and recorded in §3–4). Keep it fail-safe so provenance never aborts the job:

   { python3 -V; echo "PLATFORM=$(uname -s)-$(uname -m)"; \
     echo '--- pip freeze ---'; python3 -m pip freeze; } > env.txt 2>&1 || true
   

3. Launch fully detached and capture the PID:

   ssh $SSHCFG -o BatchMode=yes <host> "cd <remote-dir> && \
     setsid bash -c 'bash run.sh >log 2>&1; echo \$? > exit_code' </dev/null >/dev/null 2>&1 & \
     echo \$! > pid; cat pid"
   

   Report the PID and the remote dir to the user.
4. Track:
   • Running? ssh $SSHCFG <host> "kill -0 \$(cat <remote-dir>/pid) 2>/dev/null && echo RUNNING || echo DONE".
   • Progress: ssh $SSHCFG <host> "tail -n 30 <remote-dir>/log"; GPU use: ssh $SSHCFG <host> "nvidia-smi".
   • Finished: ssh $SSHCFG <host> "cat <remote-dir>/exit_code" — 0 = success, other = failure. Do not assume success from an empty queue. When a run finishes you MUST complete §3 (fetch) and §4 (record) — every time, including a quick re-run or re-fetch. A run you don't record is invisible in Runs (neither the global view nor the session), so it may as well not exist.
   • Long jobs: report the PID + running state and stop; the user can ask you to check again later. Do not poll in a loop for more than ~2 minutes.
5. Cancel (only jobs you launched, or a PID/dir the user names): kill the whole process group so children die too: ssh $SSHCFG <host> "kill -- -\$(cat <remote-dir>/pid) 2>/dev/null || kill \$(cat <remote-dir>/pid)".

2-Slurm · Run on a Slurm cluster

Use this only when caps.slurm is set.

1. Write slurm/<job-name>.sbatch in the workspace:

   #!/bin/bash
   #SBATCH --job-name=<job-name>
   #SBATCH --output=slurm-%j.out
   #SBATCH --error=slurm-%j.err
   #SBATCH --time=01:00:00
   
   set -euo pipefail
   cd "$SLURM_SUBMIT_DIR"
   <the actual commands>
   

   Only add --partition/--gres/--mem/--cpus-per-task when the user asks or the cluster rejects the default. Load modules (module load …) the user names.
2. Submit:

   REMOTE=openscience/jobs/<job-name>-$(date +%Y%m%d-%H%M%S)
   ssh $SSHCFG -o BatchMode=yes <host> "mkdir -p $REMOTE"
   scp $SSHCFG -o BatchMode=yes slurm/<job-name>.sbatch <input files> <host>:$REMOTE/
   ssh $SSHCFG -o BatchMode=yes <host> "cd $REMOTE && sbatch <job-name>.sbatch"
   

   Parse Submitted batch job <id>; remember the literal remote dir.
3. Track: ssh $SSHCFG <host> "squeue -j <id> -h -o '%T %M'"; when it returns nothing, ssh $SSHCFG <host> "sacct -j <id> --format=State,Elapsed,ExitCode -n" or read slurm-<id>.out. Cancel: ssh $SSHCFG <host> "scancel <id>".

3 · Fetch results back

Copy every file the job produced into the workspace so they become traceable artifacts — not just a summary. That means log, env.txt, and each result/data/figure file the run wrote (e.g. result.json and trajectory.npz, checkpoints, plots). Each finished run MUST use a fresh, immutable local result directory; never fetch a rerun into a directory that was already recorded. A fetched artifact is the only thing that survives; anything left on the box is not provenance.

RESULT=results/<job-name>/<YYYYmmdd-HHMMSS>-<pid-or-job-id>
mkdir -p "$RESULT"
scp $SSHCFG -o BatchMode=yes "<host>:<remote-dir>/log" "<host>:<remote-dir>/env.txt" \
    "<host>:<remote-dir>/<each output file>" "$RESULT"/

<remote-dir> is the literal directory you created — name each file explicitly. List the job dir first (ssh $SSHCFG <host> "ls -la <remote-dir>") so you fetch them all. If you want a convenience "latest" copy, create it only after recording; the recorded --output paths must stay in the immutable run directory.

4 · Record the run (reproducibility) — REQUIRED, every time

Recording is not an optional finishing flourish: the app can't see the remote machine, so this call is the ONLY thing that makes the run exist in Runs. Do it after every finished run — first runs, quick re-runs, and recovered fetches alike, always using a fresh RESULT directory. Skipping it (a common mistake on a casual "just run it again") loses the run entirely. Record it completely — the helper pins whatever you pass:

• --code once per script that actually ran — the entry script and every helper it calls (e.g. run.sh and humanoid_sim.py), not just the wrapper.
• --output once per file you fetched in §3 — every result/data/figure, not just the summary json. These paths must be under that run's fresh RESULT directory; the helper refuses to record output paths used by earlier runs.
• --env-file the fetched env.txt, so the ambient interpreter + package versions are pinned (this is what makes an SSH run reproducible).
• --hardware = what the job used, not what the box has. A CPU-only job on a GPU box is "24 CPU cores, 62 GB (CPU-only)", not "2× RTX 3090".
• --session-id from the workspace marker, so the run attaches to this session (not just the global Runs view). Pass it verbatim as shown; it's empty-safe.

python "$XDG_CONFIG_HOME/opencode/skills/remote-compute/record_run.py" \
  --surface ssh --command "bash run.sh" --status <ok|failed> --host <host> \
  --hardware "<hardware the job used>" \
  --code run.sh --code <each other script> \
  --output "$RESULT"/<each output file> \
  --env-file "$RESULT"/env.txt \
  --session-id "$(cat .openscience/session.txt 2>/dev/null)"

The helper warns if a recorded file is missing or if code/outputs are empty — fix those rather than ignoring them. For a Slurm run use --surface hpc, --command "sbatch <name>.sbatch", --job-id <id>, and the sacct hardware/state; the environment there is pinned by the sbatch module load lines in the versioned script, so --env-file is not needed. Use --status failed on a non-success exit code / sacct state.

Summarize: the machine, the final state (quote the exit_code/sacct state — do not assume success), elapsed time, and the fetched files. If it failed, show the tail of log (or slurm-<id>.err) and propose a fix instead of silently rerunning.