Deploy Prover

Interactive helper for deploying the Boundless prover stack to GPU servers.

What you can help with

1. New prover deployment — create inventory, deploy full stack (NVIDIA + Docker + bento)
2. Redeploy/update — update images, config, or git version on an existing server
3. Tear down — stop services and clean up
4. Troubleshoot — check service health, logs, GPU status
5. Configure — tune broker.toml settings (peak_prove_khz, concurrency, etc.)

Prerequisites

Before deploying, verify the user has:

1. Ansible — ansible --version (install: brew install ansible)
2. SSH access — ssh <user>@<host> "echo connected"
3. GPU server — Ubuntu 22.04 or 24.04 with NVIDIA GPU hardware

Deployment workflow

1. Create inventory file

Create ansible/inventory-<name>.yml with the server details. NEVER hardcode secrets — use env var lookups:

---
all:
  vars:
    ansible_user: ubuntu
    ansible_ssh_private_key_file: ~/.ssh/<key>
    # ansible_become_password: "{{ lookup('env', 'ANSIBLE_BECOME_PASSWORD') }}"  # uncomment if server needs sudo password
    prover_version: main  # or a specific branch/tag
  children:
    <group_name>:
      hosts:
        <ip_address>:
          prover_postgres_password: "<password>"
          prover_minio_root_pass: "<password>"
          prover_image_tag: "nightly-latest"
          prover_boundless_build: ""
          prover_rpc_urls: "{{ lookup('env', 'PROVER_RPC_URLS') }}"
          prover_private_key: "{{ lookup('env', 'PROVER_PRIVATE_KEY') }}"
          # prover_executor_count and max_concurrent_preflights default to 8 (matching broker-template.toml)
          prover_broker_toml_url: "https://raw.githubusercontent.com/boundless-xyz/boundless/refs/heads/main/broker-template.toml"
          # prover_broker_toml_local: "/path/to/custom/broker.toml"  # uncomment to use a local file instead

Ensure the inventory file is in .gitignore (ansible/inventory-*.yml pattern).

2. Set environment variables

The inventory uses {{ lookup('env', 'VAR') }} for secrets. These resolve on the machine running ansible-playbook, not on the remote server. They must be exported in the same shell that runs the playbook.

export PROVER_PRIVATE_KEY="<broker private key>"
export PROVER_RPC_URL_8453="<Base RPC URL>"
export PROVER_RPC_URL_167000="<Taiko RPC URL>"
ansible-playbook -i inventory-<name>.yml prover.yml  # must be in the same shell

3. Test connectivity

cd ansible
ansible all -i inventory-<name>.yml -m ping

4. Deploy

PROVER_PRIVATE_KEY="$PROVER_PRIVATE_KEY" \
PROVER_RPC_URLS="$PROVER_RPC_URLS" \
ansible-playbook -i inventory-<name>.yml prover.yml

This installs: NVIDIA drivers, Docker + NVIDIA Container Toolkit, then deploys the bento compose stack.

5. Verify

# Check all services are running
ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env ps"

# Check GPU is detected
ssh <user>@<host> "nvidia-smi --query-gpu=name,memory.total --format=csv,noheader"

# Check broker logs
ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env logs broker --tail 20"

# Check all images are correct
ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env ps --format '{{.Name}}\t{{.Image}}'"

Image modes

Before generating the inventory, ask the user which image mode they want. This matters because building from source is slow (Rust compilation) but required when deploying code from a feature branch that doesn't have CI-built images.

Ask: "Do you want to use pre-built images or build from source? If building from source, which components?"

Mode	Config	Description
Pre-built nightly	prover_image_tag: "nightly-latest"	Pull latest CI images from GHCR (recommended for main branch)
Pinned nightly SHA	prover_image_tag: "nightly-abc1234"	Pin to a specific CI build by commit SHA
Custom tag	prover_image_tag: "my-custom-tag"	Any tag from GHCR (e.g. from gh workflow run docker-services.yml -f custom_tag="my-custom-tag")
Pinned release	prover_image_version: "1.3"	Use release-tagged images (e.g. bento-v1.3, broker-v1.3)
Build all from source	prover_boundless_build: "all"	Build all Docker images on the server (slow, not recommended unless needed)
Selective build	prover_boundless_build: "broker"	Build specific service(s) from source, pull pre-built images for the rest

prover_image_tag takes precedence over prover_image_version. When set, ALL images use the exact same tag.

Selective build + pre-built (hybrid mode)

You can combine prover_image_tag with prover_boundless_build to build only specific services from source while pulling pre-built images for everything else. The justfile handles this by clearing the image tag for the built service at runtime (export BROKER_IMAGE="") so docker compose builds it from the local Dockerfile instead of pulling.

Example: deploy a feature branch that only changes the broker:

prover_image_tag: "nightly-latest"       # pre-built images for agents, rest_api, etc.
prover_boundless_build: "broker"          # build only the broker from the branch source

Valid prover_boundless_build values: "all", "broker", "rest_api", "exec_agent", "aux_agent", "gpu_prove_agent", "miner", or space-separated combinations (e.g. "broker rest_api").

Broker configuration

Ask the user: "Do you want to use an existing broker config from the repo, or specify your own config directory?"

Option A: Use an existing config. List the directories in ansible/roles/prover/configs/broker/ and let the user pick:

prover_broker_config_dir: "roles/prover/configs/broker/<chosen-config>"

Option B: Specify a custom directory. The user provides a path to a directory containing their own config files:

prover_broker_config_dir: "/path/to/my/broker-config"

Ansible copies the entire directory contents to /opt/bento/ on the remote server.

The config directory should contain:

• broker.toml — base config shared across all chains
• broker.{chain_id}.toml — per-chain overrides (e.g. broker.8453.toml, broker.167000.toml)

Per-chain override files must go in a chain-overrides/ subdirectory (e.g. chain-overrides/broker.8453.toml). This is the directory mounted into the Docker container.

IMPORTANT: broker.toml must include min_mcycle_price and max_collateral even if per-chain overrides replace them. These are required fields — the broker fails to parse the base config without them.

Fallback options (when prover_broker_config_dir is not set):

• prover_broker_toml_local — copies a single local file to /opt/bento/broker.toml
• prover_broker_toml_url — downloads from a URL (default: broker-template.toml from main branch)

Auto-generating config with the CLI wizard

The boundless CLI has a wizard that generates optimized configs:

# Run locally — auto-detects local hardware, or you can manually enter the server's specs
boundless prover generate-config --broker-toml-file ./broker-<host>.toml

# Then use it in the inventory:
# prover_broker_toml_local: "./broker-<host>.toml"

The wizard calculates peak_prove_khz, max_concurrent_preflights, max_exec_agents, and other values. If run on a machine without GPUs (e.g. your Mac), it will prompt you to enter the values manually.

Key settings to tune per GPU:

Setting	Default	Description
peak_prove_khz	100	Estimated GPU proving throughput. Too low = underutilizes GPU
max_concurrent_preflights	8	Should match prover_executor_count
max_concurrent_proofs	1	Number of concurrent GPU proofs

Selective deployment

Use --skip-tags to skip specific roles:

# Skip NVIDIA driver installation
ansible-playbook -i inventory-<name>.yml prover.yml --skip-tags nvidia

# Skip both NVIDIA and Docker (host already has them)
ansible-playbook -i inventory-<name>.yml prover.yml --skip-tags nvidia,docker

Note: --tags prover does NOT work as expected — the individual tasks inside the role are not tagged. Use --skip-tags instead. IMPORTANT: --skip-tags nvidia,docker will ALSO skip the prover role because it is tagged with both prover and docker. To skip only the NVIDIA and Docker install roles, use --skip-tags nvidia-install,docker-install. The NVIDIA/Docker roles are idempotent so running the full playbook on an already-configured host is safe and fast.

Tear down

# Stop all services
ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env down"

# Or stop via systemd
ssh <user>@<host> "sudo systemctl stop bento"

Troubleshooting

SSH permission denied

Check which SSH key the server expects:

ssh -v <user>@<host> 2>&1 | grep "Offering\|Accepted"

Add ansible_ssh_private_key_file to the inventory.

NVIDIA drivers not detected after deploy

Reboot the server: ssh <user>@<host> "sudo reboot"

Services crash-looping

Check logs: ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env logs --tail 50"

Postgres password mismatch after redeployment

If you change prover_postgres_password on a server that already has a running postgres, the old password persists in the Docker volume. Fix:

ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env down -v"
ssh <user>@<host> "sudo systemctl start bento"

Warning: -v deletes all volumes including broker DB and MinIO data.

Sudo password required

If ansible fails with "Missing sudo password", add to inventory vars:

ansible_become_password: "{{ lookup('env', 'ANSIBLE_BECOME_PASSWORD') }}"

Then export before running: export ANSIBLE_BECOME_PASSWORD="<password>"

Cloud providers (Latitude.sh, AWS) typically have passwordless sudo. Local/dev machines often don't.

Broker not picking up orders

1. Check RPC URL is correct and accessible
2. Check broker.toml has correct contract addresses
3. Check broker logs for error codes

Deploying a branch to staging

Two options for deploying a feature branch instead of main:

Option 1: Via CI pipeline (inventory override)

The pipeline inventory is in AWS Secrets Manager (l-prover-ansible-inventory, base64-encoded YAML). Override prover_version for staging hosts:

# Decode current inventory
aws secretsmanager get-secret-value --secret-id l-prover-ansible-inventory --region us-west-2 \
  --query SecretString --output text | base64 -D > inventory.yml

# Edit: set prover_version to your branch for staging hosts
# Then update the secret:
aws secretsmanager put-secret-value \
  --secret-id l-prover-ansible-inventory \
  --secret-string "$(base64 -i inventory.yml | tr -d '\n')" \
  --region us-west-2

Then run the pipeline (push to main or manual execution in CodePipeline).

To also use broker config from the branch, set prover_broker_toml_url to point at the branch's raw file.

Revert when done: set prover_version back to main, update the secret, re-run pipeline.

Option 2: Local ansible (no pipeline)

cd ansible
ansible-playbook -i inventory.yml prover.yml --limit staging -e prover_version=feature/my-branch

# With broker config from the same branch:
ansible-playbook -i inventory.yml prover.yml --limit staging \
  -e prover_version=feature/my-branch \
  -e "prover_broker_toml_url=https://raw.githubusercontent.com/boundless-xyz/boundless/refs/heads/feature/my-branch/broker-template.toml"

Skip NVIDIA/Docker (host already set up)

ansible-playbook -i inventory.yml prover.yml --limit staging --skip-tags nvidia,docker -e prover_version=feature/my-branch

Bento worker overflow (production burst fleet)

The bento-worker-overflow ASG in the production account is a manually-scaled GPU burst fleet.

# Check current state
aws-vault exec boundless-prod
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names bento-worker-overflow \
  --region us-west-2 \
  --query 'AutoScalingGroups[0].{Desired:DesiredCapacity,Min:MinSize,Max:MaxSize,Instances:length(Instances)}' \
  --output table

# Scale to N instances
aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name bento-worker-overflow \
  --desired-capacity 4 --region us-west-2

# Drain (scale to zero)
aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name bento-worker-overflow \
  --desired-capacity 0 --region us-west-2

When to scale

Queue Depth	Suggested Overflow
< 20	0 (primary handles it)
20-50	2-4
50-200	4-8
200+	8+ (check ASG max)

Scale down gradually (8 → 4 → 0) so in-flight proofs can finish. Wait until queue is stable/near-zero for 30+ minutes.

Check broker queue depth

LOG_GROUP="/boundless/bento/YOUR_HOSTNAME"
START=$(date -v-24H +%s)
END=$(date +%s)
Q=$(aws logs start-query --log-group-name "$LOG_GROUP" --start-time $START --end-time $END \
  --query-string 'fields @timestamp, @message | filter @message like /.* orders ready to be locked .*/ | sort @timestamp desc | limit 1' \
  --region us-west-2 --output text --query 'queryId')
until [ "$(aws logs get-query-results --query-id $Q --region us-west-2 --output text --query 'status')" = "Complete" ]; do sleep 1; done
aws logs get-query-results --query-id $Q --region us-west-2 --output json | jq -r '.results[0][] | select(.field == "@message") | .value'

Key references

What	Location
Pipeline definition	infra/pipelines/pipelines/l-prover-ansible.ts
Pipeline inventory	AWS Secrets Manager l-prover-ansible-inventory
Prover role	ansible/roles/prover/tasks/main.yml
Role defaults	ansible/roles/prover/defaults/main.yml
All variables	ansible/ENV_VARS.md
Inventory guide	ansible/INVENTORY.md

Supported providers

Tested with:

• Latitude.sh — bare metal and GPU VMs, ubuntu user, SSH key via their CLI (lsh)
• AWS — EC2 GPU instances
• Local dev machine — Ubuntu 24.04 with RTX 5090 (NVIDIA/Docker roles are idempotent and skip if already installed)

Any Ubuntu 22.04/24.04 server with NVIDIA GPU and SSH access will work.

IMPORTANT

• NEVER hardcode private keys or RPC URLs in inventory files — use {{ lookup('env', 'VAR') }}
• Ensure ansible/inventory-*.yml is in .gitignore
• The default broker-template.toml may not be optimal for your GPU — tune peak_prove_khz after deployment
• prover_executor_count and max_concurrent_preflights should match. Default role has 4 executors but broker-template.toml has 8 preflights — always override one or both