Run the Omnigent load test
Drives dev/loadtest/ end to end: collect inputs → run → read summary.md →
explain the latencies. Each Locust user is a real omnigent host that
registers over the host tunnel, creates host-bound sessions, and drives real
multi-turn conversations — every turn is a genuine post→idle loop through the
host's runner, with the LLM mocked (zero latency) so the numbers are
Omnigent's own overhead. -u N scales the number of hosts.
It boots its own local stack (server + mock LLM), so there is no server to
point at, and it runs from a repo checkout only. For single-request latency
micro-benchmarks (not concurrency), that is a different tool: dev/benchmarks/.
1. Ensure deps (repo checkout)
uv sync --extra loadtest --extra agents-sdk
Run with that same interpreter (e.g. .venv/bin/python), from the repo root.
2. Gather inputs
Ask the user (AskUserQuestion when several are unknown); all have defaults.
| Input | Flag | Default | Notes |
|---|
| Hosts | --users | 4 | Concurrent hosts (N) — the main scale knob. |
| Spawn rate | --spawn-rate | 1 | Hosts started per second. |
| Run time | --run-time | 120s | 40s / 5m / 1h. |
| Sessions/host | --sessions-per-user | 2 | Host-bound sessions each host drives. |
| Turns/session | --turns-per-session | 4 | Turns per session — history grows across them. |
| Reply length | --reply-words | 60 | Words in the mocked (streamed) reply per turn. |
Capacity caveat — say this to the user if they ask for large N: turns run on
real host + runner subprocesses, so N hosts × M sessions = N×M runner processes
on this box. It is capacity-limited by design (real turns, not faked). Start at
--users 2 --sessions-per-user 1 --turns-per-session 2 --run-time 40s to confirm
the stack boots (~10-30s), then ramp to a few dozen hosts at most. At high N the
load box saturates before the server (Locust warns about CPU).
3. Run
python dev/loadtest/run.py \
--users <N> --spawn-rate <R> --run-time <T> \
--sessions-per-user <S> --turns-per-session <TU>
It boots the stack, prints the server URL + registered agent, runs Locust, and
writes dev/loadtest/results/omnigent_load_test-<timestamp>/.
4. Read and explain
Read the summary.md and relay it. Focus on:
- Outcome / failures first. Exit 0 + 0 failures = PASS. Non-zero failures are
the headline — check
console.log and, for a host that failed to register,
the per-host results/.../host-workspaces/<name>/host.log. At high N, failures
usually mean the load box saturated, not the server.
- turn — the headline latency: one full post→idle agent turn on a host's
runner (mocked LLM), so it is Omnigent's per-turn overhead. It grows across a
conversation as history accumulates, so a rising p95/p99 with larger
--turns-per-session is expected and is the interesting signal.
- host online — host tunnel registration cost; session create — the
host-bound create; Ops/s — aggregate throughput at this concurrency.
If failures appeared or the tail looks high, suggest a concrete next step (lower
N if the load box is saturated, raise --turns-per-session to study history
growth, lengthen --run-time for steady state, or check server logs/metrics).
Notes
- Scenario file:
dev/loadtest/omnigent_load_test.py; driver + report:
dev/loadtest/run.py. Full reference: dev/loadtest/README.md.