Agent Observability
You instrument the user's product agents — not the coding agent — so every run leaves a trace that can be inspected, scored, and learned from. You explain everything in plain language: the user is non-technical.
Plain-Language Primer (teach this first, once)
- A trace is the full story of one request: every model call, tool call, and retrieval step, in order, with timing, tokens, and cost attached.
- A span is one step inside that story (one LLM call, one tool call).
- Observability means you can answer "why did this output happen?" and "what did this run cost?" after the fact, without guessing.
- The backend is a separate managed service. Your product sends trace data to it over HTTPS as a side effect of running. You do NOT host it yourself in the common case, and it never runs on your laptop.
- Why it matters: an agent can return a confident wrong answer with HTTP 200. Without traces you cannot debug it, cannot measure quality, and cannot run any learning loop (
runtime-learning-loop hard-fails without this).
Hard Rules
Never log secrets, API keys, or user PII into trace payloads — configure redaction/masking before production traffic, not after.
Never propose self-hosting on the user's machine. Managed free tier first; self-host-in-cloud is a clearly-labeled last resort.
Always instrument against OpenTelemetry GenAI conventions (gen_ai.*) or OpenInference — never a proprietary-only SDK — so the backend can be swapped later by config, not a rewrite.
Always verify the first trace end-to-end (run once, open the trace, see spans + token counts) before calling the setup done.
Always set a sampling/cost policy before production scale — keep failures at 100%, sample successes.
Workflow
Step 1 — Scope
Identify: which agent flows matter most (start with ONE), expected volume/month, whether data is sensitive (PII/health/finance), and which framework the product uses (Google ADK, LangGraph/LangChain, OpenAI/Anthropic SDK direct, custom).
Step 2 — Choose a backend
Read references/backends.md (decision table, free tiers, tradeoffs — verify current pricing at decision time; tiers change). Defaults: Langfuse Cloud free tier for general use; Phoenix for agent-heavy/ADK products (deepest multi-step agent views); LangSmith only if the product is LangChain/LangGraph-first. Sensitive data → check the backend's redaction options FIRST; most redact nothing by default.
Step 3 — Instrument
Prefer auto-instrumentation for the detected framework (one register()/instrument() call + env vars for keys/endpoint). Put API keys in environment variables — never in code. Wrap only the entry point; let the instrumentor capture nested spans. Tag spans with user_id/session_id/feature name so cost and quality can be sliced later.
Step 4 — Verify first trace
Run one real request. Open the backend UI. Confirm: full span tree visible, prompts/completions captured, token counts + cost present, no secrets/PII in payloads. If any check fails, fix before proceeding.
Step 5 — Cost and sampling policy
Async exporters add negligible latency; the real costs are storage and noise. Policy: 100% of failed runs, sampled successes (start 10–25% at scale), retention matched to the free tier's window. Set a monthly span budget aligned to the free-tier cap.
Step 6 — Wire to quality loops
Point eval-pipeline at production traces (score samples with eval-judge); this trace+eval pair is the precondition runtime-learning-loop checks. Record backend choice + config in docs/observability/OBSERVABILITY.md and log it to docs/skill-outputs/SKILL-OUTPUTS.md.
Output Format
=== Observability Plan: [product] ===
Flow instrumented: [flow] | Framework: [ADK|LangGraph|SDK|custom]
Backend: [name + tier] — why: [1 line] | Est. volume: [spans/mo] vs free cap: [cap]
Redaction: [what is masked] | Sampling: [failures 100%, successes N%]
First trace verified: [yes/no] | Saved: docs/observability/OBSERVABILITY.md
Next: eval-pipeline wiring → runtime-learning-loop precondition met
Gotchas
- Traces are for the shipped product;
run-trace is for the coding agent. Different data, different store — never mix them.
- SaaS backends receive your prompts. That is user data leaving your infrastructure — check redaction defaults (usually OFF) before pointing production at them.
- Instrument to the standard, not the vendor. All major backends ingest OTel GenAI
gen_ai.* spans; the backend is a 2-year decision, the instrumentation is forever.
- Free-tier caps are per month and small (e.g. tens of thousands of spans). One agent request can emit 5–30 spans. Do the multiplication before promising "free".
- A trace that omits tool calls is half a trace. Multi-agent debugging lives in the tool/handoff spans, not the final completion.
Example
<examples>
<example>
<input>My appeal-drafting app gives weird answers sometimes and I have no idea why. Add observability. It's Google ADK + FastAPI on Cloud Run.</input>
<output>
Primer given (traces/spans/backend in plain language).
Scope: draft-appeal flow, ~3k requests/mo (~30k spans), synthetic PHI only — still mask document text.
Backend: Phoenix (deepest agent-trace views; OpenInference auto-instrumentation for ADK) — cloud free tier fits 30k spans/mo.
Instrumented via openinference-instrumentation for ADK + env-var keys. First trace verified: 14 spans, tokens + cost visible, document text masked.
Sampling: failures 100%, successes 25%. Saved docs/observability/OBSERVABILITY.md. Logged to SKILL-OUTPUTS.
Next: wire eval-pipeline to score 20 sampled traces/week.
</output>
</example>
</examples>
Read references/examples.md for full walkthroughs (Langfuse path, redaction setup, backend swap).
Common Rationalizations
| Excuse | Reality |
|---|
| "Add observability later, ship first" | You cannot debug the past — traces only exist from instrumentation day forward. |
| "Logs are enough" | Logs show 200 OK; traces show the wrong retrieval that caused the bad answer. |
| "Self-host to save money" | Ops burden (DB + Redis + storage) exceeds free-tier value at this scale; laptop hosting is banned. |
| "Pick the backend by brand" | Pick by framework fit + free-tier math + redaction needs; instrumentation stays portable. |
| "Trace everything forever" | Cost and noise explode; sample successes, keep failures, match retention to need. |
Verification
Red Flags
- Backend chosen before checking framework auto-instrumentation support
- Production traffic flowing with default (empty) redaction config
- Proprietary SDK wrapped around every call site instead of one instrumentor
- "Observability done" claimed without opening a single real trace
- Coding-agent activity and product traces mixed in one store
Impact Report
Observability set up for: [product/flow]
Backend: [name + tier] | Framework instrumented: [name]
First trace verified: [yes/no] | Redaction: [configured/not needed]
Sampling: [policy] | Files: docs/observability/OBSERVABILITY.md
Ready for: eval-pipeline wiring, runtime-learning-loop precondition