Cloud Trace Querying Skill (cloud-trace-querying)
This skill equips the agent to interact with Cloud Trace by providing standard query patterns, diagnostic workflows for latency and errors, and tools to fetch and inspect trace hierarchies.
[!IMPORTANT]
AUTONOMOUS TOOL EXECUTION MANDATE:
You MUST execute the tool scripts directly in bash to query telemetry.
Do NOT just describe query strings or write Python scripts. Run the provided bash tools directly and report the exact results.
Prerequisites & Setup
- Google Cloud SDK: Verify
gcloud is installed (Installation Guide).
- Authentication: Authenticate CLI and application default credentials:
gcloud auth login && gcloud auth application-default login
- Project & Billing: Identify the target
<PROJECT_ID> (from the user's prompt or via bash scripts/get_trace_project/run.sh), ensure an active billing account is attached, and configure the project if not already set:
gcloud config set project <PROJECT_ID>
- API Enablement: Verify Cloud Trace and Cloud Logging APIs are enabled:
gcloud services enable cloudtrace.googleapis.com logging.googleapis.com
Quickstart Workflows
1. Identify Google Cloud Project ID
If the user specified a Google Cloud project ID in their prompt, use it.
If they did not, ALWAYS determine the Google Cloud project ID by running:
bash scripts/get_trace_project/run.sh
2. Search Traces (Latency & Errors)
To find traces matching specific criteria, such as a root span of /v1/booking/book with a duration of at least 1.5s:
bash scripts/search_traces/run.sh --filter 'root:/v1/booking/book latency:1.5s' --projects '<PROJECT_ID>'
Note: --filter is required. To list traces without specific filtering, pass --filter '' or --filter 'root:'.
Filter syntax: The Cloud Trace API uses latency:<duration> (such as latency:1.5s for duration >= 1.5s), which corresponds to "Span duration" in Trace Explorer.
The script outputs JSON where each cluster's examples array contains sample trace objects with trace_id, duration_seconds, span count, and error status.
3. Fetch Entire Trace Hierarchy
To view all spans and status codes/labels within a trace:
bash scripts/fetch_entire_trace/run.sh --trace-id '<TRACE_ID>' --project '<PROJECT_ID>'
4. Fetch Correlated Application Logs
To retrieve application log entries correlated with a trace ID:
bash scripts/fetch_related_logs/run.sh --trace-id '<TRACE_ID>' --trace-projects '<PROJECT_ID>'
Application log entries are useful when inspecting stack traces, exception messages, and log events emitted during span execution. When diagnosing errors or service crashes, inspect the returned log entries for severity: "ERROR" to confirm root causes or uncover unhandled exceptions.
Note: fetch_related_logs queries the default log scope (_Default), searching across configured log views in the target project. An empty response can occur if: (1) no log entries match the criteria, (2) the caller lacks IAM permissions to view the matching entries, or (3) matching entries reside in log views outside the searched log scope.
5. Fetch Specific Trace Span
To inspect attributes and metadata of a single span within a trace:
bash scripts/fetch_trace_span/run.sh --trace-id '<TRACE_ID>' --span-id '<SPAN_ID>' --project '<PROJECT_ID>'
6. Generate Cloud Console Deep Links
To generate direct Cloud Console URLs for browser inspection of traces or correlated logs:
bash scripts/generate_links/run.sh --trace-id '<TRACE_ID>' --project '<PROJECT_ID>' [--span-id '<SPAN_ID>']
7. Format Output
- Terminal / CLI: Render span trees as indented ASCII timelines using
|--.
- UI Presentation: Render as A2UI JSON graph structures when requested for UI.
8. Upstream Service Error Handling
If the Cloud Trace API or Cloud Logging API returns an upstream HTTP service error (such as 500 Internal Server Error, 503 Service Unavailable, or GoogleAPICallError), report the upstream service outage directly to the user. Do not attempt to inspect virtual environments, modify internal scripts, or debug local libraries.
Reference Documentation
Operational Workflows
Query Syntax & Tools
Semantic Conventions
Concepts & Architecture
Templates & Assets
Official Documentation Links