Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Agent Observability: Traces, Costs and SLOs for LLM Agents

    1

    If you cannot answer "what did the agent see, decide and do on run X, and what did it cost?" within five minutes, the agent is not production-ready.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Agent Observability: Traces, Costs and SLOs for LLM Agents

    Agent Observability: Traces, Costs and SLOs for LLM Agents

    Example session with this skill installed

    We run three LLM agents in production on GCP and have no idea what they cost per task or why some runs take 5 minutes. Design the observability for them: what to trace, what to log, which metrics and alerts, and the BigQuery queries for a dashboard. Write it to observability.md.

    • Read your context and instructions
    • Compiled the agent observability

    observability.md: one trace per run (invoke_agent → chat and execute_tool spans with gen_ai.* attributes, served model, request id, cost); event log to BigQuery via a log sink with agent_version, hashed tenant and run_finished always emitted; metrics and alerts (success rate drop vs 7-day baseline, cost per successful run +30%, budget exhaustion > 2%, tool error rate > 5% over 15 min, cache hit ratio drop > 20 points after a deploy); content capture off by default, redacted with a 30-day retention where enabled; dashboard SQL for health, cost per successful run, cache hit ratio by version, tool reliability and the slowest runs.

    observability.md

    Connects securely to your tools. The creator never sees your data.

    About this skill

    If you cannot answer "what did the agent see, decide and do on run X, and what did it cost?" within five minutes, the agent is not production-ready. This skill gives your agent the telemetry to answer it.

    What your agent builds with it

    • One trace per run with spans for every model and tool call, using the OpenTelemetry GenAI semantic conventions; served model (not requested), provider request ids, policy decisions, cost; content capture treated as a privacy decision.
    • A structured event log (JSON Schema included) for cross-run questions, with an agent_version hash so every metric splits by release, hashed tenants and a guaranteed run_finished row.
    • Metrics, SLOs and alerts that matter: success rate, cost per successful run, budget-exhausted rate, refusal and fallback rate, tool error and timeout rates, policy denials, cache hit ratio, latency, provider errors.
    • A dashboard built from six ready BigQuery queries (health, cost per successful run with time-valid prices, cache effectiveness by version, tool reliability, runaway runs, served-model mix).
    • A cost report (cost_report.py) from call logs: cost per run and per successful run, cache hit ratio, p50/p95 latency, unpriced models, grouped by agent, model or day.
    • Closing the loop: weekly transcript review and post-deploy comparisons by version.

    Included: SKILL.md, event schema, example pricing file, BigQuery analytics SQL, cost_report.py (standard library), and an eval case.

    Part of the Production AI Agents series. Pair with Agent Evals (quality) and Agent Reliability (the loop that emits the events).

    The example pricing file is illustrative and dated; always use your provider's current prices. It does not score answer quality (see Agent Evals) and does not include a hosted dashboard, only the queries.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 day ago

    Requires

    Python 3.10+
    Python

    Permissions

    File Scopes

    Your
    project
    folder
    (reads
    exported
    call
    logs
    writes
    reports)

    cost_report.py reads local JSONL and pricing files and makes no network calls.

    Listed1 day ago

    What's inside

    Frequently Asked Questions

    No. It uses OpenTelemetry conventions, so any backend that accepts OTel traces works; the analytics SQL targets BigQuery but ports easily.

    Only if you choose to. The default is metadata-only spans; content capture is opt-in, redacted and time-limited.

    New in LLM Ops