Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    LLM App Tracing and Agent Observability with OpenTelemetry

    1

    Add OpenTelemetry GenAI tracing to an LLM app or agent, with redaction, sampling, dashboard spec, five alerts and a trace check. Verdict OBSERVABLE / PARTIAL / BLIND.

    $7

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    LLM App Tracing and Agent Observability with OpenTelemetry

    LLM App Tracing and Agent Observability with OpenTelemetry

    Example session with this skill installed

    Our FastAPI support agent uses the Anthropic SDK and LangGraph with two tools, and we have zero tracing. Add observability that sends to Langfuse EU, keep customer emails out of the traces, and tell me what alerts to set.

    • Read your context and instructions
    • Compiled the llm app tracing

    Observability Report: support-agent

    Defaults used: content capture OFF in prod, 100% sampling (about 3k traces/day), user IDs hashed, cost unpriced until you add a price table.
    Stack: Python 3.12 · FastAPI · anthropic SDK · LangGraph · existing tracing: logging.info only

    Verdict: BLIND -> PARTIAL

    Before: there were no traces, and app/agent.py:88 logged full prompts with customer emails. After: LLM calls, tools and the agent loop are nested spans sent to Langfuse EU. Verification is still NOT RUN in staging.

    Capability scorecard

    #CapabilityBeforeAfterEvidence
    1Nested trace per requestmissingpresentapp/otel.py:12
    2-3LLM spans, tokens, latencymissingpresentapp/llm.py:31
    4Errorsmissingpresentapp/llm.py:44
    5Tool spansmissingpresentapp/tools.py:19
    8Session / user IDsmissingpresent (baggage)app/main.py:27
    9Redactionraw emails loggedregex scrub + capture offapp/redact.py
    10Feedback scoresmissingmissingTODO: /feedback route

    Changes made

    with tracer.start_as_current_span(f"execute_tool {name}") as t:
        t.set_attributes({"gen_ai.operation.name": "execute_tool",
                          "gen_ai.tool.name": name})
    

    Export config

    OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel
    OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64 pk:sk>,x-langfuse-ingestion-version=4"
    OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
    

    Starter alerts

    AlertConditionSuggested start
    p95 latency> <SET_ME>s, 15m2x current p95
    Error rate> <SET_ME>%, 10m5%
    Daily cost> $<SET_ME>1.5x 7-day avg
    Tool failures> <SET_ME>%, 30m10%
    Token spikep95 output > <SET_ME>2x 7-day p95

    Verification (to run)

    make dev, then send "Where is order 1042?" and expect invoke_agent > chat > execute_tool lookup_order > chat, with a session.id on every span.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most LLM apps go to production with print statements and a vendor dashboard that shows only the model calls. When an agent loops, a tool fails without an error, or costs jump, nobody can see which step did it. And prompts full of customer data often end up in logs.

    This skill has your coding agent audit and instrument the app:

    • Detects the stack: language, framework, LLM SDKs (OpenAI, Anthropic, Google, Vercel AI SDK), agent framework, and any tracing you already have
    • Scores 11 observability capabilities with file:line evidence
    • Adds OpenTelemetry spans that follow the GenAI semantic conventions for LLM calls, tool calls, retrieval steps and agent steps, with tokens, latency, errors, finish reasons, session IDs and hashed user IDs
    • Leaves content capture off by default and adds regex redaction for emails, card numbers, SSNs, phone numbers and API keys, plus a Collector backstop and tail sampling that keeps every error trace
    • Gives export settings for Langfuse, Arize Phoenix, LangSmith, Datadog LLM Observability and Honeycomb over OTLP
    • Writes a dashboard spec (latency p50/p95, error rate, estimated cost per day, tokens by model, tool failure rate) and five starter alerts with thresholds you set
    • Runs one request and checks the trace tree before calling the app observable

    Who it's for: developers and platform engineers shipping LLM features, RAG pipelines or agents who want vendor-neutral tracing instead of lock-in.

    Honesty note: the GenAI conventions are still Development status in OpenTelemetry, and the skill says so and sets the opt-in flag. It never invents prices, latencies or package versions. Cost is labeled as an estimate built from a price table you supply. If it could not run a real request, verification is reported as NOT RUN. It does not write eval suites or forecast spend.

    OpenTelemetry GenAI semantic conventions are still Development status, so attribute names may change. Backend endpoints, headers and instrumentation package versions were checked in October 2026 and should be re-verified. Cost is an estimate from your own price table. The skill writes a dashboard and alert spec but does not create them inside your backend. Verification needs the app to be runnable by the agent.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Requires

    Python 3 or Node.js
    OTLP-compatible tracing backend
    Python

    Permissions

    Terminal / Shell
    Environment Variables
    Listed1 month ago

    What's inside

    Frequently Asked Questions

    A SKILL.md with the audit steps, Python and TypeScript instrumentation code, redaction and sampling defaults, a dashboard spec, five alert templates and the verdict rules. It also includes references/genai-otel-facts.md, which lists GenAI attribute names, spec status, backend OTLP endpoints and headers, and sources checked as of October 2026.

    Any agent that reads SKILL.md files, such as Claude Code, Codex CLI, Cursor, VS Code Copilot and Gemini CLI. It works best when the agent can read and edit your repo and run your app.

    Unzip the folder into your agent's skills directory (for example .claude/skills/) and ask your agent to add tracing or observability to your app. It asks once for your backend, content capture preference and price table, then uses stated defaults.

    The OpenTelemetry GenAI conventions are still in Development, so attribute names can change. Backend endpoints and package versions should be re-checked against current docs. It cannot build dashboards inside your backend for you, and cost figures are only as good as the price table you give it.

    It follows a fixed audit with evidence, uses checked attribute names and backend headers instead of guessed ones, turns redaction and error-keeping sampling on by default, and will not call the app observable until a real trace has been seen. It is separate from eval writing and cost modeling.

    They are still marked Development, so attribute names can change. The reference file records the names and status as of October 2026 with sources, and the skill tells your agent to check the current spec and your backend docs when it has web access before writing attribute names.

    New in LLM Ops