Works with the AI Agents you already use
LLM App Tracing and Agent Observability with OpenTelemetry
Add OpenTelemetry GenAI tracing to an LLM app or agent, with redaction, sampling, dashboard spec, five alerts and a trace check. Verdict OBSERVABLE / PARTIAL / BLIND.
$7
LLM App Tracing and Agent Observability with OpenTelemetry
Example session with this skill installed
Our FastAPI support agent uses the Anthropic SDK and LangGraph with two tools, and we have zero tracing. Add observability that sends to Langfuse EU, keep customer emails out of the traces, and tell me what alerts to set.
- Read your context and instructions
- Compiled the llm app tracing
Observability Report: support-agent
Defaults used: content capture OFF in prod, 100% sampling (about 3k traces/day), user IDs hashed, cost unpriced until you add a price table.
Stack: Python 3.12 · FastAPI · anthropic SDK · LangGraph · existing tracing: logging.info only
Verdict: BLIND -> PARTIAL
Before: there were no traces, and app/agent.py:88 logged full prompts with customer emails. After: LLM calls, tools and the agent loop are nested spans sent to Langfuse EU. Verification is still NOT RUN in staging.
Capability scorecard
| # | Capability | Before | After | Evidence |
|---|---|---|---|---|
| 1 | Nested trace per request | missing | present | app/otel.py:12 |
| 2-3 | LLM spans, tokens, latency | missing | present | app/llm.py:31 |
| 4 | Errors | missing | present | app/llm.py:44 |
| 5 | Tool spans | missing | present | app/tools.py:19 |
| 8 | Session / user IDs | missing | present (baggage) | app/main.py:27 |
| 9 | Redaction | raw emails logged | regex scrub + capture off | app/redact.py |
| 10 | Feedback scores | missing | missing | TODO: /feedback route |
Changes made
with tracer.start_as_current_span(f"execute_tool {name}") as t:
t.set_attributes({"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": name})
Export config
OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64 pk:sk>,x-langfuse-ingestion-version=4"
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
Starter alerts
| Alert | Condition | Suggested start |
|---|---|---|
| p95 latency | > <SET_ME>s, 15m | 2x current p95 |
| Error rate | > <SET_ME>%, 10m | 5% |
| Daily cost | > $<SET_ME> | 1.5x 7-day avg |
| Tool failures | > <SET_ME>%, 30m | 10% |
| Token spike | p95 output > <SET_ME> | 2x 7-day p95 |
Verification (to run)
make dev, then send "Where is order 1042?" and expect invoke_agent > chat > execute_tool lookup_order > chat, with a session.id on every span.
Connects securely to your tools. The creator never sees your data.
About this skill
Most LLM apps go to production with print statements and a vendor dashboard that shows only the model calls. When an agent loops, a tool fails without an error, or costs jump, nobody can see which step did it. And prompts full of customer data often end up in logs.
This skill has your coding agent audit and instrument the app:
- Detects the stack: language, framework, LLM SDKs (OpenAI, Anthropic, Google, Vercel AI SDK), agent framework, and any tracing you already have
- Scores 11 observability capabilities with file:line evidence
- Adds OpenTelemetry spans that follow the GenAI semantic conventions for LLM calls, tool calls, retrieval steps and agent steps, with tokens, latency, errors, finish reasons, session IDs and hashed user IDs
- Leaves content capture off by default and adds regex redaction for emails, card numbers, SSNs, phone numbers and API keys, plus a Collector backstop and tail sampling that keeps every error trace
- Gives export settings for Langfuse, Arize Phoenix, LangSmith, Datadog LLM Observability and Honeycomb over OTLP
- Writes a dashboard spec (latency p50/p95, error rate, estimated cost per day, tokens by model, tool failure rate) and five starter alerts with thresholds you set
- Runs one request and checks the trace tree before calling the app observable
Who it's for: developers and platform engineers shipping LLM features, RAG pipelines or agents who want vendor-neutral tracing instead of lock-in.
Honesty note: the GenAI conventions are still Development status in OpenTelemetry, and the skill says so and sets the opt-in flag. It never invents prices, latencies or package versions. Cost is labeled as an estimate built from a price table you supply. If it could not run a real request, verification is reported as NOT RUN. It does not write eval suites or forecast spend.
OpenTelemetry GenAI semantic conventions are still Development status, so attribute names may change. Backend endpoints, headers and instrumentation package versions were checked in October 2026 and should be re-verified. Cost is an estimate from your own price table. The skill writes a dashboard and alert spec but does not create them inside your backend. Verification needs the app to be runnable by the agent.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago
Requires