More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Phoenix: See What Your LLM App Is Actually Doing

    1

    Observe LLM apps with Arize Phoenix — OpenTelemetry traces, dataset evals, prompt/model experiments, and production monitoring that catches drift before users do.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Phoenix: See What Your LLM App Is Actually Doing

    Phoenix: See What Your LLM App Is Actually Doing

    Example session with this skill installed

    Users say our support bot is making things up. Set up Phoenix tracing and evals to find whether it is retrieval, the model, or the prompt — and stop it recurring.

    • Read your context and instructions
    • Compiled the phoenix

    Launched Phoenix locally with px.launch_app() for interactive debugging, then registered a tracer provider and instrumented the OpenAI client and the retriever — two calls, and from that point every LLM call is captured. Opened the prod project, filtered traces by low user-feedback score, and picked a bad one. The trace showed the retriever span returned 3 chunks in 40ms — but the top chunk was from an outdated policy doc. The LLM span showed the model correctly cited the stale chunk. Root cause: retrieval, not generation. Without the trace, the team would have blamed the model and burned a week on prompt tuning. Fixed the document version in the index and added an eval that catches stale citations. Built a dataset of 50–200 representative inputs, versioned it, and ran prompt variants side by side in experiments — code-based checks first (exact match, JSON validity), LLM judges for what code cannot check (faithfulness, tone). Re-running the dataset on every prompt change became the regression suite. In production, judge scores are sampled on a fraction of traffic, embedding-space views watch for query-topic drift, and alerts fire on latency p99, error rate, and judge-score drops — because averages hide the failures and percentiles find them.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    LLM apps fail in ways unit tests cannot catch: the prompt drifted, retrieval returned junk, the model quietly got worse. This skill is the flight recorder: OpenTelemetry tracing where every request is captured with inputs, outputs, token counts, and latency; dataset-based evaluations that replace anecdotes with versioned experiments; LLM-as-judge evaluators plus code-based checks; and production monitoring that watches quality signals, input drift, and percentiles over time. Self-hosted and open source.

    It is self-hosted: if you want managed hosting with zero ops, LangSmith or Arize Cloud is the answer.
    It observes serving behavior, not training runs — loss curves and hyperparameters belong to Weights & Biases or MLflow.
    It is not a model registry or lifecycle governance tool; MLflow owns that job.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    No special permissions declared or detected

    Listed3 days ago

    What's inside

    Frequently Asked Questions

    Read the trace first — start at the LLM span's input, what the model was actually told, before judging its output. Most 'hallucinations' are retrieval failures wearing a generation costume.

    Code-based checks first: fast, deterministic, free. LLM judges for what code cannot check — tone, faithfulness, nuance.

    Production. Tracing in prod is drift detection: quality signals over time, input shifts, and judge-score drops that dev never sees.

    New in LLM Ops