Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Tracegrade

    1

    Turn production agent traces into a calibrated eval suite: failure clusters, ready assertions, and a judge checked against humans.

    $19

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Tracegrade

    Tracegrade

    Example session with this skill installed

    Our support agent has a 20% failure rate. I exported 27 production traces to traces.jsonl. Build me an eval suite I can put in CI, and tell me whether our judge is trustworthy.

    • Read your context and instructions
    • Compiled the tracegrade

    Eval suite from 27 production traces

    Analysis: 27 traces · 22 failing (81.5%) · 13 failure modes · 16 cases

    Failure modes found

    ClusterTraces
    Tool failure: couldn, right, now4
    Wrong answer: return, days, ago4
    Refusal: order, already, gave2
    Schema violation: order, total, return2
    Unknown tool: order, couldn, find2
    Empty answer1
    Skipped the tool1
    Too many steps1
    Tool loop1
    Token overrun1
    Latency overrun1

    Sample cases

    IDCaseAssertions
    case-001Tool failure — Where is order A-1042?no_tool_error, answer_matches
    case-003Wrong answer — Can I return order A-1042? It arrived 40 days ago.answer_matches
    case-007Schema violation — Return the order summary as JSON with id, status and total.valid_json
    case-013Skipped the tool — Can I change the delivery address on A-1042?must_call_tool

    Each case carries the input, the expected behaviour, and a rubric built from its failure mode.

    Judge calibration

    16 cases scored · agreement 87.5% · Cohen's kappa 0.600

    Moderate agreement. Usable, but re-read the disagreements before gating.

    judge passjudge fail
    human pass22
    human fail012

    Two disagreements: one human accepted an escalation the judge punished, one human accepted a slow but successful email. Both are rubric problems, not model problems — exactly what calibration is for.

    CI gate

    tracegrade gate baseline.json current.json --max-regression 0.05
    

    Exit code 1 when the failure rate regresses or a new failure mode appears.


    Worked example: the traces are fictional, the output is real — generated by running Tracegrade on the bundled dataset.

    tracegrade gate baseline.json current.json

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Tracegrade turns production agent traces into a calibrated eval suite: failure modes, assertions, rubrics and a judge checked against human labels.

    Use this skill when:

    • turn traces into evals
    • grade agent runs
    • build a failure taxonomy
    • gate deploys on quality

    Who it's for: Teams running agents in production.

    Example request

    Turn last week's failing traces into an eval suite.

    What the example returns

    • 11 failure modes detected
    • Cases with assertions and rubrics
    • Judge vs humans: kappa 0.78

    In the package: five files — SKILL.md with the trigger and workflow, README.md, references/METHOD.md (the full method), references/CHECKLIST.md (the pass/fail gate) and examples/EXAMPLE.md (a worked run you can copy).

    Works with Claude Code, Cursor, Codex CLI and any agent that reads SKILL.md. Plain markdown: no API key, no account, no network access.

    Tracegrade

    Requires Node 18.17+. Requires existing JSONL traces with input/output fields. Does not perform LLM calls; user must provide judge verdicts for calibration.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 2 days ago

    Requires

    GitHub
    Node.js

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Network Access

    Reads local trace files only. No network access, no credentials requested, and no files modified outside the output directory.

    Listed3 days ago
    Updated2 days ago

    What's inside

    Frequently Asked Questions

    Tracegrade analyzes your raw production logs to identify real-world failure patterns, transforming them into a structured regression suite with specific rubrics and assertions that reflect how your agent actually fails in the wild.

    This skill is framework-agnostic and works with any AI agent that can export traces (inputs, outputs, and tool steps) to a JSONL format, regardless of whether you use LangChain, AutoGPT, or a custom stack.

    Tracegrade uses Cohen’s kappa to measure the agreement between your human labels and the LLM judge's scores, ensuring your automated evaluation pipeline is statistically reliable before you trust it to gate your code.

    You receive the Tracegrade CLI tool designed for Node.js environments, along with documentation on how to import JSONL datasets, generate failure clusters, and export calibrated eval suites for your CI/CD pipeline.

    No, Tracegrade is an analysis and testing tool that requires existing production or staging traces to function; it is designed to help you learn from real failures rather than inventing synthetic ones.

    The tool requires Node.js version 18.17 or higher and operates locally on your datasets, making it easy to integrate into existing automated testing workflows and CI pipelines.

    New in LLM Ops