Works with the AI Agents you already use
Tracegrade
Turn production agent traces into a calibrated eval suite: failure clusters, ready assertions, and a judge checked against humans.
$19
Tracegrade
Example session with this skill installed
Our support agent has a 20% failure rate. I exported 27 production traces to traces.jsonl. Build me an eval suite I can put in CI, and tell me whether our judge is trustworthy.
- Read your context and instructions
- Compiled the tracegrade
Eval suite from 27 production traces
Analysis: 27 traces · 22 failing (81.5%) · 13 failure modes · 16 cases
Failure modes found
| Cluster | Traces |
|---|---|
| Tool failure: couldn, right, now | 4 |
| Wrong answer: return, days, ago | 4 |
| Refusal: order, already, gave | 2 |
| Schema violation: order, total, return | 2 |
| Unknown tool: order, couldn, find | 2 |
| Empty answer | 1 |
| Skipped the tool | 1 |
| Too many steps | 1 |
| Tool loop | 1 |
| Token overrun | 1 |
| Latency overrun | 1 |
Sample cases
| ID | Case | Assertions |
|---|---|---|
| case-001 | Tool failure — Where is order A-1042? | no_tool_error, answer_matches |
| case-003 | Wrong answer — Can I return order A-1042? It arrived 40 days ago. | answer_matches |
| case-007 | Schema violation — Return the order summary as JSON with id, status and total. | valid_json |
| case-013 | Skipped the tool — Can I change the delivery address on A-1042? | must_call_tool |
Each case carries the input, the expected behaviour, and a rubric built from its failure mode.
Judge calibration
16 cases scored · agreement 87.5% · Cohen's kappa 0.600
Moderate agreement. Usable, but re-read the disagreements before gating.
| judge pass | judge fail | |
|---|---|---|
| human pass | 2 | 2 |
| human fail | 0 | 12 |
Two disagreements: one human accepted an escalation the judge punished, one human accepted a slow but successful email. Both are rubric problems, not model problems — exactly what calibration is for.
CI gate
tracegrade gate baseline.json current.json --max-regression 0.05
Exit code 1 when the failure rate regresses or a new failure mode appears.
Worked example: the traces are fictional, the output is real — generated by running Tracegrade on the bundled dataset.
Connects securely to your tools. The creator never sees your data.
About this skill
Tracegrade turns production agent traces into a calibrated eval suite: failure modes, assertions, rubrics and a judge checked against human labels.
Use this skill when:
- turn traces into evals
- grade agent runs
- build a failure taxonomy
- gate deploys on quality
Who it's for: Teams running agents in production.
Example request
Turn last week's failing traces into an eval suite.
What the example returns
- 11 failure modes detected
- Cases with assertions and rubrics
- Judge vs humans: kappa 0.78
In the package: five files — SKILL.md with the trigger and workflow, README.md, references/METHOD.md (the full method), references/CHECKLIST.md (the pass/fail gate) and examples/EXAMPLE.md (a worked run you can copy).
Works with Claude Code, Cursor, Codex CLI and any agent that reads SKILL.md. Plain markdown: no API key, no account, no network access.

Requires Node 18.17+. Requires existing JSONL traces with input/output fields. Does not perform LLM calls; user must provide judge verdicts for calibration.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 2 days ago
Requires
Permissions
Reads local trace files only. No network access, no credentials requested, and no files modified outside the output directory.