More screenshots
Works with the AI Agents you already use
LLM Eval Regression Check: Compare saved runs offline
Your prompt tweak fixed one case and quietly broke three. Compare a saved eval run against a baseline, offline, and see exactly which cases regressed, with pass rate and cost. Exact, contains and JSON-subset checks.
$15
LLM Eval Regression Check: Compare saved runs offline
Example session with this skill installed
Compare the bundled regressing candidate against the saved known-good baseline using the invented token prices. Show the gate decision, failing case IDs and next fixes.
- Read your context and instructions
- Compiled the llm eval regression
Gate: FAIL (exit 3). 3 of 5 cases regressed against the saved baseline, which passed all 5. Pass rate: 2/5.
| Case | Check | Result | Reason |
|---|---|---|---|
| en-capital | exact | FAIL, regressed | exact string differs (case and whitespace are significant) |
| zh-thanks | exact | pass | exact match |
| en-refund | contains | pass | substring found |
| zh-weather | contains | FAIL, regressed | required literal substring not found in string output |
| en-ticket | json_subset | FAIL, regressed | $["tags"][0]: value or type differs |
Latency p95: 250 ms. Total cost: 0.0000393 USD across 5 priced cases.
Real stdout from the worked command in SKILL.md, using all five shipped cases, pricing and saved baseline, with --quiet (exit 3):
{
"schema_version": 1,
"suite_sha256": "24843ae48eeb0cb13af8a3ccaea29ad3518b69978a4552db001d3353382acb93",
"results": [
{
"id": "en-capital",
"passed": false,
"checks": [
{
"type": "exact",
"passed": false,
"reason": "exact string differs (case and whitespace are significant)"
}
],
"output": "paris",
"metrics": {
"latency_ms": "105",
"input_tokens": 14,
"output_tokens": 2
},
"cost": "0.0000033"
},
{
"id": "zh-thanks",
"passed": true,
"checks": [
{
"type": "exact",
"passed": true,
"reason": "exact match"
}
],
"output": "谢谢。",
"metrics": {
"latency_ms": "92",
"input_tokens": 9,
"output_tokens": 3
},
"cost": "0.00000315"
},
{
"id": "en-refund",
"passed": true,
"checks": [
{
"type": "contains",
"passed": true,
"reason": "substring found"
}
],
"output": "You can request a refund within 30 days of purchase.",
"metrics": {
"latency_ms": "220",
"input_tokens": 22,
"output_tokens": 13
},
"cost": "0.0000111"
},
{
"id": "zh-weather",
"passed": false,
"checks": [
{
"type": "contains",
"passed": false,
"reason": "required literal substring not found in string output"
}
],
"output": "苏州今天多云。",
"metrics": {
"latency_ms": "175",
"input_tokens": 18,
"output_tokens": 8
},
"cost": "0.0000075"
},
{
"id": "en-ticket",
"passed": false,
"checks": [
{
"type": "json_subset",
"passed": false,
"reason": "$[\"tags\"][0]: value or type differs"
}
],
"output": {
"priority": "high",
"tags": [
"urgent",
"billing"
],
"escalate": "true"
},
"metrics": {
"latency_ms": "250",
"input_tokens": 31,
"output_tokens": 16
},
"cost": "0.00001425"
}
],
"summary": {
"cases": 5,
"passed": 2,
"failed": 3,
"pass_rate": "0.400000",
"latency_ms": {
"observed": 5,
"missing": 0,
"min": "92",
"max": "250",
"mean": "168.4",
"p50": "175",
"p95": "250"
},
"cost": {
"currency": "USD",
"total": "0.0000393",
"priced_cases": 5,
"missing_token_cases": 0
}
},
"comparison": {
"baseline_passed": 5,
"current_passed": 2,
"regressions": [
"en-capital",
"en-ticket",
"zh-weather"
],
"improvements": [],
"unchanged": 2
}
}
FAIL — exit 3: en-capital, en-ticket and zh-weather regressed; 2/5 cases pass. Restore the exact capital string, required weather city and ticket tag order. Inspect the ticket escalation type too: the subset check stops at the first mismatch. The supplied fixture tokens cost 0.0000393 USD and supplied latency p95 is 250 ms; neither changes the decision. Record corrected outputs and rerun against the unchanged baseline.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem You changed a prompt, reran your eval set, and the overall score went up. That doesn't tell you which answers used to pass and now fail. A higher total can hide real breakage: in one public pair of saved runs I checked, 253 answers got better while 22 that used to pass got worse.
See it before you buy: real input, real output Command (from the package folder, exit code 3, 0.03 s):
python3 -I -B scripts/tb_gate.py --cases references/samples/cases.jsonl --outputs references/samples/outputs-candidate-regress.jsonl --pricing references/samples/pricing.json --baseline references/samples/baseline-report.json
Before: 5 test cases (exact, contains and JSON-subset checks), a saved baseline run that passed 5 of 5, and the new candidate run.
After: lines copied from the script's output (excerpt):
"pass_rate": "0.400000",
"comparison": {
"baseline_passed": 5,
"current_passed": 2,
"regressions": [
"en-capital",
"en-ticket",
"zh-weather"
"unchanged": 2
What the numbers show
- Ran on three public saved GSM8K runs (1,319 answers each): the script's pass/fail matched a separate plain-Python check on every case (0 mismatches).
- Run B against A (same model, later date): 22 got worse, 253 got better. Run C against A (different model): 153 worse, 198 better.
- Honest limit: 'must contain' is literal text matching (expecting
#### 18also passes#### 180), so this proves the bookkeeping, not answer quality.
What you get
- One Python script (standard library only) that reads your saved test cases and saved model answers. It never calls a model.
- Three kinds of check: exact string match, "must contain this text", and "this JSON must include these fields". Every result comes with a one-line reason.
- A per-case comparison against a run you already trust. New passes never cancel out new failures.
- Token cost from the prices you give it, and average, median and 95th-percentile latency from the timings you give it.
- Exit codes made for CI (0 pass, 1 failures, 2 bad input, 3 regressions), a JSON report and a short summary on stderr.
- Five sample cases in English and Chinese, four saved sample reports, a written input format and instructions your agent follows when it explains failures.
Your agent checks your files against the format, runs the script, tells you which case IDs failed and why, and helps you rerun once you've fixed the answers.
Measured on real data (full detail) I took three public saved runs of the GSM8K math benchmark, 1,319 answers each. Runs A and B are the same model on two dates. Run C is a different model. I pulled the final number out of each answer and checked it against the right answer. The script passed 261, 492 and 306 answers, and a separate plain-Python check got the same pass/fail result on every single case.
| Comparison | Got worse | Got better | Mismatches against the separate check |
|---|---|---|---|
| Run B against A (same model, later date) | 22 | 253 | 0 |
| Run C against A (different model) | 153 | 198 | 0 |
The leaderboard's own pass flags differ from my number extraction on 22, 3 and 5 of the 1,319 answers per run. The reference file lists each one with the reason. The "must contain" check has a real weakness here: expecting #### 18 also passes #### 180. So these numbers show the bookkeeping is right. They don't show that text checks measure answer quality.
Known limitations
- No semantic, factual or safety grading. Number extraction can accept incidental numbers, and substrings can match incorrect answers.
- You supply outputs, token counts, latencies and prices. The script collects none of them. Bundled sample metrics are invented.
- Every failure blocks a passing verdict. There are no configurable failure tolerances or cost and latency budgets.
- Inputs stay in memory. Each JSONL file is limited to 100,000 rows, each row to 1 MiB and a baseline to 16 MiB.
- Baselines are structurally checked, not authenticated. Reports contain full outputs without redaction.
Who it's for Developers who already have recorded outputs and explicit string or JSON assertions, and need to review prompt or pipeline changes locally or in CI.
Requires Python 3.8 or newer, plus your cases and saved answers as JSONL in the documented format. No extra Python packages. (Converting the public benchmark files for the test above needed a Parquet reader. The script itself doesn't.)
It runs locally with no network access and no telemetry. It only writes a file when you pass --report or redirect the output.
Check it yourself Ran the 22 bundled test cases and checked the four saved sample reports against fresh output. Ran 16 more cases with real public runs and awkward inputs (empty, broken, Unicode, CRLF, oversized). Rechecked the three 1,319-answer runs and both comparisons with separate Python code. To re-run it yourself, go to the package folder and run the worked command in SKILL.md. The public-data commands are in references/benchmarks.md.
Price and guarantee $15 one time, yours to use with any agent that can run Python. Covered by Agensi's money-back guarantee if it doesn't do what this page promises.
Deterministic only; no semantic or safety grading. No live model calls or token counting. Memory-bound with 100k record limit. Requires pre-recorded JSONL inputs.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 15 days ago
Permissions
Uses a terminal to run Python and file-reading tools for user-selected cases, outputs, optional prices and a baseline. By default it writes JSON to stdout and a summary to stderr. It writes a file only for an explicit report destination or shell redirection; that destination can replace an existing report. No network access, no telemetry and nothing is sent anywhere.