More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    LLM Eval Regression Check: Compare saved runs offline

    1

    Your prompt tweak fixed one case and quietly broke three. Compare a saved eval run against a baseline, offline, and see exactly which cases regressed, with pass rate and cost. Exact, contains and JSON-subset checks.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    LLM Eval Regression Check: Compare saved runs offline

    Example session with this skill installed

    Compare the bundled regressing candidate against the saved known-good baseline using the invented token prices. Show the gate decision, failing case IDs and next fixes.

    • Read your context and instructions
    • Compiled the llm eval regression

    Gate: FAIL (exit 3). 3 of 5 cases regressed against the saved baseline, which passed all 5. Pass rate: 2/5.

    CaseCheckResultReason
    en-capitalexactFAIL, regressedexact string differs (case and whitespace are significant)
    zh-thanksexactpassexact match
    en-refundcontainspasssubstring found
    zh-weathercontainsFAIL, regressedrequired literal substring not found in string output
    en-ticketjson_subsetFAIL, regressed$["tags"][0]: value or type differs

    Latency p95: 250 ms. Total cost: 0.0000393 USD across 5 priced cases.

    Real stdout from the worked command in SKILL.md, using all five shipped cases, pricing and saved baseline, with --quiet (exit 3):

    {
      "schema_version": 1,
      "suite_sha256": "24843ae48eeb0cb13af8a3ccaea29ad3518b69978a4552db001d3353382acb93",
      "results": [
        {
          "id": "en-capital",
          "passed": false,
          "checks": [
            {
              "type": "exact",
              "passed": false,
              "reason": "exact string differs (case and whitespace are significant)"
            }
          ],
          "output": "paris",
          "metrics": {
            "latency_ms": "105",
            "input_tokens": 14,
            "output_tokens": 2
          },
          "cost": "0.0000033"
        },
        {
          "id": "zh-thanks",
          "passed": true,
          "checks": [
            {
              "type": "exact",
              "passed": true,
              "reason": "exact match"
            }
          ],
          "output": "谢谢。",
          "metrics": {
            "latency_ms": "92",
            "input_tokens": 9,
            "output_tokens": 3
          },
          "cost": "0.00000315"
        },
        {
          "id": "en-refund",
          "passed": true,
          "checks": [
            {
              "type": "contains",
              "passed": true,
              "reason": "substring found"
            }
          ],
          "output": "You can request a refund within 30 days of purchase.",
          "metrics": {
            "latency_ms": "220",
            "input_tokens": 22,
            "output_tokens": 13
          },
          "cost": "0.0000111"
        },
        {
          "id": "zh-weather",
          "passed": false,
          "checks": [
            {
              "type": "contains",
              "passed": false,
              "reason": "required literal substring not found in string output"
            }
          ],
          "output": "苏州今天多云。",
          "metrics": {
            "latency_ms": "175",
            "input_tokens": 18,
            "output_tokens": 8
          },
          "cost": "0.0000075"
        },
        {
          "id": "en-ticket",
          "passed": false,
          "checks": [
            {
              "type": "json_subset",
              "passed": false,
              "reason": "$[\"tags\"][0]: value or type differs"
            }
          ],
          "output": {
            "priority": "high",
            "tags": [
              "urgent",
              "billing"
            ],
            "escalate": "true"
          },
          "metrics": {
            "latency_ms": "250",
            "input_tokens": 31,
            "output_tokens": 16
          },
          "cost": "0.00001425"
        }
      ],
      "summary": {
        "cases": 5,
        "passed": 2,
        "failed": 3,
        "pass_rate": "0.400000",
        "latency_ms": {
          "observed": 5,
          "missing": 0,
          "min": "92",
          "max": "250",
          "mean": "168.4",
          "p50": "175",
          "p95": "250"
        },
        "cost": {
          "currency": "USD",
          "total": "0.0000393",
          "priced_cases": 5,
          "missing_token_cases": 0
        }
      },
      "comparison": {
        "baseline_passed": 5,
        "current_passed": 2,
        "regressions": [
          "en-capital",
          "en-ticket",
          "zh-weather"
        ],
        "improvements": [],
        "unchanged": 2
      }
    }
    

    FAIL — exit 3: en-capital, en-ticket and zh-weather regressed; 2/5 cases pass. Restore the exact capital string, required weather city and ticket tag order. Inspect the ticket escalation type too: the subset check stops at the first mismatch. The supplied fixture tokens cost 0.0000393 USD and supplied latency p95 is 250 ms; neither changes the decision. Record corrected outputs and rerun against the unchanged baseline.

    Real stdout from the worked command in SKILL.md

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem You changed a prompt, reran your eval set, and the overall score went up. That doesn't tell you which answers used to pass and now fail. A higher total can hide real breakage: in one public pair of saved runs I checked, 253 answers got better while 22 that used to pass got worse.

    See it before you buy: real input, real output Command (from the package folder, exit code 3, 0.03 s):

    python3 -I -B scripts/tb_gate.py --cases references/samples/cases.jsonl --outputs references/samples/outputs-candidate-regress.jsonl --pricing references/samples/pricing.json --baseline references/samples/baseline-report.json
    

    Before: 5 test cases (exact, contains and JSON-subset checks), a saved baseline run that passed 5 of 5, and the new candidate run.

    After: lines copied from the script's output (excerpt):

        "pass_rate": "0.400000",
      "comparison": {
        "baseline_passed": 5,
        "current_passed": 2,
        "regressions": [
          "en-capital",
          "en-ticket",
          "zh-weather"
        "unchanged": 2
    

    What the numbers show

    • Ran on three public saved GSM8K runs (1,319 answers each): the script's pass/fail matched a separate plain-Python check on every case (0 mismatches).
    • Run B against A (same model, later date): 22 got worse, 253 got better. Run C against A (different model): 153 worse, 198 better.
    • Honest limit: 'must contain' is literal text matching (expecting #### 18 also passes #### 180), so this proves the bookkeeping, not answer quality.

    What you get

    • One Python script (standard library only) that reads your saved test cases and saved model answers. It never calls a model.
    • Three kinds of check: exact string match, "must contain this text", and "this JSON must include these fields". Every result comes with a one-line reason.
    • A per-case comparison against a run you already trust. New passes never cancel out new failures.
    • Token cost from the prices you give it, and average, median and 95th-percentile latency from the timings you give it.
    • Exit codes made for CI (0 pass, 1 failures, 2 bad input, 3 regressions), a JSON report and a short summary on stderr.
    • Five sample cases in English and Chinese, four saved sample reports, a written input format and instructions your agent follows when it explains failures.

    Your agent checks your files against the format, runs the script, tells you which case IDs failed and why, and helps you rerun once you've fixed the answers.

    Measured on real data (full detail) I took three public saved runs of the GSM8K math benchmark, 1,319 answers each. Runs A and B are the same model on two dates. Run C is a different model. I pulled the final number out of each answer and checked it against the right answer. The script passed 261, 492 and 306 answers, and a separate plain-Python check got the same pass/fail result on every single case.

    ComparisonGot worseGot betterMismatches against the separate check
    Run B against A (same model, later date)222530
    Run C against A (different model)1531980

    The leaderboard's own pass flags differ from my number extraction on 22, 3 and 5 of the 1,319 answers per run. The reference file lists each one with the reason. The "must contain" check has a real weakness here: expecting #### 18 also passes #### 180. So these numbers show the bookkeeping is right. They don't show that text checks measure answer quality.

    Known limitations

    • No semantic, factual or safety grading. Number extraction can accept incidental numbers, and substrings can match incorrect answers.
    • You supply outputs, token counts, latencies and prices. The script collects none of them. Bundled sample metrics are invented.
    • Every failure blocks a passing verdict. There are no configurable failure tolerances or cost and latency budgets.
    • Inputs stay in memory. Each JSONL file is limited to 100,000 rows, each row to 1 MiB and a baseline to 16 MiB.
    • Baselines are structurally checked, not authenticated. Reports contain full outputs without redaction.

    Who it's for Developers who already have recorded outputs and explicit string or JSON assertions, and need to review prompt or pipeline changes locally or in CI.

    Requires Python 3.8 or newer, plus your cases and saved answers as JSONL in the documented format. No extra Python packages. (Converting the public benchmark files for the test above needed a Parquet reader. The script itself doesn't.)

    It runs locally with no network access and no telemetry. It only writes a file when you pass --report or redirect the output.

    Check it yourself Ran the 22 bundled test cases and checked the four saved sample reports against fresh output. Ran 16 more cases with real public runs and awkward inputs (empty, broken, Unicode, CRLF, oversized). Rechecked the three 1,319-answer runs and both comparisons with separate Python code. To re-run it yourself, go to the package folder and run the worked command in SKILL.md. The public-data commands are in references/benchmarks.md.

    Price and guarantee $15 one time, yours to use with any agent that can run Python. Covered by Agensi's money-back guarantee if it doesn't do what this page promises.

    Deterministic only; no semantic or safety grading. No live model calls or token counting. Memory-bound with 100k record limit. Requires pre-recorded JSONL inputs.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 15 days ago

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    Uses a terminal to run Python and file-reading tools for user-selected cases, outputs, optional prices and a baseline. By default it writes JSON to stdout and a summary to stderr. It writes a file only for an explicit report destination or shell redirection; that destination can replace an existing report. No network access, no telemetry and nothing is sent anywhere.

    Listed15 days ago
    Updated5 days ago

    What's inside

    Frequently Asked Questions

    If promptfoo already gates your saved runs as you need, this adds little. This package gives you a small standard-library evaluator, a strict recorded-output contract and agent instructions for baseline review. It has only three literal check types.

    No. They verify scoring arithmetic and regression bookkeeping on real saved outputs. The documented adapter disagrees with some leaderboard flags, and substring checks accept numeric prefixes.

    Fail on any nonzero exit: 1 means failures, 2 invalid input or I/O, and 3 regressions. Reports include full recorded outputs, so choose what to retain and share.

    Agensi's money-back guarantee covers every paid skill: if it doesn't do what this page promises, you get your money back. Everything above is output from the script in the package, so you can see what it does before you buy.

    New in LLM Ops