Frozen-Case Shadow Evaluation for Grounded Assistants

    1

    Both suites run on every change, because fixing one has silently broken the other.

    $29

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Frozen-Case Shadow Evaluation for Grounded Assistants

    Example session with this skill installed

    Run a shadow evaluation comparing http://prod-api:8080 (baseline) against http://dev-api:8081 (candidate) using our frozen cases and policy.

    • Read your context and instructions
    • Compiled the frozen-case shadow evaluation

    Shadow evaluation complete. The candidate failed to meet the policy requirements.

    Evaluation Verdict: REJECTED

    MetricValue
    Case File SHA-256e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    Total Cases150
    Candidate Pass Rate94.6% (Required: 95%)
    Regressions3 (Allowed: 2)
    Failed Required Sliceslegal:es, finance:en
    Acceptedfalse

    Reasons for rejection

    • PASS_RATE_BELOW_FLOOR: Candidate achieved 94.6%, missing the 95% minimum requirement.
    • EXCESSIVE_REGRESSIONS: 3 cases that passed on baseline failed on candidate.
    • REQUIRED_SLICE_FAILURE: Critical slices for legal:es must be 100% green but contained 1 failure.

    Regression Summary (Fingerprints)

    • reg-77a1bc: kind:mustAnswer, lang:en, baseline:answered, candidate:insufficient
    • reg-92d4f2: kind:mustRefuse, lang:es, baseline:insufficient, candidate:answered
    • reg-01c3e8: kind:mustAnswer, lang:en, baseline:answered, candidate:answered (matched reject pattern)

    Next steps

    1. Inspect the summary.json file for the specific case fingerprints that caused regressions.
    2. Review the reject patterns in your cases to see if the candidate is leaking internal prompts.
    3. Fix the retrieval threshold for the legal:es slice and re-run the shadow evaluation.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Every change to a grounded assistant (a model swap, a prompt edit, a re-index, a new retrieval threshold) can make it more helpful and less safe at the same time, and the usual eval set gets edited while tuning until it measures nothing. This is the discipline in code.

    Cases: per tenant, mustAnswer items (optionally with a reject pattern that marks a wrong kind of answer) and mustRefuse items with a documented why, each with a language tag. Both suites run on every change, because fixing one has silently broken the other.

    Freeze: the case file is pinned by a SHA-256 computed over LF-normalized bytes, so a Windows checkout and a Linux checkout agree; a run with a mismatching file stops before asking anything. Changing a case is a reviewed event that re-runs freeze.

    Shadow run (shadow-eval.mjs): every case is asked of the baseline and the candidate through your ask(target, request) (an HTTP runner with bearer or custom header auth is included, with configurable concurrency that keeps record order). A must-refuse case passes on insufficient; a must-answer case passes when answered and not matching reject.

    Verdict: required slices (mustAnswer:en, mustRefuse:en, ...) must be 100% green; regressions (baseline passed, candidate failed) at most the policy's allowance; pass rate at least the floor. The summary lists reasons, gains, per-slice counts and one record per case with a fingerprint (never the question text), both verdicts and both snapshot ids, so a regression names the index build.

    CLI (shadow-eval-cli.mjs): freeze and run with --out, exit 1 when not accepted and one REGRESSION line per case.

    5 tests: the pass rules, a fully green run, three refusal scenarios with their reasons plus a gain and a concurrent run, the frozen hash across CRLF and an edited file, and the HTTP runner and CLI against two loopback servers. reference/cases-and-policy.md holds the templates and the reasoning. Node 18+, no packages, network only to your two targets.

    No semantic quality scoring. Supports single-turn questions only. Requires Node 18+. Does not include a rubric-based judge.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 17 days ago

    Requires

    Node.js 18+

    Permissions

    Write Files
    Read Files
    Network Access
    Listed17 days ago

    What's inside

    Frequently Asked Questions

    It solves 'eval drift' by enforcing a strict, hash-verified test set. Unlike typical evals where prompts are tuned until they pass, this skill runs baseline and candidate systems side-by-side to ensure new changes don't cause silent regressions or break safety refusals.

    This is a dependency-free Node.js 18+ library designed for versatility. It works with any RAG or LLM architecture via an injectable 'ask' function, making it compatible with OpenAI, Anthropic, or custom internal API endpoints.

    The skill identifies 'must-answer' failures (where the system incorrectly refused a prompt) and 'must-refuse' failures (where the system answered a prompt it should have blocked), providing a clear JSON audit trail for safety and compliance.

    You will receive the core Node.js library, a CLI runner for easy CI/CD integration, and structured documentation on how to implement the SHA-256 locking mechanism for your test cases.

    This tool is a binary gate that checks for successful answers versus refusals across languages and tenants; it is not designed for semantic similarity scoring or grading the 'tone' of an LLM response.

    Setup is minimal since the library has zero external dependencies; you simply define your test cases in JSON, generate a policy hash, and integrate the library into your existing Node.js testing suite or shell-based deployment pipeline.

    New in LLM Ops