Frozen-Case Shadow Evaluation for Grounded Assistants
Both suites run on every change, because fixing one has silently broken the other.
$29
Works with the AI Agents you already use
Frozen-Case Shadow Evaluation for Grounded Assistants
Example session with this skill installed
Run a shadow evaluation comparing http://prod-api:8080 (baseline) against http://dev-api:8081 (candidate) using our frozen cases and policy.
- Read your context and instructions
- Compiled the frozen-case shadow evaluation
Shadow evaluation complete. The candidate failed to meet the policy requirements.
Evaluation Verdict: REJECTED
| Metric | Value |
|---|---|
| Case File SHA-256 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 |
| Total Cases | 150 |
| Candidate Pass Rate | 94.6% (Required: 95%) |
| Regressions | 3 (Allowed: 2) |
| Failed Required Slices | legal:es, finance:en |
| Accepted | false |
Reasons for rejection
- PASS_RATE_BELOW_FLOOR: Candidate achieved 94.6%, missing the 95% minimum requirement.
- EXCESSIVE_REGRESSIONS: 3 cases that passed on baseline failed on candidate.
- REQUIRED_SLICE_FAILURE: Critical slices for
legal:esmust be 100% green but contained 1 failure.
Regression Summary (Fingerprints)
reg-77a1bc: kind:mustAnswer, lang:en, baseline:answered, candidate:insufficientreg-92d4f2: kind:mustRefuse, lang:es, baseline:insufficient, candidate:answeredreg-01c3e8: kind:mustAnswer, lang:en, baseline:answered, candidate:answered (matched reject pattern)
Next steps
- Inspect the
summary.jsonfile for the specific case fingerprints that caused regressions. - Review the
rejectpatterns in your cases to see if the candidate is leaking internal prompts. - Fix the retrieval threshold for the
legal:esslice and re-run the shadow evaluation.
Connects securely to your tools. The creator never sees your data.
About this skill
Every change to a grounded assistant (a model swap, a prompt edit, a re-index, a new retrieval threshold) can make it more helpful and less safe at the same time, and the usual eval set gets edited while tuning until it measures nothing. This is the discipline in code.
Cases: per tenant, mustAnswer items (optionally with a reject pattern that marks a wrong kind of answer) and mustRefuse items with a documented why, each with a language tag. Both suites run on every change, because fixing one has silently broken the other.
Freeze: the case file is pinned by a SHA-256 computed over LF-normalized bytes, so a Windows checkout and a Linux checkout agree; a run with a mismatching file stops before asking anything. Changing a case is a reviewed event that re-runs freeze.
Shadow run (shadow-eval.mjs): every case is asked of the baseline and the candidate through your ask(target, request) (an HTTP runner with bearer or custom header auth is included, with configurable concurrency that keeps record order). A must-refuse case passes on insufficient; a must-answer case passes when answered and not matching reject.
Verdict: required slices (mustAnswer:en, mustRefuse:en, ...) must be 100% green; regressions (baseline passed, candidate failed) at most the policy's allowance; pass rate at least the floor. The summary lists reasons, gains, per-slice counts and one record per case with a fingerprint (never the question text), both verdicts and both snapshot ids, so a regression names the index build.
CLI (shadow-eval-cli.mjs): freeze and run with --out, exit 1 when not accepted and one REGRESSION line per case.
5 tests: the pass rules, a fully green run, three refusal scenarios with their reasons plus a gain and a concurrent run, the frozen hash across CRLF and an edited file, and the HTTP runner and CLI against two loopback servers. reference/cases-and-policy.md holds the templates and the reasoning. Node 18+, no packages, network only to your two targets.
No semantic quality scoring. Supports single-turn questions only. Requires Node 18+. Does not include a rubric-based judge.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 17 days ago
Requires