- Home
- Skills
- Testing & Debugging
- eval-harness-starter
Eval Harness Starter
Ship a reusable prompt eval harness with golden sets
$7
Works with the AI Agents you already use
About this skill
Every prompt change is a regression risk: you tweak wording, a case that worked yesterday breaks silently, and nobody notices until a user complains. This skill ships a small, dependency-free Python harness (eval_run.py) that runs your prompt against a golden set of known cases, scores each answer with exact-match and rubric rules you define, and writes a Markdown regression report you can diff across runs. It never calls an LLM itself — you plug your own model call into one explicit seam.
Give any agent a repeatable, versioned way to answer "did my prompt change break anything?" The harness owns three jobs: (1) load a golden set in a simple JSONL format, (2) get one answer per case — via your answer function or from recorded predictions — and (3) score every answer and report pass rates plus per-case failures. The agent owns everything the harness doesn't: the prompt, the golden cases, the rubric rules, and the interpretation of a failing run.
-
You are iterating on a prompt (system instructions, few-shot examples, output format) and want a stable measurement before/after each change.
-
You need to compare two models, two temperatures, or two agent designs on the same task.
-
You want a regression suite you can re-run in CI or before shipping a prompt to production.
-
You are building a classifier, extractor, router, or any task where "right answer" is checkable by rules.
-
Open-ended creative tasks where no rubric you write would be fair (story quality, humor, persuasion) — human review beats a rubric there.
-
When you don't yet have 10+ representative cases; build the golden set first, then harness it.
-
As a substitute for unit tests on deterministic code — use a real test framework for that.
-
If you need statistical significance: a 4-case smoke set shows mechanics, not confidence. 50+ cases is a real eval.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago