Works with the AI Agents you already use
RAG Eval Kit: Did Your Change Really Help?
There's a CI gate too, so a build fails when accuracy's lower confidence bound drops below your threshold.
Free
RAG Eval Kit: Did Your Change Really Help?
Example session with this skill installed
My RAG answers questions about a set of index rulebooks. I tried two changes on the same 126 labelled questions: a new answer prompt (v3 instead of v1) and a new answer model (qwen3 instead of qwen2.5). The runs are in this folder. Which change actually helped?
- Read your context and instructions
- Compiled the rag eval kit
Summary
Only the model swap is a real improvement. Prompt v3 scored 8.7 points lower than v1 (69.8% to 61.1%), but with 20 questions only v1 got right against 9 only v3 got right, the exact McNemar p is 0.0614:
not distinguishable from noise at the 5% level. The model change raised accuracy from 69.8% to 81.7%, with 19 questions only the new model got right against 4:
p = 0.0026, a real difference. It costs speed: median latency went from 2.75 s to 17.18 s.
The runs, as scored
prompt_v1 (126 questions)
| Metric | Score | 95% interval |
|---|---|---|
| Accuracy | 69.8% (88/126) | 61–77% |
| Grounded accuracy | 55.6% (70/126) | 47–64% |
| Answerable accuracy | 68.6% (81/118) | 60–76% |
| Unanswerable refused | 87.5% (7/8) | 53–98% |
| False refusals | 7.6% (9/118) | 4–14% |
| Retrieval recall | 81.4% (96/118) | 73–87% |
| Citation precision | 57.3% (71/124) | 48–66% |
| Accuracy · paraphrase | 52.8% (19/36) | 37–68% |
| Accuracy · unanswerable | 87.5% (7/8) | 53–98% |
| Accuracy · untagged | 75.6% (62/82) | 65–84% |
Latency p50 2.18 s, p95 3.38 s.
1 answer(s) contained a forbidden fact.
prompt_v3 (126 questions)
| Metric | Score | 95% interval |
|---|---|---|
| Accuracy | 61.1% (77/126) | 52–69% |
| Grounded accuracy | 51.6% (65/126) | 43–60% |
| Answerable accuracy | 58.5% (69/118) | 50–67% |
| Unanswerable refused | 100.0% (8/8) | 68–100% |
| False refusals | 11.9% (14/118) | 7–19% |
| Retrieval recall | 81.4% (96/118) | 73–87% |
| Citation precision | 59.0% (69/117) | 50–68% |
| Accuracy · paraphrase | 36.1% (13/36) | 22–52% |
| Accuracy · unanswerable | 100.0% (8/8) | 68–100% |
| Accuracy · untagged | 68.3% (56/82) | 58–77% |
Latency p50 2.02 s, p95 3.6 s.
model_qwen2.5 (126 questions)
| Metric | Score | 95% interval |
|---|---|---|
| Accuracy | 69.8% (88/126) | 61–77% |
| Grounded accuracy | 56.3% (71/126) | 48–65% |
| Answerable accuracy | 69.5% (82/118) | 61–77% |
| Unanswerable refused | 75.0% (6/8) | 41–93% |
| False refusals | 8.5% (10/118) | 5–15% |
| Retrieval recall | 80.5% (95/118) | 72–87% |
| Citation precision | 52.9% (72/136) | 45–61% |
| Accuracy · paraphrase | 52.8% (19/36) | 37–68% |
| Accuracy · unanswerable | 75.0% (6/8) | 41–93% |
| Accuracy · untagged | 76.8% (63/82) | 67–85% |
Latency p50 2.75 s, p95 4.19 s.
2 answer(s) contained a forbidden fact.
model_qwen3 (126 questions)
| Metric | Score | 95% interval |
|---|---|---|
| Accuracy | 81.7% (103/126) | 74–88% |
| Grounded accuracy | 75.4% (95/126) | 67–82% |
| Answerable accuracy | 80.5% (95/118) | 72–87% |
| Unanswerable refused | 100.0% (8/8) | 68–100% |
| False refusals | 11.9% (14/118) | 7–19% |
| Retrieval recall | 82.2% (97/118) | 74–88% |
| Citation precision | 69.0% (89/129) | 61–76% |
| Accuracy · paraphrase | 55.6% (20/36) | 40–70% |
| Accuracy · unanswerable | 100.0% (8/8) | 68–100% |
| Accuracy · untagged | 91.5% (75/82) | 83–96% |
Latency p50 17.18 s, p95 23.6 s.
The paired tests
Prompt v1 (A) against prompt v3 (B):
correct on 126 shared questions: A 69.8% -> B 61.1% (-8.7 points)
only A right: 20 p001 p002 p003 p005 p010 p022 p030 p033 p034 q002 q008 q016 q023 q026 q027 q034 q043 q047 q051 q055
only B right: 9 p009 p012 p015 q009 q024 q050 q071 q073 u001
exact McNemar p = 0.0614: not distinguishable from noise at the 5% level
Old model (A) against new model (B):
correct on 126 shared questions: A 69.8% -> B 81.7% (+11.9 points)
only A right: 4 p012 p028 p029 p035
only B right: 19 p003 p008 p015 p022 p025 q009 q013 q023 q030 q045 q046 q050 q055 q061 q073 q080 q082 u001 u002
exact McNemar p = 0.0026: a real difference at the 5% level
What the numbers say
The new model wins where it counts: answerable accuracy 69.5% (61-77%) to 80.5% (72-87%), grounded accuracy 56.3% to 75.4%, citation precision 52.9% to 69.0%, and every unanswerable trap refused (8/8, against 6/8).
- Prompt v3 should not be adopted, but it is not proven worse either: its drop is largest on paraphrased questions (52.8% to 36.1%), and false refusals rose from 7.6% to 11.9%. A larger set could settle it.
Retrieval did not change: recall stays between 80.5% and 82.2% in all four runs, so the gains come from how the model uses the pages it gets. The 18-20% of questions whose page is never retrieved remain a ceiling on accuracy.
- Paraphrased questions remain the weak spot: 55.6% (40-70%) for the best run, against 91.5% on the rest.
Read by hand (old model wrong, new model right): q009, the old model quoted the threshold for a neighbouring case (5%) instead of the one asked (15%), although the right page was retrieved; q023, it refused, and in that run the right page had not been retrieved, so that one is a retrieval miss rather than a model error. And the other way: on p012 the new model said the documents give no threshold, missing the expected 10%.
Next steps
- Keep the new model and prompt v1. Decide whether 17 s per answer is acceptable before shipping.
- Work on paraphrased questions: query rewriting or retrieval tuned for them, measured on this dev set with
compare. - Then score the held-out set once:
python scripts/eval.py score heldout.jsonl pred_held.jsonl --heldout --system <your files>. - Add the gate to CI:
python scripts/eval.py gate report.json --min-ci-low accuracy=72.
Connects securely to your tools. The creator never sees your data.
About this skill
What it does
Most RAG "improvements" are noise. This skill tells you which are real. Your agent builds a labelled test set from your own documents, scores your system's answers without an LLM judge, and runs an exact paired test between two versions, so a new prompt, model, retriever or chunking gets a verdict you can defend.
- Test set from your documents: text, Markdown, HTML, or PDFs (via
pdftotext) cut into pages; pages drawn at random so questions aren't cherry-picked; paraphrased questions and unanswerable traps; every label checked against its page - Blind human review: a sheet with each label next to its source passage, reviewed without seeing any system's answers
- Deterministic scoring: accuracy, grounded accuracy (right and citing the right page), refusals on unanswerable questions, false refusals, retrieval recall, citation precision, all with 95% intervals, broken down by question style
- Is B really better than A? Exact McNemar test on the questions the two versions disagree on
- Held-out discipline: a dev / held-out split with no shared pages, and a ledger that warns when the held-out set is reused on a new version
- Fits your logs: converts JSONL, JSON or CSV output, LangChain-style source documents included, and matches by question text when there are no ids
How it works
Scripts do the counting and the statistics; your agent drafts the questions, runs the steps and explains the numbers. Three workflows: compare two runs you already have; build a test set from your documents (pages, random draw, draft, check, review, split); score a run and find where the errors come from (retrieval misses, wrong fact from the right page, false refusals). There's a CI gate too, so a build fails when accuracy's lower confidence bound drops below your threshold.
Why you can trust it
- No LLM judge: the same answers always get the same score, and every verdict traces back to the facts found or missing.
- The agent may not call a difference real unless the paired test says so, and must quote intervals, never bare percentages.
- Labels must pass a check against the source text before anything is scored, and unreviewed labels are reported as a draft.
- Everything runs locally. Your documents and answers never leave your machine.
Tested
- 27 automated tests, all offline: the engine (scoring, normalisation, Wilson intervals, exact McNemar, label checks, held-out ledger) and the workflow (pages from text, Markdown, HTML and PDF page breaks, seeded sampling, the dev / held-out split, output conversion, the whole chain). All pass on Python 3.12 and 3.13.
- Built on an open-source engine used to audit a 220-question RAG evaluation, where only 1 of 9 apparent improvements turned out to be statistically real.
- Run in Claude Code.
Known limits
- It checks that the right facts are present, not that nothing wrong was added; forbidden facts cover known traps only, so read some answers.
- Built for factual answers from documents, not summaries, open-ended writing or chat quality.
- PDFs need converting to text first (
pdftotext -layoutkeeps page numbers). - Intervals assume independent questions; near-duplicates make them too narrow.
Requires Python 3.10+. Checks that the right facts are present, not that nothing wrong was added; forbidden facts cover known traps only, so read some answers. Built for factual answers from documents, not summaries or chat quality. PDFs need converting to text first (pdftotext keeps page numbers). Intervals assume independent questions.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 11 days ago
- Free with an Agensi account
Requires
Permissions
File Scopes
No network access at all. Read Files: the documents, question sets and system outputs you name. Write Files: only the outputs you name (pages, questions, predictions, reports, review sheet).