rag eval harness
Build a statistically rigorous evaluation harness for RAG apps with golden sets, retrieval metrics, and error analysis.
$12
Works with the AI Agents you already use
rag eval harness
Example session with this skill installed
I've made some changes to my RAG pipeline's chunking strategy and added a reranker. I need to know if it's actually better than the old version. Can you build an eval harness for this?
- Read your context and instructions
- Compiled the rag eval harness
I have initialized the evaluation harness in the eval/ and scripts/ directories. I analyzed your retriever.py and app.py to identify the retrieval and generation entry points.
RAG Evaluation Harness
I have generated a starter golden set of 30 questions. Please review eval/questions.jsonl to ensure the gold_ids match the expected source chunks before running the full benchmark.
# eval/run_eval.py snippet
import json
import time
from your_app import rag_pipeline
def run_benchmark(questions_path, output_name):
results = []
with open(questions_path, 'r') as f:
for line in f:
q = json.loads(line)
start = time.time()
# Captures retrieved_ids, answer, and citations
response = rag_pipeline(q['question'])
results.append({
"id": q['id'],
"question": q['question'],
"answer": response.text,
"retrieved_ids": response.chunk_ids,
"gold_ids": q['gold_ids'],
"latency_s": time.time() - start
})
# Save to eval/results/<output_name>.jsonl
Initial Baseline Metrics (N=30)
| Metric | Mean | 95% Conf. Interval |
|---|---|---|
| hit@3 | 0.83 | [0.71, 0.94] |
| MRR | 0.76 | [0.65, 0.88] |
| Faithfulness | 0.88 | [0.82, 0.94] |
| Context Precision | 0.72 | [0.60, 0.84] |
Next steps
- Review
eval/questions.jsonland mark at least 10 questions assplit: "test". - Run
python scripts/metrics.py eval/results/new_chunking.jsonl --baseline eval/results/baseline.jsonlto see the delta. - Check the "Error Analysis" section in the generated report to see the top 5 failure cases.
Connects securely to your tools. The creator never sees your data.
About this skill
RAG applications often ship with zero measurement or anecdotal evidence that fails in production. This skill builds a repeatable evaluation harness directly in your repository to provide defensible metrics on retrieval quality and generation accuracy. It moves you from guessing if a change helped to knowing with statistical confidence.
What it does
- Golden set generation creates a structured
questions.jsonlcovering single facts, paraphrases, and multi-chunk queries. - Retrieval benchmarking calculates hit@k and Mean Reciprocal Rank (MRR) to verify your embedding and reranking logic.
- Generation auditing measures faithfulness, context precision, and answer relevance using LLM-as-a-judge patterns.
- Cost estimation predicts token spend for evaluation runs before execution to prevent surprise bills.
- Statistical aggregation computes mean, standard deviation, and 95 percent confidence intervals to distinguish signal from noise.
- Error diagnosis maps failure symptoms to root causes like chunk size, retrieval misses, or model hallucination.
How it works
- Analyze pipeline by identifying your entry points, retrieval functions, and current configuration.
- Bootstrap golden set using synthetic generation followed by a mandatory human review phase to ensure label quality.
- Execute runs via a dedicated script that captures inputs, outputs, and intermediate retrieval states.
- Compute and compare results against baselines to see if changes to your stack actually improved performance.
Frameworks & tools
The harness is built with Python, utilizing JSONL for data storage and basic statistical libraries for bootstrap intervals. It is designed to wrap any RAG stack regardless of the specific vector database or LLM provider.
Why this beats prompting it yourself
Generic prompts cannot handle the nuances of RAG evaluation like data leakage or judge bias. This skill enforces rigorous practices like held-out test splits, paired comparisons, and reporting N-size with confidence intervals to ensure your numbers are actually meaningful.
Use cases
- Benchmark a new embedding model or reranker against your current production baseline.
- Detect regressions in faithfulness after updating your system prompt or LLM version.
- Build a defensible quality report for stakeholders using statistically sound metrics.
- Create a CI smoke test to block pull requests that degrade retrieval performance.
Known limitations
Requires a manual review of at least a third of generated questions to ensure accuracy. Small question sets under 20 items will result in wide confidence intervals.
Requires manual review of generated questions. Small N-sizes (under 20) yield wide confidence intervals. Judge agreement must be hand-checked on 20 items.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 14 days ago
Requires