More screenshots
Works with the AI Agents you already use
rag retrieval eval set builder
Turn your docs into chunks plus grounded test questions, then score any retriever with recall@1/3/5/10 and MRR. Find out if RAG search actually finds the right chunk.
$8.40
rag retrieval eval set builder
Example session with this skill installed
Build a retrieval test set from the technical docs in /docs and run a score check against my current results in retrieved.jsonl. Set the minimum recall@5 to 80%.
- Read your context and instructions
- Compiled the rag retrieval eval
I have processed the documents in /docs, generated a grounded QA test set, and evaluated your current retrieval performance. The system failed the performance gate as the Recall@5 was 74%, which is below your 80% requirement.
RAG Retrieval Evaluation Report
| Metric | Value | Status |
|---|---|---|
| Total Test Queries | 150 | - |
| Recall@1 | 0.52 | ⚠️ Low |
| Recall@3 | 0.68 | ⚠️ Low |
| Recall@5 | 0.74 | ❌ Failed Gate (<0.80) |
| Recall@10 | 0.82 | ✅ Pass |
| MRR (Mean Reciprocal Rank) | 0.61 | ⚠️ Needs improvement |
Top Retrieval Failures
- Query ID 42: "How do I configure mutual TLS?"
- Expected Chunk:
chunk_tls_09.md - Actual Rank: Not in top 10
- Suggested Fix: Chunk size is too small; the context spans two headers. Increase chunk word cap.
- Expected Chunk:
- Query ID 89: "Rate limit for tier 2 users"
- Expected Chunk:
limits_table_v2.md - Actual Rank: 7
- Suggested Fix: Metadata filtering issue. The retriever is pulling v1 docs instead of v2.
- Expected Chunk:
Next steps
- Review
eval-qa.jsonlto see the generated questions for the failed chunks. - Adjust the chunking word cap in the build script and regenerate the set.
- Update your vector store metadata filters to prioritize the latest documentation versions.
Connects securely to your tools. The creator never sees your data.
About this skill
Most RAG bugs are retrieval bugs. Measure them before you tune the prompt.
build splits Markdown/text docs into heading-aware chunks and generates grounded questions (definitions, numbers, how-tos, section questions) whose answer chunk is known. Run those questions through your retriever, save the returned chunk ids, and score reports recall@k, MRR and the questions that missed.
What it does
- Heading-aware chunking with a word cap
- Auto-generated questions with known gold chunk ids
- recall@1/3/5/10 + MRR, list of misses
--min-recall5gate for CI
Who it's for
Developers building RAG chatbots, doc search or support assistants.
Limits
Template questions are simpler than real user queries - add real ones too. Not legal/financial advice.
What's included
SKILL.md: workflow, commands, how to explain findingsscripts/: the tool (stdlib Python)references/sources.md: specs and guidance citedexamples/: sample inputs used in the demotests/: unit tests (Python 3.8 / 3.9 / 3.13)
Standard-library Python only. No API keys, no network calls, no installs.
Offline / local files only - does not fetch live URLs or call APIs.
Template questions are simpler than real user queries - add real ones too.
Scores retrieval only, not answer quality.
Not legal/financial advice.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 16 days ago
Permissions
File Scopes
Fully offline. No subprocess, no network calls, no credential access.