More screenshots
Works with the AI Agents you already use
Search Ranking Eval: nDCG, MRR, recall regression gate
Did your ranking change make search better or worse? Compare a candidate run against relevance labels and a baseline: nDCG, MRR, precision and recall per query, the worst queries listed, and a pass/fail gate for CI.
$12
Search Ranking Eval: nDCG, MRR, recall regression gate
Example session with this skill installed
Evaluate the supplied candidate against the accepted ideal report at k=5 with an absolute 0.05 drop allowance. Show failed gates and prioritize query investigations.
- Read your context and instructions
- Compiled the search ranking eval
Actual stdout, exit 1, from python3 -I -B scripts/search_eval.py references/qrels.json references/candidate.json --k 5 --baseline references/baseline.json --max-drop 0.05 --format markdown:
Search relevance evaluation
Status: FAIL; queries: 3; k: 5
| Query | P@5 | R@5 | RR (MRR mean) | AP (MAP mean) | nDCG linear@5 | nDCG exp@5 |
|---|---|---|---|---|---|---|
| "MEAN" | 0.200000 | 0.555556 | 0.500000 | 0.444444 | 0.495977 | 0.505351 |
| "q1" | 0.400000 | 0.666667 | 0.500000 | 0.333333 | 0.487932 | 0.516054 |
| "q2" | 0.200000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
| "q3" | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
Worst queries (linear nDCG ascending; ID breaks ties): "q3", "q1", "q2"
Diagnostics: {"missing_run_queries": ["q3"], "ignored_run_queries": ["extra"], "no_positive_judgment_queries": [], "duplicate_run_ids_removed": 0}
Gate: absolute mean drop > 0.05; all six metrics.
| Metric | Candidate minus baseline | Failed |
|---|---|---|
| precision | -0.133333 | yes |
| recall | -0.444444 | yes |
| mrr | -0.500000 | yes |
| map | -0.555556 | yes |
| ndcg_linear | -0.504023 | yes |
| ndcg_exponential | -0.494649 | yes |
FAIL: all six means exceed the allowance. First investigate q3's missing ranking; then q1's missing d7 and relevant documents at ranks 2 and 4. q2's only relevant document is already first. These are small made-up example files that show how the check works. They say nothing about a real search system.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem You've changed a reranker. Two queries look better, but a third has quietly dropped out of the export. Before you call the change a win, you need every labeled query scored and compared with the version you trust. This package does that on your saved files and points at the queries where coverage or ordering got worse.
See it before you buy: real input, real output Command (from the package folder, exit code 1, 0.027 s):
python3 -I -B scripts/search_eval.py references/qrels.json references/candidate.json --k 5 --baseline references/baseline.json --max-drop 0.05 --format markdown
Before: A saved candidate ranking that covers 2 of the 3 labeled queries (q1 and q2, with an extra unlabeled query), the relevance labels, and an accepted baseline.
After: lines copied from the script's output (excerpt):
Status: FAIL; queries: 3; k: 5
Worst queries (linear nDCG ascending; ID breaks ties): "q3", "q1", "q2"
Diagnostics: {"missing_run_queries": ["q3"], "ignored_run_queries": ["extra"], "no_positive_judgment_queries": [], "duplicate_run_ids_removed": 0}
| precision | -0.133333 | yes |
| mrr | -0.500000 | yes |
| ndcg_linear | -0.504023 | yes |
What the numbers show
- Matches 5 worked examples with known answers (scikit-learn docs, university lecture notes, and my own Searchlite test): all 26 numbers compared agreed to four decimal places.
- Each run took under 0.07 seconds on Python 3.13, startup included.
- Honest limit: this shows the scores are computed correctly, not how good your search is.
What you get
- A standalone Python script and instructions your agent can follow to evaluate your files and explain the results.
- Six metrics: precision@k, recall@k, MRR, MAP, linear nDCG@k and exponential nDCG@k.
- JSON or headerless TSV inputs, with JSON or Markdown reports.
- Diagnostics for missing queries, extra queries, queries without positive labels and duplicate document IDs.
- A pass/fail check against a report you already accepted. You set how much each average score may drop, and the JSON output shows the change for every query.
- Format notes, example inputs and 10 saved reports you can reproduce from the package folder.
Measured on real data (full detail) I checked the math against worked examples with known answers: two from the scikit-learn docs, one from university lecture notes, and a test case from my own Searchlite project. This proves the scores are computed correctly. It says nothing about how good your search is.
| Input | Expected score | Script result | Queries |
|---|---|---|---|
| scikit-learn example A, linear nDCG@5 | 0.6957 | 0.6957 | 1 |
| scikit-learn example B, linear nDCG@5 | 0.4937 | 0.4937 | 1 |
| scikit-learn example B, linear nDCG@4 | 0.3520 | 0.3520 | 1 |
| University MRR example, k=2 | 0.6111 | 0.6111 | 3 |
| Searchlite test case, mean recall@5 | 0.5556 | 0.5556 | 3 |
Across those five runs I compared 26 numbers (per-query and average scores for each metric) with values I worked out by hand from the formulas. All 26 matched to four decimal places. The scikit-learn page only prints two digits (0.69, 0.49, 0.35), so I recomputed its expected values at full precision from the same inputs. Each run took under 0.07 seconds on Python 3.13, startup included. The commands and sources are in references/benchmarks.md.
Known limitations
- You must supply labels and saved rankings. The script doesn't retrieve documents or create judgments.
- Unjudged documents count as nonrelevant, so incomplete labels can distort the scores.
- MRR and MAP use the full supplied ranking. The cutoff applies only to precision, recall and nDCG.
- The pass/fail check looks at averages only. It doesn't test statistical significance, and a pass doesn't mean every query got better.
- Raw TREC files need converting to the documented JSON or TSV first. If your scores have ties, your export has to fix their order. The script won't average them.
Who it's for Retrieval engineers and RAG teams comparing retrievers or rerankers on labeled queries.
Requires Python 3.8+ and an agent that can read files and run terminal commands. Only the standard library is needed. The code sticks to Python 3.8 syntax, but I've only run it on 3.13. Everything runs locally. There's no network access or telemetry, and the script writes nothing unless you redirect output.
Price and guarantee $12 one time, yours to use with any agent that can run Python. Covered by Agensi's money-back guarantee if it doesn't do what this page promises.
No support for live search, latency tests, or LLM-based answer grading. 16 MiB file limit per input. Requires Python 3.8+. UTF-8 only.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 15 days ago
Requires