RAG Pipeline Production Auditor
Scan your RAG codebase and eval set, then get a scorecard and ranked fixes for chunking, retrieval, grounding, guardrails and cost.
Free
Works with the AI Agents you already use
RAG Pipeline Production Auditor
Example session with this skill installed
Audit our support bot before launch: repo in rag_app/, eval questions in eval.jsonl.
- Read your context and instructions
- Compiled the rag pipeline production
(This is the scanner's real output on the bundled example app, shortened.)
# RAG pipeline audit
Files scanned: 2 | areas with evidence: 2/17 | risky settings: 5 | gaps: 15
Risky settings
| medium | answer.py:5 | Passing 30 chunks (k=30) | rerank down to ~3-8 |
| medium | answer.py:8 | temperature = 0.9 for answer generation | use 0-0.3 for grounded Q&A |
| medium | answer.py:7 | user input + context in one f-string | delimit context, mark untrusted |
| medium | ingest.py:4 | Large chunk_size (4000 chars) | compare smaller sizes on eval set |
| low | ingest.py:4 | chunk_overlap = 0 | 10-20% overlap / structure-aware |
No evidence found: grounding_prompt (high), eval (high), guardrails_injection (high), citations, reranking, access_control, cost_controls, observability ...
Eval set: 3 rows | 2 with expected sources | 0 unanswerable | 1 duplicate
The agent then writes the scorecard (for example Generation & grounding 0/2: no "answer only from context" rule and no citations). The fix plan is ordered: build a 50+ question golden set with source ids and unanswerables → grounding prompt and citations → reranker over k=30 → max_tokens and usage logging. Each step names the metric to compare before and after.
Connects securely to your tools. The creator never sees your data.
About this skill
A pre-launch review for retrieval-augmented generation apps, based on your actual code rather than a generic questionnaire.
- Maps your pipeline. loader → splitter → embeddings → store → retriever → reranker → prompt → LLM → citations and guardrails, with file paths.
- Runs a read-only scanner (Python stdlib, no model calls). It finds evidence across 17 areas with file:line: chunking, chunk metadata, embedding pinning and versioning, hybrid search, metadata filters, reranking, relevance thresholds, grounding prompt, citations, evals, prompt-injection and PII guardrails, access control, cost controls, tracing, timeouts.
- Flags risky settings. Very large or zero-overlap chunks, k over 20 without a reranker, high temperature, user input and context concatenated into one f-string, and hard-coded API keys (reported redacted, never echoed).
- Checks your eval set (JSONL/CSV): size, reference answers, expected source ids, unanswerable questions and duplicates.
- Scores 8 areas 0–2 with a 37-item checklist, and gives an ordered fix plan where every fix is paired with the metric that proves it (recall@k, MRR, faithfulness, refusal rate, cost per request).
Built by an engineer who ships production RAG, including a solo-built consumer AI app.
What it won't do: call your models or vector DB, send documents anywhere, or edit code. It also won't claim accuracy gains. You measure them on your eval set.
Won't call your models or vector DB, send documents anywhere, or edit code. It also won't claim accuracy gains; you measure them on your eval set. It doesn't run evals. "No evidence" doesn't necessarily mean missing (logic may live in another service); the agent confirms by reading the code.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
1 use
Times used to date
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 1 month ago
- Free with an Agensi account
Requires
Permissions
Read: repository files and the eval set you provide Bash: `python scripts/audit_rag.py` (Python 3 standard library, read-only, no network, no writes) No API keys or service access needed