Works with the AI Agents you already use
RAG Failure Triage
Find out whether a bad RAG answer came from the query, retrieval, ranking, generation or the corpus, before you rewrite a single prompt.
Free
RAG Failure Triage
Example session with this skill installed
Triage this RAG failure. Question: "What is our refund window?" Query: "refund window". Top-3 chunks: [Doc A: Shipping info, Doc B: Pricing, Doc C: Contact]. Answer: "We don't have a refund window."
- Read your context and instructions
- Compiled the rag failure triage
I have analyzed the artifacts provided for this RAG failure. The system failed to retrieve the relevant document despite the information being present in the corpus.
RAG triage — case_001
Question: What is our refund window?
Query sent: "refund window"
Top-k contained the answer: no (n/a)
Answer used the context: yes (it correctly reflected the lack of info in the provided chunks)
Diagnosis
Layer: retrieval
Cause: Lexical mismatch between the user query and the document vocabulary.
Evidence
- Chunk A (Score 0.82): Covers shipping timelines and carrier options.
- Chunk B (Score 0.79): Lists subscription pricing tiers.
- Chunk C (Score 0.75): General support contact email and hours.
- Manual check: The source document uses the term "return period" instead of "refund window".
Smallest next test
Search the index using the specific phrase "return period". If the correct document appears in the top-3, the diagnosis of lexical mismatch is confirmed.
If confirmed, fix
- Implement query expansion to include synonyms like "return", "policy", and "reimbursement".
- Enable hybrid search (BM25 + Vector) to better handle keyword matching for specific policy terms.
Next steps
- Verify if the "return period" document is successfully indexed and not excluded by filters.
- Test the query "return period" manually in your vector DB console.
- Add "refund" and "return" to a synonym map in your search configuration.
Connects securely to your tools. The creator never sees your data.
About this skill
Most RAG fixes are wasted because the wrong layer gets blamed: prompts get rewritten when the retriever never returned the answer, rerankers get added when the question was ambiguous. This is the ten-minute triage that finds the failure first.
What you get
- A decision tree that stops at the first true branch: coverage, retrieval miss, ranking, generation.
- The evidence a real diagnosis needs (the query actually sent, the top-k chunks, the answer), and the finding to raise when it is missing.
- Ten checks ordered by cost, from "is the document even indexed" to a fresh-index comparison.
- A failure report template with the layer, the cause, the evidence and the smallest next test.
How it works
Take one failing case, print the retrieved chunks, read them by eye. Whether the answer is present, absent, or present but ignored tells you which layer to fix. Then change one thing and re-run the same case.
Who it is for
Anyone maintaining a retrieval system whose answers are sometimes wrong, incomplete or confidently invented.
Limits
It locates a failure; it does not measure how often it happens. One case tells you where to look, not a quality score.
Free starter. For a 40-case failure taxonomy, retrieval metrics you can compute without labels, adversarial query generation and a before/after report, see RAG Diagnostics: https://www.agensi.io/skills/rag-diagnostics
Locates single-case failures only. Does not measure aggregate quality or provide dataset-wide metrics. Requires raw retrieval logs (query, chunks, scores) to be effective.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 2 days ago
- Free with an Agensi account
Permissions
No special permissions declared or detected