Works with the AI Agents you already use
Diagnosing RAG Failure Modes Skill
Diagnose silent RAG failures, hallucinated citations, and multi-hop reasoning gaps in retrieval pipelines.
$9.99
Diagnosing RAG Failure Modes Skill
Example session with this skill installed
I have a RAG failure. Query: 'How did the 2023 tax change affect Q4 revenue?' The answer says it decreased by 5%, but I can't find that in the retrieved chunks. Here is the trace.
- Read your context and instructions
- Compiled the diagnosing rag failure
The diagnosis indicates a
Grounding Failure combined with a
Ranking Miss. While the model generated a specific figure, the retrieved evidence only discussed general tax policy without Q4 revenue data.
RAG Diagnostic Report: Case #882
| Hop | Sub-question | Required evidence | Retrieved evidence | Status |
|---|---|---|---|---|
| 1 | 2023 Tax Law changes | Specific provisions | Passage ID: doc_442 | Supported |
| 2 | Q4 Revenue figures | Financial statements | None found | Missing |
| 3 | Causal link | Impact analysis | None found | Missing |
Executive Diagnosis
The system suffered a silent failure where the generator hallucinated a 5% decrease to satisfy the user's causal intent. The retrieval stage successfully found the tax law but failed to retrieve the corresponding Q4 financial report, likely because the financial document's embedding was not semantically similar to the "tax change" query.
Pipeline Findings
- Retrieval: Failed to fetch financial documents. Top-k (5) was filled with general tax commentary.
- Ranking: A relevant summary document was ranked at position #12, falling outside the context window.
- Generation: The model filled the evidentiary gap with prior knowledge/hallucination rather than abstaining.
Recommended Fixes
Hybrid Retrieval: Implement BM25 keyword search alongside vector search to capture specific entities like "Q4" and "Revenue".
2.
Query Expansion: Rewrite the query to explicitly include "financial results" when fiscal years or quarters are mentioned.
3.
Abstention Logic: Update the system prompt to require a "Data not available" response if Hop 2 evidence is missing.
Next steps
- Increase
top_kto 20 and implement a Reranker to surface the financial summaries. - Add a validation step to check if retrieved chunks contain numerical values before generating financial answers.
Connects securely to your tools. The creator never sees your data.
About this skill
RAG systems often fail silently by returning fluent, authoritative answers that lack actual evidentiary support. This skill provides a systematic framework to decompose your retrieval-augmented generation pipeline and identify exactly where the logic breaks.
What it does
- Multi-hop analysis identifies missing intermediate facts or relationships in complex queries.
- Stage-by-stage inspection isolates failures in query rewriting, vector retrieval, reranking, or context assembly.
- Grounding verification maps individual answer claims to specific retrieved passages to detect hallucinations.
- Causal auditing distinguishes between documented causality and inferred correlation in model responses.
- Failure taxonomy classifies issues into 12 distinct categories like chunk-boundary errors or metadata filter misses.
How it works
- Trace capture assembles the full lifecycle of a query including retrieved chunk IDs, similarity scores, and the final prompt.
- Evidence chain mapping defines the required facts and logical hops necessary to answer the user's intent.
- Pipeline traversal tests each stage sequentially to find the first point of failure.
- Fix recommendation generates targeted interventions for the specific root cause discovered.
Frameworks & tools
Works with any RAG stack including LangChain, LlamaIndex, or custom implementations using vector databases like Pinecone, Weaviate, or Milvus. It is model-agnostic and applies to any LLM-based generation.
Why this beats prompting it yourself
Generic prompts often miss "silent" failures where the answer looks correct but the citations are hollow. This skill enforces a rigorous evidence-chain methodology that forces the agent to prove every link in the reasoning path.
Use cases
- Debugging why a RAG system ignores relevant documents already present in the vector store.
- Investigating "hallucinated" connections between two unrelated but correctly retrieved facts.
- Optimizing chunking strategies for technical manuals where context is lost across headers.
- Auditing citation accuracy to ensure links actually support the claims they are attached to.
Known limitations
Does not perform automated code execution to fix pipelines. Requires manual provision of traces or logs. Accuracy depends on the availability of retrieved context and metadata for inspection.
Requires manual provision of query traces and retrieved chunks. Does not automatically execute code fixes. Effectiveness limited if pipeline logs are incomplete or redacted.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 14 days ago
Permissions
No special permissions declared or detected