Works with the AI Agents you already use
rag pipeline doctor
Diagnose and fix RAG retrieval failures, optimize chunking, and install evaluation loops for better AI accuracy.
Free
rag pipeline doctor
Example session with this skill installed
Our customer support bot is giving wrong answers for technical specs. I have the docs in Pinecone using standard chunking. Help me diagnose why it's missing specific part numbers.
- Read your context and instructions
- Compiled the rag pipeline doctor
I have analyzed the reported failure regarding technical specification retrieval. Based on the symptom of missing specific part numbers, the primary diagnosis points to a
Recall failure caused by a vocabulary gap in the vector-only search.
RAG Diagnostic Report
| Failure Layer | Status | Observation | Recommended Fix |
|---|---|---|---|
| Corpus | Pass | Source documents contain the part numbers. | N/A |
| Chunking | Warning | Tables are being split, losing header context. | Implement structure-aware splitting. |
| Recall | FAIL | Vector search fails on exact alphanumeric strings. | Enable Hybrid Search (BM25 + Vector). |
| Ranking | Warning | Top-k results are visually similar but irrelevant. | Add a Cross-Encoder Reranker. |
| Generation | Pass | Model tries to answer but lacks the right context. | Improve grounding prompt. |
Technical Implementation Plan
Hybrid Search: Integrate BM25 keyword matching alongside your Pinecone vector search. This ensures that exact part numbers (e.g., "TX-99-B") trigger a direct hit regardless of embedding distance.
2.
Structure-Aware Chunking: Re-index technical tables using Markdown-based splitting to keep row/column relationships intact.
3.
Reranking: Implement a cross-encoder/ms-marco-MiniLM-L-6-v2 model to re-score the top 50 retrieved chunks, passing only the top 5 to the LLM.
Next steps
- Provide a sample of 10 failed queries and their corresponding source snippets.
- Update the retrieval function to fusion-score BM25 and Vector results.
- Run a benchmark test to compare Recall@5 before and after these changes.
Connects securely to your tools. The creator never sees your data.
About this skill
RAG performance issues are rarely just "the model is hallucinating." Usually, the pipeline is diseased at the ingestion, retrieval, or ranking stage. This skill provides a structured diagnostic framework to isolate where data is being lost or misinterpreted and applies targeted fixes to the specific failure layer.
What it does
- Classifies failures by mapping symptoms to specific layers like corpus gaps, chunking errors, or recall mismatches.
- Audits indexing strategy by evaluating chunk boundaries, metadata attachment, and embedding model suitability for your domain.
- Optimizes retrieval by testing hybrid search (BM25 + vector) and cross-encoder reranking implementations.
- Validates generation by checking citation accuracy and enforcing refusal paths for out-of-bounds queries.
- Establishes evaluation loops by creating golden test sets and tracking recall@k and MRR metrics.
How it works
- Target queries: Build a test set of 20 real user questions paired with golden answers and source documents.
- Diagnostics: Run the test set through the retrieval layer to calculate recall and identify the exact failure point.
- Anchor review: Audit chunking, embeddings, and reranking logic based on the diagnostic data.
- Citation check: Verify the model sticks to context and cites sources correctly.
- Evaluation: Install a versioned eval loop to measure the impact of every pipeline change.
Frameworks & tools
Works with any RAG stack including LangChain, LlamaIndex, Pinecone, Weaviate, and Chroma. Supports evaluation of embedding models, cross-encoders, and hybrid search configurations.
Why this beats prompting it yourself
Prompting for better answers won't fix a pipeline where the relevant context never reaches the LLM. This skill focuses on the engineering architecture, ensuring the retrieval mechanics are mathematically sound before adjusting the final prompt.
Use cases
- Fix chatbots that provide incorrect information despite having the right documents.
- Improve search accuracy for technical documentation containing SKUs, error codes, or names.
- Reduce hallucinations by enforcing strict context grounding and refusal paths.
- Transition from prototype to production with a repeatable evaluation framework.
Known limitations
Requires a set of source documents and access to retrieval logs to perform diagnostics. Does not fix low-level infrastructure outages or network latency issues.
Requires source documents and retrieval logs. Focused on architectural pipeline fixes rather than hosting or infrastructure issues.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 22 days ago
- Free with an Agensi account
Permissions
No special permissions declared or detected