Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    RAG Diagnostics

    1

    Separate query, retrieval and generation failures with evidence, then fix the layer that is actually broken.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    RAG Diagnostics

    RAG Diagnostics

    Example session with this skill installed

    "It hallucinates" is a symptom with three possible causes, and teams usually fix the wrong one: they swap the model when the chunking is broken, or rewrite prompts when the index is missing the answer.

    RAG Diagnostics separates the layers with evidence and tells you which one is failing, how often, and what to change first: query failure, retrieval failure, or generation failure.

    What it does

    • Labels real queries by whether the gold answer was retrieved and whether the answer used it, giving a 2x2 that splits failures by layer.
    • Distinguishes query failure from retrieval failure by rephrasing the query: if the rewrite finds the gold chunk, the index was fine.
    • Separates unsupported answers from wrong ones, which is the difference between a grounding problem and a knowledge problem.
    • Finds the pattern inside each class — tables, new documents, follow-ups, comparative questions — because "recall is 0.62" is not actionable.
    • Prescribes one fix per class with expected effect, then turns the failures into a regression set.

    What you get

    A verdict (X% retrieval, Y% generation, Z% query), findings with query IDs and chunk scores, a ranked fix table, and the regression set that keeps the fix from being undone by the next prompt tweak.

    Honest scope

    It needs retrieval logs and, ideally, gold answers. Without them it says the analysis is qualitative rather than inventing recall numbers. It does not re-index your corpus or benchmark vector databases.

    No dependencies. Works with the logs you already have.

    • Read your context and instructions
    • Compiled the rag diagnostics

    RAG diagnostics: docs assistant

    Verdict

    61% of bad answers are retrieval, 24% generation, 15% query.
    The highest-impact fix is semantic chunking for tables, followed by hybrid search for IDs.

    Sample

    120 queries, support logs, last 30 days. Gold answers available for 98.

    Layer split

    LayerShareEvidence
    Retrieval61%Gold chunk absent from the top 10 in 60 cases
    Generation24%Gold chunk present, answer contradicts it
    Query15%Follow-up questions with no conversation context

    Findings

    R1 — Tables are split across chunks

    Evidence: 22 failures, all on pricing and spec tables; chunk boundaries fall mid-row.
    Fix: semantic chunking with 15% overlap, tables kept whole. Expected: 22 failures recovered.
    Verify: re-run the 98 queries, compare gold-present rate.

    R2 — Documents added after the last reindex are invisible

    Evidence: recall 0.31 on documents newer than 14 days.
    Fix: nightly reindex. Expected: 11 failures recovered.

    G1 — Comparative answers ignore sources

    Evidence: 14 cases where the right chunk was retrieved and the model answered from memory.
    Fix: source-before-question ordering plus citation requirement and an abstain path.

    Fix order

    #FixFailuresEffort
    1Semantic chunking for tables22M
    2Hybrid BM25 for IDs and exact terms18L
    3Citation-required prompt with abstain14L
    4Nightly reindex11L
    5Context-aware query rewriting9M

    Regression set

    The 98 labelled queries committed as the baseline; every change re-runs it.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    RAG diagnostics: separate query, retrieval and generation failures with evidence, then fix the layer that is actually broken.

    Use this skill when:

    • RAG answers are wrong
    • the assistant hallucinates
    • retrieval quality is unclear
    • work out why documents are missed

    Who it's for: Teams whose RAG assistant gives bad answers.

    Example request

    Our docs assistant makes things up. Where is the fault?

    What the example returns

    • 61% retrieval, 24% generation
    • Tables recall 0.31
    • Fix order with case counts

    In the package: five files — SKILL.md with the trigger and workflow, README.md, references/METHOD.md (the full method), references/CHECKLIST.md (the pass/fail gate) and examples/EXAMPLE.md (a worked run you can copy).

    Works with Claude Code, Cursor, Codex CLI and any agent that reads SKILL.md. Plain markdown: no API key, no account, no network access.

    RAG Diagnostics

    Requires retrieval logs (chunk IDs and scores) to function. Cannot assess content gaps if the answer is entirely missing from the corpus. Does not perform re-indexing or code execution.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 2 days ago

    Permissions

    Read Files
    Listed3 days ago
    Updated2 days ago

    What's inside

    Frequently Asked Questions

    You need query logs, retrieval scores, and a small sample of 'gold' (correct) answers; the skill uses these to map your system state and isolate where context is being lost or ignored.

    No, this skill is framework-agnostic and works with any stack, including LangChain, LlamaIndex, or custom setups, as long as you can export your retrieval and generation data.

    Unlike basic prompts, this skill uses a systematic 2x2 matrix to distinguish between 'missing context' and 'ignored context,' ensuring you don't waste time rewriting prompts when the issue is actually your chunking strategy.

    The diagnostic identifies the root cause and provides a prioritized fix plan, but it does not directly modify your code or re-index your database; you maintain full control over the implementation.

    You receive the diagnostic framework, the automated auditing logic, and the regression scaffolding tool that turns identified failures into a permanent evaluation dataset for future testing.

    New in LLM Ops