Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    RAG Reality Check

    1

    Audits what a RAG system really does, not what its docs or scores claim: the executed code path, silent failures, retrieval measured on its own, and the blind spots of metrics like faithfulness.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    RAG Reality Check

    RAG Reality Check

    Example session with this skill installed

    Audit our RAG assistant in ./app before we present it to the client. The README says hybrid search with an HNSW index and a faithfulness of 0.9.

    • Read your context and instructions
    • Compiled the rag reality check

    An audit report: claims table (e.g. "HNSW index: false, exact search on the executed path, index only reached by dead code"), the executed path of one query, silent failures ranked worst first, hit@k / recall@k / MRR per probe type on a stated number of probes, the cross-table showing questions scored high on faithfulness although retrieval missed, failure modes, security notes and ranked recommendations.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A RAG pipeline can score 0.9 on faithfulness while retrieving the wrong documents. Faithfulness checks that the answer sticks to the context it was given, not that the context was the right one, nor that the answer is correct. This skill audits a RAG system the way a sceptical reviewer would:

    • Claims vs reality. Every claim from the docs (hybrid search, vector index, tool use, scores) ends as verified, false or unverified, with file:line or an observed request payload as evidence.
    • Executed path. Follows one real query through the code actually called, and spots dead code the docs describe as active.
    • Silent failures. A Python AST scanner ranks exception handlers that swallow errors, print instead of logging or return empty values: the kind that turns an outage into "no content found".
    • Retrieval measured on its own. A probe set covering answerable, out-of-scope, cross-lingual, near-topic and multi-hop questions; a script computes hit@k, recall@k and MRR per probe type, detects a retriever with no relevance threshold, and crosses retrieval with your answer metric to show where the metric is blind.
    • Security quick pass and a structured report, with every figure tied to its command and probe count.

    Built from a real audit. Standard-library Python only; works with any RAG stack (you export retrieved ids per question).

    The silent-failure scanner reads Python only; other languages are checked by hand following the same patterns. Retrieval figures depend on the probe set: expected sources must be written by someone who knows the corpus. You need to export retrieved ids per question from your system (guidance included). Model-judged metrics are not computed by the skill: it crosses the ones you already have with retrieval.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 17 days ago

    Requires

    Python 3.8+

    Permissions

    File Scopes

    Reads
    the
    source
    code
    of
    RAG
    system
    under
    audit
    (never
    modifies
    it)
    Writes
    probe
    set
    results
    and
    report
    to
    a
    folder
    you
    choose

    No network access in the scripts. Exporting results may require running your own system, in a throwaway branch.

    Listed17 days ago

    What's inside

    Frequently Asked Questions

    Audits what a RAG system really does, not what its docs or scores claim: the executed code path, silent failures, retrieval measured on its own, and the blind spots of metrics like faithfulness.

    This skill uses the open SKILL.md standard. Use it through an MCP-compatible agent connected to Agensi, and check the compatibility information on this page for confirmed agents and required capabilities.

    Connect your AI agent to Agensi through MCP, then ask it to use RAG Reality Check on your task. Your agent loads the latest approved version each time. Paid skills require a purchase or active subscription. Skill files are not available for download; your licence does not allow you to save, copy or export any part of a skill.

    New in LLM Ops