More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Citation Checker: Find unsupported claims in AI answers

    1

    **The problem** You have a draft, a RAG answer or a report summary with citations, plus the passages it was written from.

    $9

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Citation Checker: Find unsupported claims in AI answers

    Example session with this skill installed

    Check the six-sentence synthetic expense answer in references/example.json against its three passages. Run the strict gate, then read the sources for contradictions.

    • Read your context and instructions
    • Compiled the citation checker

    Captured stdout, exit 1, from python3 -I -B scripts/check_citations.py references/example.json --strict --format markdown:

    #SentenceCitationsSupportLabelReason
    1Submit expense reports with itemized receipts within 30 days of the purchase.expenses#deadline1.00supported9 of 9 content words found in the cited passages
    2Late reports are returned to the employee and the manager is notified.expenses#deadline0.50weakly_supported3 of 6 content words found in the cited passages (below 0.80)
    3Expenses above $1,000 need approval from the finance team.expenses#approval0.86unsupportednumbers not in the cited passages: 1000
    4Meals on business trips are reimbursed up to $60 per day.travel#meals (rejected: travel#lodging)0.86supported6 of 7 content words found in the cited passages; rejected citations: travel#lodging
    5Alcohol is reimbursed on business travel.travel#meals1.00supported4 of 4 content words found in the cited passages
    6Most employees find the process easy.-0.00unsupportedthe sentence has no citations

    Sentences: 6 | supported 3 | weakly supported 1 | unsupported 2 | citation coverage 83%

    Lexical heuristic only: review every label before relying on it.

    Reviewed verdict: Needs fixes. Sentence 3 says $1,000 where the source says $500. Sentence 5 reverses the source's "Alcohol is not reimbursed" even though the script marked it supported. Sentence 4's meals claim is fine, but its lodging citation points to a passage that wasn't supplied. Sentence 2 adds manager notification, and sentence 6 has no citation. Fix the limit and the alcohol rule first, then the citations and additions. The 83% coverage counts sentences with a valid citation. It says nothing about whether they're true.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem You have a draft, a RAG answer or a report summary with citations, plus the passages it was written from. The citations look fine at a glance, but one sentence quotes a number the source never gives, another cites an ID that isn't there, and a third says the opposite of its source. Reading every claim against every passage by hand is slow, and it's easy to miss things.

    See it before you buy: real input, real output Command (from the package folder, exit code 1, 0.02 s):

    python3 -I -B scripts/check_citations.py references/example.json --strict --format markdown
    

    Before: Passage and answer sentences (excerpt of references/example.json)

          "text": "Your team lead approves expenses up to $500. Anything above $500 also needs approval from the finance team."
          "text": "Expenses above $1,000 need approval from the finance team.",
          "citations": ["expenses#approval"]
          "text": "Alcohol is reimbursed on business travel.",
          "citations": ["travel#meals"]
    

    After: lines copied from the script's output (excerpt):

    | 2 | Late reports are returned to the employee and the manager is notified. | expenses#deadline | 0.50 | weakly_supported | 3 of 6 content words found in the cited passages (below 0.80) |
    | 3 | Expenses above $1,000 need approval from the finance team. | expenses#approval | 0.86 | unsupported | numbers not in the cited passages: 1000 |
    | 6 | Most employees find the process easy. | - | 0.00 | unsupported | the sentence has no citations |
    Sentences: 6 | supported 3 | weakly supported 1 | unsupported 2 | citation coverage 83%
    

    What the numbers show

    • Tested on RAGTruth: 86,566 sentences from 11,592 model-written answers, each checked against all of its source passages.
    • The default 'unsupported' flag caught 52% of annotator-marked sentences (3,745 of 7,157); flagging unsupported or weak caught 81% (5,802 of 7,157).
    • Honest limit: roughly four out of five flags were sentences the annotators hadn't marked, so the agent reads the sources instead of trusting the label.

    What you get

    • A small Python checker. For each sentence it checks that the cited IDs exist, how many of the sentence's words appear in the cited text, and whether every number in the sentence appears there too.
    • A report in JSON, Markdown or both, with a plain reason on every row ("3 of 6 content words found", "numbers not in the cited passages: 1000").
    • Instructions that make the agent read each cited passage before giving its own verdict, so you see the script's label and the reviewed verdict side by side, with fixes in priority order.
    • An optional --strict mode that exits 1 when any sentence isn't supported, so you can use it as a pipeline check. You can adjust the two cut-offs (0.80 and 0.50 by default).
    • 23 test cases, four saved sample outputs, a worked six-sentence example and the scoring script I used for the numbers below.

    Measured on real data (full detail) I ran it on RAGTruth, a public dataset where people marked the made-up parts of 11,592 model-written answers and summaries (86,566 sentences). Each sentence was checked against all of its source passages. The full run took about 40 seconds on one machine.

    What counts as a flagMarked sentences it caughtFlags that hit a marked sentence
    Labelled unsupported52% (3,745 of 7,157)19% (3,745 of 19,821)
    Unsupported or weak81% (5,802 of 7,157)14% (5,802 of 41,099)
    A number missing from the source22% (1,571 of 7,157)17% (1,571 of 9,335)

    In plain terms: the default flag finds about half of the problem sentences, and roughly four out of five flags are sentences the annotators didn't mark. That's why the agent reads the sources instead of trusting the label. I also read ten examples by hand. The number rule caught invented years and durations. The misses were things like a net worth assigned to the wrong person and a hedged finding turned into a flat denial, where every word still matched the source. The annotations cover whole answers, so splitting them into sentences adds some noise.

    Known limitations

    • Word overlap can't see meaning. A sentence that flips "not reimbursed" to "reimbursed", or credits a fact to the wrong person, can still score 1.00. Every row needs a human or agent read.
    • Expect a lot of false alarms. On the public data most flags were on sentences nobody marked, so treat --strict as a tripwire, not a hallucination detector.
    • It only checks text you supply. It never fetches URLs, doesn't confirm a paper exists, and needs PDFs, BibTeX or Markdown bibliographies turned into JSON first.
    • The word matching uses simple English rules. Other languages and heavy paraphrase score poorly.

    Everything runs locally, with no network, no telemetry and no files written unless you redirect the output. The agent may write a temporary input file.

    Who it's for Editors, researchers and teams building RAG systems who already have the source excerpts and want a repeatable first pass they can check line by line.

    Requires Python 3.8 or newer and an agent that can read files and run Python. The checker uses only the standard library. No API keys, no account.

    Check it yourself 23 test cases (passing, failing and invalid inputs) and four saved sample outputs, all re-run against the current script. From the package folder, try python3 -I -B scripts/check_citations.py references/example.json --strict --format markdown, or run the cases listed in references/checks.json. The RAGTruth figures can be reproduced with the included scoring script once you download the dataset. Steps are in references/benchmarks.md.

    Price and guarantee $9 one time, yours to use with any agent that can run Python. Covered by Agensi's money-back guarantee if it doesn't do what this page promises.

    Does not fetch URLs, parse PDFs, or verify source truth. Uses English stemming; not for semantic reasoning. Requires structured JSON input. Maximum row limits depend on available system memory.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 15 days ago

    Requires

    Python 3

    Permissions

    Terminal / Shell
    Read Files

    Uses file-reading and terminal tools to run Python on the supplied JSON file or standard input. The agent may write temporary input JSON; the checker only prints reports and writes no files itself. No network access, no telemetry, and nothing is sent anywhere. It never fetches URLs.

    Listed15 days ago
    Updated5 days ago

    What's inside

    Frequently Asked Questions

    Zotero stores and formats your references. It doesn't compare what a sentence says with the passage it cites, which is what this does. Many people will use both.

    On 86,566 RAGTruth sentences the default flag caught 52% of the marked ones, and only about one flag in five was on a marked sentence. It's a fast first pass that tells you where to look, not a verdict.

    The checker runs locally on UTF-8 JSON and sends nothing anywhere. Its report repeats your sentences, so store saved reports as carefully as the originals.

    Agensi's money-back guarantee covers every paid skill: if it doesn't do what this page promises, you get your money back. Everything above is output from the script in the package, so you can see what it does before you buy.

    New in LLM Ops