Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Medical AI Study Appraisal Gate

    1

    Is that 'AI beats doctors' study real? Appraise medical AI papers against CONSORT-AI, TRIPOD+AI, STARD-AI, CLAIM and CHART, then get accurate wording. CREDIBLE / PROMISING BUT LIMITED / OVERCLAIMED.

    $29

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Medical AI Study Appraisal Gate

    Medical AI Study Appraisal Gate

    Example session with this skill installed

    A preprint says an AI diagnostic system 'outperformed physicians' on complex cases, and it's all over X. Here's the PDF. Is it credible, and how should I describe it in our newsletter?

    • Read your context and instructions
    • Compiled the medical ai study

    Medical AI appraisal: diagnostic AI vs physicians on complex cases (2026-10-23)

    Verdict: PROMISING BUT LIMITED. The design is sound for a benchmark, but "outperformed physicians" goes beyond what was tested.
    Source appraised: full-text preprint plus supplement (not yet peer reviewed).
    Study type: comparative accuracy on written case vignettes. Guidelines used: STARD-AI items where applicable, CHART for the LLM components, and the core checklist.

    Domain concerns

    DomainConcernEvidence (location)Why it matters
    CasesSOME"cases drawn from a published case-challenge series" (Methods 2.1)Curated, difficult teaching cases, not consecutive patients
    ContaminationHIGHMost cases were published before the model's training cutoff; no post-cutoff subset reported (Methods 2.1; Supp. Table 1)The model may have seen the answers
    ComparatorHIGH21 generalist physicians, no references or colleagues allowed (Methods 2.4)Not how physicians actually work on hard cases
    Reference standardLOWPublished final diagnoses, adjudicated by 2 blinded specialists (Methods 2.5)Strong
    MetricSOMEPrimary result reported as "correct diagnosis in final answer"; top-10 inclusion featured in the abstract (Results 3.1)Metric emphasis inflates the gap
    ValidationHIGHSingle case source; no external or prospective evaluationGeneralisability unknown
    Real-world outcomesHIGHNone studiedAccuracy is not patient benefit
    ConflictsSOMEMost authors employed by the developer (Disclosures)Independent replication needed
    SpinSOMEPress release says "will reduce misdiagnosis"Causal claim not supported by the design

    Strengths: pre-specified primary outcome, blinded adjudication, code and prompts shared.

    What it shows / doesn't show

    It shows that on curated published cases the system named the final diagnosis more often than constrained generalists. It doesn't show better care, safety, or performance on real patients.

    Plain-language summary (newsletter)

    "In a study not yet peer reviewed, an AI system was tested on difficult published medical cases and identified the final diagnosis more often than a group of general doctors working without their usual resources. The cases may have been in the AI's training data, and no real patients were involved, so it's too early to say whether the system would improve diagnosis in practice."

    Accurate wording

    • Headline: "AI system tops generalist doctors on published case challenges, in early, unreviewed study"
    • Avoid: "AI beats doctors", "will reduce misdiagnosis", "ready for clinics"

    Questions for the authors

    1. Performance on cases published after the training cutoff?
    2. Results against specialists working with their usual resources?
    3. Any plans for prospective evaluation with patient outcomes?

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Every week a new headline says an AI system outperformed doctors. Sometimes it's real. Often the AI was tested on published teaching cases it may already have seen in training, against generalists working without their usual references, scored on a generous metric, with no real patients and no outcomes. Journalists, medical communicators, hospital AI committees, investors and clinicians all need to know which is which, and how to describe it without overclaiming.

    This skill makes your AI agent a rigorous medical publications reviewer, built by a CMPP-certified, PhD medical writer. It includes a guideline map, a 12-domain appraisal checklist and a spin-pattern guide. It:

    • Works from the full paper, supplement and registration, not the press release. If only an abstract or press release is available, it says so and caps the verdict
    • Picks the right AI reporting guideline for the study type: CONSORT-AI and SPIRIT-AI for trials, TRIPOD+AI and PROBAST+AI for prediction models, STARD-AI for diagnostic accuracy, DECIDE-AI for early clinical evaluation, CLAIM for imaging, CHART for chatbot health-advice studies
    • Appraises 12 domains with quotes and locations: intended use, case selection and spectrum, training-data contamination and leakage, comparator fairness, reference standard, outcome metrics and uncertainty, external validation, subgroup performance, human-AI interaction, real-world outcomes, transparency, and spin
    • Separates three questions: is the study internally valid, does it generalise to real care, and does the headline match the evidence
    • Gives a verdict, CREDIBLE, PROMISING BUT LIMITED or OVERCLAIMED, and credits strengths as clearly as it flags weaknesses
    • Writes a plain-language summary, accurate headline and claim options with phrases to avoid, a fair-balance slide claim for medical communications, and pointed questions for the authors or developer

    For medical writers and MedComms teams, health journalists, hospital AI and procurement committees, clinicians, health-tech investors and analysts, and educators teaching critical appraisal.

    Honesty built in: it never invents results, sample sizes, guideline items or citations, marks anything it could not verify, and does not reproduce guideline text at length. It is critical appraisal for communication and decision support, not clinical advice.

    Appraises design and reporting, not raw data. Guideline versions change; check EQUATOR for the current one. Verdict is capped when only an abstract or press release is available. Not clinical advice.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Requires

    Study PDF or full text

    Permissions

    Read Files
    Network Access
    Listed24 days ago

    What's inside

    Frequently Asked Questions

    Any agent that supports the SKILL.md standard, including Claude Code, Codex CLI, Cursor, VS Code Copilot and Gemini CLI. It works best when the agent can read the PDF and browse to check guideline versions.

    SKILL.md with the appraisal method, verdict rules and report format; a guideline map matching study types to CONSORT-AI, SPIRIT-AI, TRIPOD+AI, PROBAST+AI, STARD-AI, DECIDE-AI, CLAIM and CHART; a 12-domain appraisal checklist; and a spin-pattern guide with accurate alternative wording.

    Medical writers and MedComms teams, health journalists, hospital AI and procurement committees, clinicians, health-tech investors and analysts, and educators teaching critical appraisal.

    Yes. It applies CHART items for chatbot health-advice studies and checks LLM-specific issues: training-data contamination, model version, prompts, number of runs and variance.

    It will appraise what is available, state that in the first line, and cap the verdict at PROMISING BUT LIMITED until the full paper is reviewed.

    It appraises reporting and design, not raw data, so it cannot detect fabricated data. Guidelines are periodically updated, so check the current version. It is not clinical advice.

    New in Research & Analysis