- Home
- Skills
- Research & Analysis
- Medical AI Study Appraisal Gate
Works with the AI Agents you already use
Medical AI Study Appraisal Gate
Is that 'AI beats doctors' study real? Appraise medical AI papers against CONSORT-AI, TRIPOD+AI, STARD-AI, CLAIM and CHART, then get accurate wording. CREDIBLE / PROMISING BUT LIMITED / OVERCLAIMED.
$29
Medical AI Study Appraisal Gate
Example session with this skill installed
A preprint says an AI diagnostic system 'outperformed physicians' on complex cases, and it's all over X. Here's the PDF. Is it credible, and how should I describe it in our newsletter?
- Read your context and instructions
- Compiled the medical ai study
Medical AI appraisal: diagnostic AI vs physicians on complex cases (2026-10-23)
Verdict: PROMISING BUT LIMITED. The design is sound for a benchmark, but "outperformed physicians" goes beyond what was tested.
Source appraised: full-text preprint plus supplement (not yet peer reviewed).
Study type: comparative accuracy on written case vignettes. Guidelines used: STARD-AI items where applicable, CHART for the LLM components, and the core checklist.
Domain concerns
| Domain | Concern | Evidence (location) | Why it matters |
|---|---|---|---|
| Cases | SOME | "cases drawn from a published case-challenge series" (Methods 2.1) | Curated, difficult teaching cases, not consecutive patients |
| Contamination | HIGH | Most cases were published before the model's training cutoff; no post-cutoff subset reported (Methods 2.1; Supp. Table 1) | The model may have seen the answers |
| Comparator | HIGH | 21 generalist physicians, no references or colleagues allowed (Methods 2.4) | Not how physicians actually work on hard cases |
| Reference standard | LOW | Published final diagnoses, adjudicated by 2 blinded specialists (Methods 2.5) | Strong |
| Metric | SOME | Primary result reported as "correct diagnosis in final answer"; top-10 inclusion featured in the abstract (Results 3.1) | Metric emphasis inflates the gap |
| Validation | HIGH | Single case source; no external or prospective evaluation | Generalisability unknown |
| Real-world outcomes | HIGH | None studied | Accuracy is not patient benefit |
| Conflicts | SOME | Most authors employed by the developer (Disclosures) | Independent replication needed |
| Spin | SOME | Press release says "will reduce misdiagnosis" | Causal claim not supported by the design |
Strengths: pre-specified primary outcome, blinded adjudication, code and prompts shared.
What it shows / doesn't show
It shows that on curated published cases the system named the final diagnosis more often than constrained generalists. It doesn't show better care, safety, or performance on real patients.
Plain-language summary (newsletter)
"In a study not yet peer reviewed, an AI system was tested on difficult published medical cases and identified the final diagnosis more often than a group of general doctors working without their usual resources. The cases may have been in the AI's training data, and no real patients were involved, so it's too early to say whether the system would improve diagnosis in practice."
Accurate wording
- Headline: "AI system tops generalist doctors on published case challenges, in early, unreviewed study"
- Avoid: "AI beats doctors", "will reduce misdiagnosis", "ready for clinics"
Questions for the authors
- Performance on cases published after the training cutoff?
- Results against specialists working with their usual resources?
- Any plans for prospective evaluation with patient outcomes?
Connects securely to your tools. The creator never sees your data.
About this skill
Every week a new headline says an AI system outperformed doctors. Sometimes it's real. Often the AI was tested on published teaching cases it may already have seen in training, against generalists working without their usual references, scored on a generous metric, with no real patients and no outcomes. Journalists, medical communicators, hospital AI committees, investors and clinicians all need to know which is which, and how to describe it without overclaiming.
This skill makes your AI agent a rigorous medical publications reviewer, built by a CMPP-certified, PhD medical writer. It includes a guideline map, a 12-domain appraisal checklist and a spin-pattern guide. It:
- Works from the full paper, supplement and registration, not the press release. If only an abstract or press release is available, it says so and caps the verdict
- Picks the right AI reporting guideline for the study type: CONSORT-AI and SPIRIT-AI for trials, TRIPOD+AI and PROBAST+AI for prediction models, STARD-AI for diagnostic accuracy, DECIDE-AI for early clinical evaluation, CLAIM for imaging, CHART for chatbot health-advice studies
- Appraises 12 domains with quotes and locations: intended use, case selection and spectrum, training-data contamination and leakage, comparator fairness, reference standard, outcome metrics and uncertainty, external validation, subgroup performance, human-AI interaction, real-world outcomes, transparency, and spin
- Separates three questions: is the study internally valid, does it generalise to real care, and does the headline match the evidence
- Gives a verdict, CREDIBLE, PROMISING BUT LIMITED or OVERCLAIMED, and credits strengths as clearly as it flags weaknesses
- Writes a plain-language summary, accurate headline and claim options with phrases to avoid, a fair-balance slide claim for medical communications, and pointed questions for the authors or developer
For medical writers and MedComms teams, health journalists, hospital AI and procurement committees, clinicians, health-tech investors and analysts, and educators teaching critical appraisal.
Honesty built in: it never invents results, sample sizes, guideline items or citations, marks anything it could not verify, and does not reproduce guideline text at length. It is critical appraisal for communication and decision support, not clinical advice.
Appraises design and reporting, not raw data. Guideline versions change; check EQUATOR for the current one. Verdict is capped when only an abstract or press release is available. Not clinical advice.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Requires