- Home
- Skills
- Research & Analysis
- Agent Quality Judge — Score AI Outputs Consistently
Works with the AI Agents you already use
Agent Quality Judge — Score AI Outputs Consistently
Turn vague quality judgments into repeatable rubrics, evidence-aware scores, calibrated comparisons, and actionable quality gates for AI outputs.
$9.99
Agent Quality Judge — Score AI Outputs Consistently
Example session with this skill installed
Evaluate two AI-generated answers to the same policy question. The answer must accurately reflect only the supplied excerpt, cite supporting sections, mention exceptions, avoid invented policy, and fit a support agent. Answer A and B are provided below with one official policy excerpt. Design a 0–4 anchored rubric with weights totaling 100, score each answer with quote-level evidence, distinguish factual errors from style preferences, flag unverifiable claims, report confidence and the decisive differences, and recommend a release threshold. If evidence is missing, say so.
- Read your context and instructions
- Compiled the agent quality judge
Evaluation protocol: policy-grounded support answers
Evaluation target
Judge which answer is safer and more useful for a support agent using only the supplied policy excerpt. No external source or live system was consulted. Scores assess the provided text against the stated contract; they do not establish current policy beyond the excerpt.
Weighted rubric (0–4 per criterion)
| Criterion | Weight | 0 | 2 | 4 |
|---|---|---|---|---|
| Factual support and no invention | 35 | Material unsupported or contradicted claims | Mostly supported; one material gap or ambiguity | Every substantive claim supported; no invented rule |
| Exceptions and conditions | 20 | Omits or reverses decisive exception | Mentions some conditions but misses a relevant one | All relevant conditions and exceptions represented |
| Citation accuracy | 15 | Missing or misleading citations | Citations exist but support is incomplete or imprecise | Each key claim linked to the correct excerpt section |
| Instruction fit and completeness | 15 | Fails requested task | Covers main answer with notable omission | Directly answers all requested parts at appropriate scope |
| Support usability and clarity | 10 | Confusing or unsafe to use | Understandable with editing | Clear, concise, actionable, and appropriately qualified |
| Uncertainty handling | 5 | Presents unknowns as facts | Some uncertainty is signaled | Clearly flags what the excerpt does not establish |
Scoring rule: criterion points = (0–4 score ÷ 4) × weight. Total out of 100. A score of 0 on factual support or citation accuracy is a critical failure regardless of total.
Evidence record for each answer
For each criterion, include: score; exact answer phrase; supporting or contradicting excerpt section; why the evidence meets the anchor; and missing evidence. If the excerpt is silent, label the claim
unverifiable from supplied source rather than false or true. Separate objective policy defects from style preference.
Pairwise decision
Prefer an answer only when its advantage is supported by material differences under the rubric. If totals differ by less than 5 points and neither has a critical failure, mark the result
inconclusive/tie and request human review. Do not treat fluent writing as evidence of accuracy.
Release gate
Recommend release only when: (1) no critical factual or citation failure exists; (2) weighted score is at least 85/100; (3) all applicable exceptions are covered; (4) no high-impact claim is unverifiable; and (5) required human review is complete. This is a proposed threshold; calibrate it on representative historical cases before operational use.
Confidence and limitations
Confidence is
high only if the full answers and authoritative excerpt are present and all key claims can be mapped. It is
medium when the excerpt is present but interpretation is ambiguous. It is
low when the source, prompt, or complete answers are missing. No answer text or excerpt was actually included in this example request, so no candidate scores can be assigned yet.
Calibration steps
Have two reviewers score the same 20–30 representative cases independently. Compare criterion-level agreement, identify ambiguous anchors, revise the rubric, and repeat. Include contradictions, exceptions, unanswerable questions, citation traps, and style variations. Track critical-error rate and false-pass rate; do not tune solely for reviewer agreement if both reviewers miss the same defect. Revalidate after material prompt, model, policy, or workflow changes.
Connects securely to your tools. The creator never sees your data.
About this skill
Make AI quality judgments explicit and repeatable
Agent Quality Judge designs evaluation protocols for AI answers, research, summaries, reports, recommendations, plans, code, structured data, tool behavior, RAG responses, customer-support replies, marketing content, documents, images (when vision is available), and workflow outcomes. It can evaluate one output, compare pairs, rank candidates, score batches, calibrate reviewers, or architect a benchmark and release gate.
What it helps build
- Output contracts and task-specific quality dimensions
- Rubrics with clear criteria, scoring anchors, weights, and evidence standards
- Fact accuracy, instruction-following, completeness, relevance, safety, style, and usability checks
- Pairwise comparisons, ranking protocols, confidence statements, and tie rules
- Human-review workflows, disagreement analysis, and evaluator calibration
- Representative benchmark sets, edge cases, regression tests, and release thresholds
- Defect diagnosis and concrete revision guidance
Evidence and fairness
Scores are grounded only in the output, prompt, reference evidence, and evaluation tools actually supplied. Separate observable defects from subjective preferences and unsupported claims. Never say an output was fact-checked, executed, tested, or independently verified unless the required evidence exists. Do not fabricate scores for missing content. Use stable scoring anchors, blinded comparisons where practical, and uncertainty labels when evidence is insufficient.
Best results
Provide the output contract, intended audience, stakes, examples of acceptable and unacceptable work, authoritative references, scoring priorities, failure costs, and how the result will be used. Remove secrets and private data.
Scores depend on the rubric, supplied evidence, evaluator capability, and task context. A rubric cannot guarantee objective truth or eliminate bias. Thresholds and scoring anchors require calibration on representative cases. Human judgment remains necessary for subjective, high-impact, ambiguous, or evidence-limited evaluations. No factual verification, execution, or testing is implied unless performed and documented.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago
Permissions
No external service is required by default. Read only evaluation cases, outputs, rubrics, and references the user supplies or authorizes. Do not send submitted content to third-party services unless explicitly authorized. Treat personal, confidential, or proprietary content as sensitive; recommend redaction or synthetic examples.