Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Agent Quality Judge — Score AI Outputs Consistently

    1

    Turn vague quality judgments into repeatable rubrics, evidence-aware scores, calibrated comparisons, and actionable quality gates for AI outputs.

    $9.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Agent Quality Judge — Score AI Outputs Consistently

    Agent Quality Judge — Score AI Outputs Consistently

    Example session with this skill installed

    Evaluate two AI-generated answers to the same policy question. The answer must accurately reflect only the supplied excerpt, cite supporting sections, mention exceptions, avoid invented policy, and fit a support agent. Answer A and B are provided below with one official policy excerpt. Design a 0–4 anchored rubric with weights totaling 100, score each answer with quote-level evidence, distinguish factual errors from style preferences, flag unverifiable claims, report confidence and the decisive differences, and recommend a release threshold. If evidence is missing, say so.

    • Read your context and instructions
    • Compiled the agent quality judge

    Evaluation protocol: policy-grounded support answers

    Evaluation target

    Judge which answer is safer and more useful for a support agent using only the supplied policy excerpt. No external source or live system was consulted. Scores assess the provided text against the stated contract; they do not establish current policy beyond the excerpt.

    Weighted rubric (0–4 per criterion)

    CriterionWeight024
    Factual support and no invention35Material unsupported or contradicted claimsMostly supported; one material gap or ambiguityEvery substantive claim supported; no invented rule
    Exceptions and conditions20Omits or reverses decisive exceptionMentions some conditions but misses a relevant oneAll relevant conditions and exceptions represented
    Citation accuracy15Missing or misleading citationsCitations exist but support is incomplete or impreciseEach key claim linked to the correct excerpt section
    Instruction fit and completeness15Fails requested taskCovers main answer with notable omissionDirectly answers all requested parts at appropriate scope
    Support usability and clarity10Confusing or unsafe to useUnderstandable with editingClear, concise, actionable, and appropriately qualified
    Uncertainty handling5Presents unknowns as factsSome uncertainty is signaledClearly flags what the excerpt does not establish

    Scoring rule: criterion points = (0–4 score ÷ 4) × weight. Total out of 100. A score of 0 on factual support or citation accuracy is a critical failure regardless of total.

    Evidence record for each answer

    For each criterion, include: score; exact answer phrase; supporting or contradicting excerpt section; why the evidence meets the anchor; and missing evidence. If the excerpt is silent, label the claim

    unverifiable from supplied source rather than false or true. Separate objective policy defects from style preference.

    Pairwise decision

    Prefer an answer only when its advantage is supported by material differences under the rubric. If totals differ by less than 5 points and neither has a critical failure, mark the result

    inconclusive/tie and request human review. Do not treat fluent writing as evidence of accuracy.

    Release gate

    Recommend release only when: (1) no critical factual or citation failure exists; (2) weighted score is at least 85/100; (3) all applicable exceptions are covered; (4) no high-impact claim is unverifiable; and (5) required human review is complete. This is a proposed threshold; calibrate it on representative historical cases before operational use.

    Confidence and limitations

    Confidence is

    high only if the full answers and authoritative excerpt are present and all key claims can be mapped. It is

    medium when the excerpt is present but interpretation is ambiguous. It is

    low when the source, prompt, or complete answers are missing. No answer text or excerpt was actually included in this example request, so no candidate scores can be assigned yet.

    Calibration steps

    Have two reviewers score the same 20–30 representative cases independently. Compare criterion-level agreement, identify ambiguous anchors, revise the rubric, and repeat. Include contradictions, exceptions, unanswerable questions, citation traps, and style variations. Track critical-error rate and false-pass rate; do not tune solely for reviewer agreement if both reviewers miss the same defect. Revalidate after material prompt, model, policy, or workflow changes.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Make AI quality judgments explicit and repeatable

    Agent Quality Judge designs evaluation protocols for AI answers, research, summaries, reports, recommendations, plans, code, structured data, tool behavior, RAG responses, customer-support replies, marketing content, documents, images (when vision is available), and workflow outcomes. It can evaluate one output, compare pairs, rank candidates, score batches, calibrate reviewers, or architect a benchmark and release gate.

    What it helps build

    • Output contracts and task-specific quality dimensions
    • Rubrics with clear criteria, scoring anchors, weights, and evidence standards
    • Fact accuracy, instruction-following, completeness, relevance, safety, style, and usability checks
    • Pairwise comparisons, ranking protocols, confidence statements, and tie rules
    • Human-review workflows, disagreement analysis, and evaluator calibration
    • Representative benchmark sets, edge cases, regression tests, and release thresholds
    • Defect diagnosis and concrete revision guidance

    Evidence and fairness

    Scores are grounded only in the output, prompt, reference evidence, and evaluation tools actually supplied. Separate observable defects from subjective preferences and unsupported claims. Never say an output was fact-checked, executed, tested, or independently verified unless the required evidence exists. Do not fabricate scores for missing content. Use stable scoring anchors, blinded comparisons where practical, and uncertainty labels when evidence is insufficient.

    Best results

    Provide the output contract, intended audience, stakes, examples of acceptable and unacceptable work, authoritative references, scoring priorities, failure costs, and how the result will be used. Remove secrets and private data.

    Scores depend on the rubric, supplied evidence, evaluator capability, and task context. A rubric cannot guarantee objective truth or eliminate bias. Thresholds and scoring anchors require calibration on representative cases. Human judgment remains necessary for subjective, high-impact, ambiguous, or evidence-limited evaluations. No factual verification, execution, or testing is implied unless performed and documented.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Permissions

    Read Files

    No external service is required by default. Read only evaluation cases, outputs, rubrics, and references the user supplies or authorizes. Do not send submitted content to third-party services unless explicitly authorized. Treat personal, confidential, or proprietary content as sensitive; recommend redaction or synthetic examples.

    Listed1 month ago

    What's inside

    Frequently Asked Questions

    Only when the relevant authoritative sources or execution tools are actually available and used. Otherwise, it can assess consistency against supplied evidence, identify claims that need checking, and state what remains unverified. It must not claim independent verification from text review alone.

    New in Research & Analysis