Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    goodhart

    1

    A red-teaming tool that identifies and fixes gameable criteria in LLM-as-judge rubrics and quality checklists.

    Free

    Security scanned
    goodhart

    goodhart

    Example session with this skill installed

    Audit my customer support rubric in rubric.md. It grades responses on accuracy, tone, and thoroughness. Use script mode with 5 runs.

    • Read your context and instructions
    • Compiled the goodhart

    I have completed the red-team audit of your customer support rubric using Script mode with 5 runs per probe. I identified 2 FOOLED criteria where the rubric rewards padding over actual utility.

    Rubric Audit Report: customer-support-v1

    CriterionStatusFinding
    C1: Accuracy🟢 ROBUSTPenalized 100% of confident-but-wrong probes.
    C2: Tone🟡 VULNERABLEFailed to penalize "over-politeness" that ignored the user's technical issue.
    C3: Thoroughness🔴 FOOLEDGamed probes with 200 words of filler scored 15% higher than concise, correct answers.

    Summary of Exploits

    The rubric currently equates "Thoroughness" with "Word Count." In 5/5 runs, a probe that repeated the user's question back to them (E01 Padding) received a perfect score, while the honest baseline was marked down for being "too brief" despite solving the ticket.

    Hardened Rubric Changes

    Applied H3 Length Neutrality: Updated C3 to explicitly state that length without new information should not increase the score.

    Negative Constraints: Added a penalty for "Rubric Echoing" where the model mimics the evaluation criteria in its response.

    Ranking Accuracy:

    • Dev Set: 0.72
    • Held-out Set: 0.88 (Verified against new "Self-Praise" exploits)

    Next steps

    1. Review the hardened version in goodhart-run/rubric-hardened.md.
    2. Replace your production judge prompt with the updated criteria.
    3. Run scripts/analyze.py in your CI pipeline to prevent future regression.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Evaluation rubrics are often flawed, rewarding superficial traits like length, confident tone, or formatting instead of genuine quality. This skill red-teams your LLM-as-judge rubrics, support checklists, or quality-review guides by attacking them like a malicious optimizer. It generates responses specifically engineered to game your criteria, grades them blind against honest baselines, and reports where the rubric fails.

    What it does

    • Decompose rubrics to identify gaps between what is intended to be measured and what is actually observable by a grader.
    • Generate gamed probes including padding, confident errors, and rubric echoing to stress-test specific criteria.
    • Blind scoring using repeated runs to ensure results are statistically significant rather than just noise.
    • Hardened rubric generation providing a revised version of your criteria with specific fixes for identified vulnerabilities.
    • Verification testing using a held-out set of attacks to prove the hardened rubric generalizes to new exploits.

    How it works

    1. Input the rubric and the specific task or policy it is designed to evaluate.
    2. Execute a probe run where the agent creates honest baselines and gamed responses.
    3. Run the judge in either Script mode (Python) or Portable mode (subagents) to score probes repeatedly.
    4. Analyze the gap to categorize criteria as FOOLED, VULNERABLE, or ROBUST.
    5. Review the report and the new, hardened rubric file.

    Frameworks & tools

    The skill is compatible with any LLM-as-judge setup. It includes Python scripts for local execution or can run entirely via agent subagents.

    Why this beats prompting it yourself

    Manual testing usually misses edge cases or relies on a single subjective pass. This skill uses a systematic red-teaming methodology with repeated blind trials and a rubric-free control judge to strip away bias.

    Use cases

    • Audit an LLM-as-judge prompt before deploying it to an automated evaluation pipeline.
    • Harden a customer support checklist to ensure agents are rewarded for solving problems, not just using empathy buzzwords.
    • Stress-test grading guides for technical documentation or code quality to prevent rewards for verbose, low-value output.

    Known limitations

    The skill is not tested for rubrics involving contested human judgments like hiring or admissions. It does not measure fairness across demographic groups. Scoring requires multiple model calls.

    Not tested for hiring or admissions. Does not test for demographic fairness. Requires multiple model runs for statistical significance. Results depend on the quality of the probe authoring.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 7 days ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Environment Variables
    Listed7 days ago

    What's inside

    Frequently Asked Questions

    Goodhart is designed for developers and AI operators using LLM-as-a-judge workflows who need to ensure their evaluation rubrics aren't being fooled by "lazy" or "gamed" model outputs like excessive verbosity or confident tone.

    The skill is model-agnostic and works with any LLM-as-a-judge setup, offering both a Python-based Script mode for local integration and a Portable mode that uses agent subagents for cloud-native workflows.

    You receive the core Goodhart agent skill, a suite of red-teaming probe templates, Python execution scripts, and a documentation guide for interpreting the vulnerability reports.

    While Goodhart identifies vulnerabilities in any qualitative rubric, it is currently not recommended or tested for high-stakes human judgments such as hiring, admissions, or demographic fairness audits.

    Setup is straightforward: you provide your existing rubric and the task description, then select your execution mode (Script or Portable) to begin the automated red-teaming cycles.

    Goodhart uses repeated blind scoring and a rubric-free control judge to ensure that identified vulnerabilities are statistically significant and not just the result of a single "lucky" model generation.

    New in LLM Ops