More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Model Output Drift Monitor

    1

    Detect, measure, and act on drift in model outputs before users notice it.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Model Output Drift Monitor

    Model Output Drift Monitor

    Example session with this skill installed

    Apply the skill to this case: A support-answer model began giving shorter answers. Complaints rose slowly over two weeks, but no single answer looked obviously wrong.

    • Read your context and instructions
    • Compiled the model output drift

    Situation: A support-answer model began giving shorter answers. Complaints rose slowly over two weeks, but no single answer looked obviously wrong. Evidence: Baseline mean length was 142 words with a 6 percent refusal rate. Current mean length was 98 words with a 14 percent refusal rate, concentrated in billing questions. No model version change had occurred, but a retrieval source had been replaced nine days earlier. Decision: The drift was traced to the new retrieval source returning shorter passages, not to the model. The team reverted that source for billing questions and kept the new source elsewhere. Outcome: Mean length recovered to 138 words and refusal rate fell to 7 percent within three days. The monitor now runs weekly, with the billing segment flagged separately.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Model outputs rarely fail loudly. They slide: more refusals, shorter answers, shifted labels, rising toxicity flags, or a slow change in score distribution. Teams notice weeks later, after bad outputs reached users, and then argue about whether anything really changed. This skill turns that argument into a measured comparison between a frozen baseline and current behaviour, with a decision about what to do next. This skill gives you a counted diagnosis, a decision table, a fill-in template, a checklist, and a scoring rubric, plus a runnable script (drift_report.py) that Reads baseline and current CSV files with a score column and writes a markdown drift report with means, spread, shift, and a flag at a configurable threshold. It is built for the bottleneck moment: when the usual quick fix has already failed, or when a silent failure is worse than a loud one. Follow the protocol in order, record numbers rather than impressions, and use the quality gate before you call the work done.

    Results depend on the quality of your evidence set: a missing baseline, a small or unrepresentative sample, or outputs that were never labelled will limit the diagnosis, and the skill says so rather than filling gaps with guesses. The script is a helper for counting and reporting; it does not replace judgement about safety, permission, or whether a fix is correct. Thresholds are starting points to adapt to your context, not universal constants.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 23 days ago

    Permissions

    Terminal / Shell
    Write Files
    Listed23 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    At least 200 representative inputs, more if traffic is diverse. Below 100, treat every comparison as provisional and widen the set before acting.

    Build one now from current outputs plus a human-labelled sample, freeze it, and start monitoring from that point. Do not backfill a baseline from memory.

    When previously blocked or unsafe outputs appear, when a regulated decision changes materially, or when a core user journey breaks. Escalate those the same day.