Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    data drift and model monitoring analyst

    1

    ML models in production often fail silently due to data drift and performance decay.

    $2.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    data drift and model monitoring analyst

    data drift and model monitoring analyst

    Example session with this skill installed

    Analyze feature drift for 'credit_score' and 'annual_income' using training baseline E1 and production window E2. Baseline counts for income: [0-50k: 500, 50k+: 500]. Production: [0-50k: 400, 50k+: 600].

    • Read your context and instructions
    • Compiled the data drift

    I have processed the monitoring aggregates for the requested features against the training baseline.

    Data Drift and Model Monitoring Report — Financial Services, 2023-10-27

    1. Scope, sources and access

    • Sources: E1 (Training Baseline), E2 (Production Window)
    • Windows: Baseline 2023-01-01 to 2023-06-30; Production 2023-09-01 to 2023-09-30
    • Primary Metric: n/a (Drift analysis only)
    • Assumptions: PSI thresholds (0.10/0.25) are system defaults.

    2. Feature drift

    FeatureBaseline nProduction nPSITotal variationJS divergenceMean shift zLevel
    annual_income100010000.0820.1000.020n/aNominal
    credit_score100010000.1250.1500.035n/aWarning

    3. Performance and subgroup checks

    No performance data was supplied for this window.

    4. Observability gaps

    • Information gap — organizational validation required: No outcome data detected for production window E2.

    5. Drift alerts

    IdTypeSeveritySubjectValueThresholdSource
    ALT-001DRIFT_WARNWarningcredit_score0.1250.100Default Policy

    6. Retraining review (draft)

    • Triggers: Feature drift warning on credit_score.
    • Approval Status: Pending.
    • Management decision required: No retraining was started; this review is a draft.

    7. Warnings, limits and items for human review

    • W-002: Mean shift Z-score unavailable for binned data without summary statistics.
    • Observed PSI for credit_score (0.125) exceeds the nominal threshold of 0.10.
    • Distribution shift in annual_income remains within acceptable policy limits.

    Next steps

    1. Review the credit_score drift alert to determine if the shift reflects a change in applicant population.
    2. Provide outcome data (TP/FP counts) to evaluate if drift is impacting model precision.
    3. Validate the annual_income bins to ensure they align with recent economic reporting.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    ML models in production often fail silently due to data drift and performance decay. This skill provides a deterministic method to detect these shifts by comparing production aggregates against training baselines using exact statistical distance measures.

    What it does

    • Calculate drift metrics for every feature using Population Stability Index (PSI), Total Variation, and Jensen-Shannon divergence.
    • Measure performance decline across specific time periods using confusion-matrix counts (TP, FP, TN, FN).
    • Audit subgroup gaps to identify metric disparities while suppressing small-cell counts to protect privacy.
    • Identify observability gaps by flagging missing outcomes, stale windows, or uncovered features.
    • Generate draft alerts and retraining reviews based on configurable or default policy thresholds.

    How it works

    1. Prepare aggregates: You provide binned counts for features or outcome counts for performance per period.
    2. Execute script: The agent constructs a request.json and runs a deterministic Python script to ensure calculation accuracy.
    3. Review findings: The agent interprets the status envelope, highlighting warnings, exclusions, and alerts.
    4. Finalize report: You receive a structured Markdown report with traceable math and draft recommendations.

    Frameworks & tools

    The skill uses a dedicated Python engine for all calculations, ensuring decimals are never rounded or estimated by hand. It adheres to schema-based inputs for strict data integrity.

    Why this beats prompting it yourself

    Generic LLM prompts often hallucinate statistical values or fail to handle small-group suppression correctly. This skill uses a deterministic script and strict evidence discipline to ensure every number is traceable and policy-aligned.

    Use cases

    • Post-deployment checks to see if production data distributions still match the training set.
    • Periodic performance audits to detect model degradation before it impacts business outcomes.
    • Observability reviews to identify features or time windows that are currently unmonitored.
    • Retraining triggers to justify new model versions with empirical drift evidence.

    Known limitations

    Does not perform record-level inspections, data quality profiling, or A/B test significance. Requires pre-aggregated counts as input rather than raw data files.

    Aggregates only; no record-level data. No data quality profiling or significance testing. Retraining and deployment decisions require human approval.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 2 days ago

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Environment Variables
    Listed2 days ago

    What's inside

    Frequently Asked Questions

    This skill calculates specific statistical distance measures like Population Stability Index (PSI) and Jensen-Shannon divergence to identify when production data deviates from training baselines, preventing silent model failure.

    No, for privacy and efficiency, this skill requires pre-aggregated binned counts for features or outcome counts for performance rather than raw CSV or JSON record logs.

    The skill is designed to work with any AI agent capable of executing Python scripts and handling JSON inputs, making it compatible with most major LLM-ops frameworks.

    Unlike standard prompts that may hallucinate calculations, this skill uses a deterministic Python engine to ensure every statistical value is mathematically accurate and traceable.

    You receive the specialized monitoring logic, the deterministic Python calculation engine, and a structured schema for generating audit-ready Markdown reports.

    The skill includes built-in privacy logic that automatically suppresses small-cell counts during subgroup audits to prevent the accidental exposure of individual data points.