Works with the AI Agents you already use
analytics model risk and validation reviewer
This skill provides an accountable, deterministic review of model dossiers by replicating metrics and stress-testing datasets against objective risk criteria.
$2.99
analytics model risk and validation reviewer
Example session with this skill installed
Review the attached model card for 'ChurnPredict-V2' and the evaluation records. Replicate the 0.85 AUC claim and check for training leakage. Impact is medium.
- Read your context and instructions
- Compiled the analytics model risk
I have processed the validation request for ChurnPredict-V2. The internal validation script has recomputed all metrics and audited the dataset overlaps. # Model Validation Dossier — ChurnPredict-V2, 2023-10-27 ## 1. Scope, sources and assumptions - Sources: Model Card (E1), Test Records (E2), Training Manifest (E3). - Purpose: Independent replication of performance claims and leakage audit. - Assumptions: Evaluation records represent the intended deployment distribution (user_supplied). ## 2. Recommendation (draft for human approval) - Decision: `needs_review` - Basis: Claimed AUC replicated, but significant data leakage detected in Section 5. - Production release authorised: no - **Management decision required** ## 3. Findings | Id | Severity | Code | Finding | Recommendation | | :--- | :--- | :--- | :--- | :--- | | 1 | Critical | LEAK_01 | 15% overlap between training and test record IDs. | Re-run evaluation with fresh holdout set. | | 2 | Medium | DOC_04 | Missing monitoring plan in deployment section. | Define drift thresholds. | ## 4. Claims and benchmark replication | Claim | Kind | Metric | Claimed | Recomputed | Difference | Status | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | C-01 | Predictive | AUC | 0.8500 | 0.8502 | +0.0002 | Replicated | | C-02 | Predictive | Precision | 0.7800 | 0.7795 | -0.0005 | Replicated | ## 5. Leakage, robustness and fairness checks - **Overlap counts:** 1,500 records found in both training and evaluation sets. - **Temporal separation:** 0 days (evaluation records are within the training date range). - **Robustness:** 4% drop in AUC during high-noise scenario (within 5% tolerance). - **Fairness screen:** TPR Gap of 0.02 (Screening indicator). ## 6. Documentation and deployment safeguards - Completeness ratio: 85% - **Information gap — organizational validation required**: Rollback procedures not documented. ## 7. Access, exclusions and warnings - Exclusions: 0 records excluded (all datasets approved). - Warnings: `WARN_LEAK_DETECTED`, `WARN_MISSING_MONITORING`. ## 8. Items for human review - Observation 1: Recomputed AUC matches the claim (computed result). - Observation 2: Training leakage (1,500 records) invalidates current evaluation integrity (computed result). - Observation 3: Model owner asserts features are available at decision time (user assertion). This review supports, and does not replace, the accountable owner's judgment. Any approval or release is a **Management decision required**. **Next steps** - Address the `LEAK_01` finding by preparing a clean evaluation dataset without training overlap. - Provide the missing monitoring plan to complete the deployment documentation. - Present this dossier to the Model Risk Committee for final review.
Connects securely to your tools. The creator never sees your data.
About this skill
Machine learning models often reach production without independent verification of their performance claims, leading to unexpected failures or bias. This skill provides an accountable, deterministic review of model dossiers by replicating metrics and stress-testing datasets against objective risk criteria.
What it does
- Metric replication recomputes AUC, accuracy, precision, recall, F1, and Brier scores from raw evaluation records to verify claimed performance.
- Leakage detection identifies train-test overlaps, temporal separation issues, and features unavailable at the time of decision.
- Fairness screening calculates selection-rate ratios and TPR gaps across supplied group labels to flag potential bias.
- Robustness testing measures performance drops across specific scenarios against defined tolerances.
- Risk documentation generates a structured validation dossier with a draft go/no-go recommendation for human review.
How it works
- Gather material such as the model card, evaluation records (true labels and scores), and decision context.
- Build the request by mapping sources to a
request.jsonfile following the provided schema. - Execute validation using the internal
scripts/run.pyto process the data deterministically. - Analyze the envelope to interpret the findings, warnings, and replication status generated by the script.
- Generate the dossier following a strict eight-section format for human-in-the-loop approval.
Frameworks & tools
The skill utilizes a self-contained Python validation engine (scripts/run.py) and JSON schema validation. It operates on pseudonymous evaluation records and model metadata.
Why this beats prompting it yourself
LLMs frequently hallucinate mathematical results or fail to catch subtle data leakage when reading text descriptions. This skill uses a deterministic script to ensure every decimal is exact and every overlap is counted, preventing the agent from "taking the model's word" for its performance.
Use cases
- Pre-deployment audit to verify that a model meets internal performance and fairness benchmarks before release.
- Independent verification when a third-party or different team provides a model card with unverified claims.
- Risk mitigation to identify data leakage or robustness issues that could cause production outages.
Known limitations
Does not perform model training, tuning, or continuous drift monitoring. Requires raw evaluation records (labels and scores) to replicate metrics. Cannot authorize production releases or certify legal compliance.
Requires raw evaluation records for replication. Does not perform training or drift monitoring. Release authorization requires human-in-the-loop approval.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago