- Home
- Skills
- Trading & Investing
- overfitting strategy bias auditor
overfitting strategy bias auditor
Audit data strategies and ML models for overfitting, data leakage, and biased evaluation logic.
$5
Works with the AI Agents you already use
overfitting strategy bias auditor
Example session with this skill installed
I ran a backtest for my crypto strategy using a random 80/20 split on 2 years of price data. I normalized the features using the full dataset mean to ensure consistency. My accuracy is 85%. Audit this.
- Read your context and instructions
- Compiled the overfitting strategy bias
Strategy Review Summary
Overall verdict: Major Changes Required
Key Findings
Area: Biased Evaluation
Severity: Critical
Issue: Random split on time-series data and global normalization leakage.
Evidence: Future data leaked into the training set.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem
Data science and trading strategies often fail in production because backtests were inadvertently tuned to noise. Developers struggle to spot subtle data leakage, look-ahead bias, and circular logic that make results appear artificially perfect during evaluation.
What it does
- Identifies data leakage in time-series splits and feature engineering pipelines.
- Detects multiple comparison bias when too many strategy variants are tested on the same dataset.
- Flags weak assumptions regarding stationarity, label quality, and environmental stability.
- Exposes misleading metrics, such as reporting relative lift without absolute base rates.
- Generates a structured audit report with severity ratings and required remediation steps.
Why this beats prompting it yourself
Generic prompts often miss the "unknown unknowns" of statistical bias. This skill applies a systematic adversarial mindset that explicitly hunts for specific red flags like random splits on temporal data or global normalization before splitting. It forces a rigorous reconstruction of the causal mechanism that a standard LLM conversation would likely skip.
Use cases
- Auditing a trading bot backtest before deploying capital.
- Reviewing a machine learning model's evaluation plan for data leakage.
- Validating A/B test results to ensure significance isn't gamed by cherry-picking time windows.
- Sanity checking a new analytics strategy before presenting to stakeholders.
Known limitations
Requires the user to provide detailed methodology and data handling procedures to be effective. It cannot execute code to verify claims; it audits the logic and plan described.
Requires detailed descriptions of data splits, feature engineering, and testing methodology. Cannot directly execute code to verify data distributions.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 months ago
Permissions
No special permissions declared or detected