out of sample edge reality checker

The problem

Backtested trading strategies often look profitable on paper but fail immediately in live markets because they were overfit to historical noise. Identifying whether a strategy has a genuine edge or just a statistical fluke requires rigorous validation against unseen data.

What it does

• Identifies the in-sample (fitting) and out-of-sample (validation) data periods to ensure true isolation.
• Calculates degradation metrics for Sharpe ratio, CAGR, and maximum drawdown across both periods.
• Performs statistical t-tests and distribution overlap checks to confirm performance stability.
• Flags overfitting signatures like outlier dependency and parameter sensitivity.
• Assigns a final verdict of Real Edge, Fragile Edge, or Overfit to guide capital allocation.

Why this beats prompting it yourself

General prompts often accept backtest results at face value or miss subtle signs of data leakage. This skill enforces strict degradation thresholds and statistical checks that act as a professional-grade filter, preventing you from deploying strategies that only work in the past.

Use cases

• Validating a quantitative strategy before moving from paper trading to live capital.
• Auditing a third-party strategy report to verify claimed performance persistence.
• Re-evaluating a modified strategy after adding new indicators or parameters.
• Assessing if a live strategy's recent performance dip is expected or a sign of edge decay.

Known limitations

Small sample sizes in the out-of-sample period can lead to an 'Insufficient Data' verdict. It cannot account for hidden data leakage if the developer already viewed the validation data during the design phase.