Trading Strategy Backtest & Overfitting Auditor

Trading Strategy Backtest & Overfitting Auditor is a premium quantitative strategy-validation skill designed to determine whether a trading backtest deserves to be trusted.

It can analyze:

Pine Script Strategies Pine Script Indicators Used as Strategy Logic TradingView Strategy Tester Results Natural-Language Trading Strategies Trade Lists CSV Exports Optimization Reports Parameter Sweeps Equity Curves Walk-Forward Tests Multi-Period Tests Multi-Market Tests Forward-Test Results Paper-Trading Results Screenshots Performance Summaries Broker-Simulation Reports

The skill does not begin by asking whether the strategy is profitable.

It begins by asking whether the test itself is valid.

Its operating sequence is:

Strategy Logic → Test Integrity → Data Integrity → Execution Realism → Sample Adequacy → In-Sample / Out-of-Sample Review → Selection-Bias Analysis → Parameter Sensitivity → Performance Distribution → Drawdown and Loss Behavior → Regime Robustness → Period Robustness → Stress Testing → Final Classification

The skill is designed to detect misleading backtest results caused by:

Look-Ahead Bias Repainting Higher-Timeframe Future Leakage Invalid Intrabar Assumptions Unrealistic Same-Bar Fills Synthetic Chart Prices Missing Commission Unrealistic Slippage Ignored Spread Unlimited Liquidity Assumptions Incorrect Position Sizing Unrealistic Leverage Pyramiding Exposure Incorrect Session Logic Insufficient Warmup Data Problems Optimization Bias Selection Bias Multiple Testing Repeated OOS Reuse Low Trade Count Correlated Trades Narrow Parameter Optima Profit Concentration Regime Dependence Period Dependence Cost Fragility Execution Fragility

The skill performs Pine Script integrity review.

It can inspect:

strategy() configuration Initial Capital Quantity Method Default Quantity Pyramiding Commission Slippage Margin Assumptions Signal Timing Entry Logic Exit Logic Stop Logic Target Logic Trailing Stops Partial Exits Reversals Bar Confirmation Higher-Timeframe Data Lower-Timeframe Data Historical vs Realtime Behavior Session Filters Timezone Date Filters Warmup State Initialization Order Processing Same-Bar Execution Alert Logic vs Strategy Logic

Repainting is classified as:

No Material Repaint Concern Detected Potential Repaint Risk Confirmed Repaint / Future Leakage Unknown

The skill gives special attention to higher-timeframe requests because incorrectly aligned higher-timeframe values can create historical signals that use information unavailable at the actual decision point.

Confirmed future leakage is treated as a critical test-integrity failure.

The skill audits synthetic chart usage.

Potential chart types include:

Heikin Ashi Renko Kagi Line Break Point & Figure

The skill distinguishes between:

Synthetic Chart Used for Signal Generation

and

Synthetic Chart Prices Used as Execution Prices

because synthetic prices may not correspond directly to executable market prices.

The skill audits intrabar ambiguity.

Examples include:

Entry and Stop Touched in the Same Bar Entry and Target Touched in the Same Bar Stop and Target Both Touched in the Same Bar Backtest Assuming the Favorable Price Sequence

When the intrabar path is unknown, the skill can recommend:

Lower-Timeframe Validation Conservative Fill Assumptions Bar Magnifier / Equivalent Validation Where Available Alternative Execution Tests

The skill audits execution realism.

It reviews:

Market Orders Limit Orders Stop Orders Stop-Limit Orders Same-Bar Orders Next-Bar Orders Bar-Close Orders Order Cancellation Order Replacement Partial Exits Trade Reversals

The skill evaluates whether historical fills are realistically obtainable.

Transaction costs receive dedicated analysis.

Commission can be reviewed as:

Per Order Per Contract Percentage Per Side Round Trip

The skill flags:

Missing Commission Wrong Commission Unit Unrealistically Small Commission One-Sided Costs Cost Assumptions Inconsistent with Trading Frequency

Slippage is evaluated relative to:

Instrument Timeframe Liquidity Volatility Order Type Session News Sensitivity Expected Size

Recommended stress scenarios can include:

Base Slippage 1.5× Slippage 2× Slippage 3× Slippage

Spread is evaluated when relevant, especially for:

Forex Crypto Scalping Systems Thinly Traded Markets Short-Timeframe Strategies

The skill never assumes unlimited liquidity.

When trade size and capacity cannot be evaluated, it explicitly states:

Capacity Not Evaluated

The skill supports market-specific validation.

For futures, it can consider:

Continuous Contract Construction Roll Logic Back Adjustment Tick Size Tick Value Contract Liquidity Per-Side Commission Slippage in Ticks Overnight Sessions Margin Assumptions

For crypto, it can consider:

24/7 Trading Exchange Differences Maker/Taker Fees Funding for Perpetual Contracts Liquidation Risk Exchange Outages Liquidity Changes Spread Variability

For forex, it can consider:

Spread Broker Feed Session Effects Rollover Bid/Ask Differences Weekend Gaps News Slippage

For equities, it can consider:

Splits Dividends Delistings Survivorship Bias Borrow Availability Short-Sale Constraints Opening Gaps Liquidity

The skill audits position sizing.

It can distinguish between:

Fixed Quantity Fixed Cash Percent of Equity Volatility-Based Sizing Compounding Leveraged Exposure

It can flag:

Unrealistic Size Growth Hidden Path Dependence Unconstrained Leverage Exposure That Expands During Favorable Historical Periods

Pyramiding can be evaluated for:

Maximum Concurrent Entries Average Exposure Worst-Case Exposure Stop Handling Margin Implications

The skill evaluates stop-loss logic.

It checks:

Stop Distance Stop Price Calculation Timing Gap Behavior Intrabar Behavior Volatility Dependence Tick-Size Constraints

Take-profit logic can be reviewed for:

Target Distance Limit Fill Assumptions Same-Bar Behavior Partial Exits Unrealistically Favorable Price Touches

Trailing stops can be reviewed for:

Activation Trail Distance Update Timing Intrabar Dependency Historical vs Realtime Differences

The skill performs data-integrity review.

It can check:

Correct Symbol Continuous vs Individual Futures Contracts Contract Rolls Adjusted vs Unadjusted Data Corporate Actions Session Templates Timezone Missing Bars Illiquid Periods Historical Depth Symbol Changes Data-Vendor Differences Survivorship Bias

The skill reviews natural-language strategies before backtesting.

Ambiguous language such as:

Strong Trend Good Momentum Near Support Clear Reversal High Volume

must be converted into measurable definitions.

The skill can produce an exact test specification containing:

Entry Exit Stop Target Position Size Session Pyramiding Reversal Cooldown Order Timing Invalidation Data Source

This prevents discretionary hindsight from being disguised as systematic logic.

The skill gives major importance to in-sample versus out-of-sample testing.

In-Sample data includes any history used to:

Create Rules Choose Indicators Choose Filters Tune Parameters Choose Stops Choose Targets Choose Markets Choose Timeframes

Out-of-Sample data should not be used during development.

The skill identifies OOS contamination.

For example:

Test OOS → Change Filter → Retest Same OOS → Change Stop → Retest Same OOS

This means the supposed OOS period has become part of strategy development.

The skill can recommend:

Development Data Validation Data Final Untouched Holdout

It can compare IS and OOS using:

Trade Count Net Profit Profit Factor Expectancy Sharpe Sortino Maximum Drawdown Win Rate Payoff Ratio Exposure Loss Streaks

The skill does not require identical IS and OOS results.

Some degradation is expected.

It instead evaluates whether degradation is plausible or catastrophic.

The skill can audit walk-forward testing.

Possible workflow:

Train / Calibrate → Freeze → Test Next Unseen Segment → Roll Forward → Repeat

It reviews:

Training-Window Length Testing-Window Length Overlap Leakage Reoptimization Frequency Parameter Drift Out-of-Sample Aggregation Regime Coverage

It supports:

Anchored Walk-Forward Rolling Walk-Forward

without assuming one is universally superior.

The skill explicitly analyzes selection bias.

Potential causes include:

Many Indicators Tested Many Assets Tested Many Timeframes Tested Many Stop/Target Combinations Many Filters Many Entry Rules Thousands of Parameter Combinations Only the Winning Strategy Reported

The skill asks:

How many materially different variants were tested before this one?

When unknown, it states:

Selection-Bias Exposure Cannot Be Quantified

The skill evaluates multiple-testing risk.

It considers:

Search-Space Size Number of Configurations Optimization Objective Manual Experimentation Stopping Rule Number of Markets Number of Timeframes

The skill evaluates research degrees of freedom.

Potential dimensions include:

Indicator Type Indicator Length Threshold Session Timeframe Stop Target Filter Market Entry Timing Exit Timing Confirmation Logic

More flexible systems require stronger independent validation.

The skill performs parameter-sensitivity analysis.

Each parameter can be reviewed using:

Selected Value Nearby Values Performance Neighborhood Parameter Surface Shape

Possible parameter-surface classifications:

Broad Plateau Gradual Region Cliff Isolated Spike

A broad stable region is generally stronger evidence than one exceptional parameter surrounded by weak results.

The skill can recommend perturbations such as:

Nearby Integers ±5% ±10% Nearby Thresholds Nearby Stop Distances Nearby Target Distances

For interacting parameters, it can evaluate joint sensitivity.

A stable ridge or plateau is generally more credible than a single isolated optimum.

The skill assesses trade-count adequacy.

It does not claim that any universal trade count guarantees robustness.

A practical evidence heuristic can classify:

Under 30 Trades: Very Weak Evidence

30–99 Trades: Weak Evidence

100–299 Trades: Moderate Evidence

300+ Trades: Stronger Statistical Base

These are context-dependent guidelines, not guarantees.

The skill also evaluates whether trades are actually independent.

Trade dependence can arise from:

Pyramiding Multiple Entries During One Trend Overlapping Positions One Event Generating Many Trades Highly Correlated Instruments Clustered Signals

The nominal trade count may therefore exaggerate effective evidence.

The skill evaluates Win Rate.

Formula:

Winning Trades / Total Trades

Win Rate is never interpreted in isolation.

The skill evaluates Payoff Ratio.

Formula:

Average Winner / Absolute Average Loser

Win rate and payoff should be analyzed together.

The skill evaluates expectancy.

Formula:

(Win Rate × Average Winner)

(Loss Rate × Absolute Average Loser)

Expectancy can be expressed in:

Currency Points Ticks R-Multiples Percentage

depending on the available data.

The skill evaluates Profit Factor.

Formula:

Gross Profit / Absolute Gross Loss

Profit Factor is evaluated for:

Magnitude Period Stability OOS Stability Cost Sensitivity Dependence on Large Winners

A high Profit Factor does not automatically imply robustness.

The skill evaluates profit concentration.

Possible checks include:

Largest Winner Contribution Top 3 Winners Contribution Top 5 Winners Contribution Top 10 Winners Contribution

The skill can recommend winner-removal stress tests.

Example:

Remove Largest Winner Remove Top 3 Winners Remove Top 5 Winners

If the entire edge disappears after removing a small number of trades, confidence is reduced.

The skill evaluates loss tails.

It can inspect:

Largest Loss Top Losses Gap Losses Stop Overruns Loss Clusters Tail Asymmetry

The skill evaluates Sharpe Ratio.

It recognizes limitations including:

Sparse Trades Return-Frequency Sensitivity Serial Correlation Skewed Returns Smoothing Platform-Specific Calculation Methods

The skill evaluates Sortino Ratio.

It recognizes that downside-focused ratios still depend heavily on:

Sample Size Return Construction Outliers Dependence Calculation Method

When platform methodology is unknown, the skill labels it explicitly as unverified.

The skill performs comprehensive drawdown analysis.

Potential dimensions include:

Absolute Drawdown Percentage Drawdown Peak-to-Trough Drawdown Closed-Equity Drawdown Open-Equity Drawdown Drawdown Duration Recovery Time Longest Underwater Period

A strategy with moderate drawdown but extremely long recovery may still be operationally difficult.

The skill evaluates drawdown concentration across:

Trend Regimes Range Regimes High Volatility Low Volatility Gaps News Structural Transitions

The skill evaluates maximum consecutive losses.

It can review:

Maximum Losing Streak Frequency of Long Streaks Loss Clustering Regime Association

The skill does not convert these results into personalized financial recommendations.

The equity curve is reviewed for:

Smoothness Large Step Changes One-Time Jumps Long Stagnation Structural Breaks Sudden Collapse Suspicious Monotonicity

A smooth equity curve is not automatically considered strong evidence.

The skill can review trade-return distributions.

Potential statistics include:

Mean Median Standard Deviation Skew Quantiles Outliers Tail Losses

A positive mean combined with a negative median can reveal dependence on rare large winners.

The skill evaluates market exposure.

Potential dimensions include:

Percent Time in Market Long Exposure Short Exposure Simultaneous Positions Leverage

Where appropriate, performance can be compared with:

Buy-and-Hold Cash Simple Trend Rule Simpler Strategy Variant

The skill can challenge excessive strategy complexity.

If a complex strategy produces little OOS improvement over a simple rule, simplification may improve robustness.

The skill performs regime analysis.

Potential regimes include:

Bull Trend Bear Trend Sideways High Volatility Low Volatility Crisis Recovery Liquid Periods Illiquid Periods

Regime classification should be defined independently where possible.

The skill does not create favorable regime definitions retrospectively merely to rescue poor performance.

A regime matrix can contain:

Regime Trades Expectancy Profit Factor Drawdown Status

Possible statuses:

Stable Acceptable Weak Failing Insufficient Sample

The skill performs period robustness analysis.

History may be split by:

Year Quarter Rolling Window Equal-Length Block Market Cycle

A strategy that earns almost all of its profit during one short historical segment receives additional scrutiny.

The skill supports rolling performance review.

Possible windows include:

3 Months 6 Months 12 Months

depending on strategy frequency and data length.

The skill evaluates:

Stability Deterioration Recovery Structural Breaks

The skill supports cross-market robustness tests where economic transferability is logical.

Potential examples include:

Multiple Equity Indices Related Futures Markets Several Liquid FX Pairs Several Large-Cap Crypto Markets

Cross-market performance is supporting evidence, not a universal requirement.

The skill supports cross-timeframe testing when the underlying logic should reasonably scale.

It recognizes that market microstructure can change materially between timeframes.

The skill performs cost-stress testing.

Potential scenarios:

Base Costs Moderately Higher Costs Severe Costs

The question is whether the strategy remains economically coherent when execution becomes less favorable.

The skill performs delay stress testing where appropriate.

Examples:

One-Bar-Later Entry Less Favorable Entry Less Favorable Exit

A strategy that collapses after tiny timing degradation may be fragile.

Price perturbation can be performed using:

Ticks Spread Fraction of ATR

depending on the market.

The skill performs rule perturbation.

Examples:

Slightly Different Threshold Small Parameter Change Slight Timing Change Removal of One Filter

If every exact historical rule is required for profitability, overfitting risk increases.

The skill supports filter ablation.

Each filter can be removed individually to determine whether it adds stable value or merely captures historical exceptions.

Feature ablation can compare:

Base Strategy Base + Filter A Base + Filter B Base + Filter C

The skill supports Monte Carlo analysis when trade-level data is available.

Possible tests include:

Trade-Order Reshuffling Bootstrap Resampling Slippage Perturbation Missed Trades Execution Noise Parameter Perturbation

Possible outputs include:

Median Terminal Result Drawdown Distribution High-Percentile Drawdown Probability of Negative Result Under the Simulation Assumptions Loss-Streak Distribution

Monte Carlo results are treated as conditional simulations, not certainty.

The skill can recommend bootstrap analysis.

Where trade dependence exists, it recognizes that independent resampling may be inappropriate and that block-based methods may be required.

The skill identifies data snooping.

Repeated exposure to the same historical data during strategy development means the strategy has learned from that period, even when tuning was performed manually rather than through automated optimization.

The skill identifies survivorship bias.

For multi-stock studies, it can ask whether:

Delisted Companies Failed Companies Historical Constituents

were included.

The skill identifies execution leakage.

A strategy may not explicitly use future data but may still create impossible historical results through unrealistic fill sequencing.

The skill audits trade lists.

Potential fields include:

Entry Timestamp Exit Timestamp Duration Direction Position Size P&L MAE MFE Entry Reason Exit Reason Order Type

MAE and MFE can help evaluate:

Stop Efficiency Target Efficiency Exit Quality Tail Exposure

The skill compares backtest and live or paper results when available.

Potential comparisons include:

Signal Count Fill Price Slippage Missed Trades Win Rate Expectancy Average Trade Drawdown Gap Behavior Latency

The skill supports forward-test discipline.

A recommended freeze protocol is:

Freeze Strategy Logic Freeze Parameters Record Version Record Assumptions Test on Unseen Data Do Not Tune During the Test Record Deviations Separately

The skill can create strategy versioning records containing:

Version Date Logic Changes Parameter Changes Reason Historical Data Seen Before the Change

This provides an audit trail for retrospective tuning.

The skill uses a strict robustness-evidence hierarchy.

Stronger evidence generally includes:

Valid Test Realistic Costs Adequate Sample True Unseen Data Stable Nearby Parameters Multiple Periods Multiple Regimes Stress-Test Survival Credible Forward Behavior

Weaker evidence typically includes:

One Optimized Backtest One Market One Period Unknown Selection History Favorable Execution Narrow Parameter Optimum Small Sample

The final result is classified into exactly one category.

ROBUST

Used only when:

No Critical Test-Integrity Failure Is Detected Execution Assumptions Are Plausible Meaningful OOS Evidence Exists OOS Behavior Remains Economically Coherent Sample Size Is Reasonable Parameters Show Stability Performance Is Not Critically Concentrated Drawdown Is Understood Multiple Periods Are Acceptable Regime Behavior Is Understood Stress Tests Do Not Immediately Destroy the Edge No Major Unresolved Bias Exists

Required interpretation:

ROBUST — The available evidence shows comparatively strong historical robustness under the tested assumptions. This does not guarantee future profitability.

PROMISING BUT UNPROVEN

Used when:

The Test Appears Valid A Potential Edge Exists Evidence Remains Incomplete

Possible causes:

Limited OOS Moderate Trade Count No Walk-Forward No Cost Stress Limited Regime Coverage No Frozen Forward Test Incomplete Parameter Analysis

OVERFIT RISK

Used when the test is technically interpretable but historical fragility is material.

Possible causes:

Narrow Parameter Optimum Major IS/OOS Collapse Large Search Space Profit Concentration One Period Producing Most Profit Unexplained Regime Dependence Cost Fragility Complex Rule Stack Repeated OOS Reuse Instability Under Small Changes

INVALID TEST

Used when a fundamental integrity issue prevents credible interpretation.

Examples:

Look-Ahead Bias Confirmed Repainting Future Leakage Impossible Execution Materially Invalid Same-Bar Fill Assumptions Synthetic Prices Used Incorrectly Corrupted or Misaligned Data Broken Logic Reported Results Not Matching the Tested Strategy

Classification precedence is:

INVALID TEST → OVERFIT RISK → PROMISING BUT UNPROVEN → ROBUST

A critical validity failure takes precedence over every profitability metric.

Every final audit can also include:

Classification Confidence: High Medium Low

The skill can create an optional 0–5 scorecard covering:

Test Integrity Execution Realism Sample Adequacy OOS Quality Selection-Bias Control Parameter Stability Drawdown Quality Regime Robustness Period Robustness Statistical Robustness Reproducibility

This score is never presented as a probability of future profitability.

The skill can produce:

Executive Verdict Final Classification Classification Confidence Strategy Summary Test Validity Review Pine Script Integrity Review Data Integrity Review Execution Realism Review Sample Adequacy IS vs OOS Analysis Selection-Bias Analysis Parameter-Sensitivity Analysis Core Metric Review Trade-Distribution Review Drawdown Analysis Consecutive-Loss Analysis Regime Analysis Period Robustness Cross-Market Robustness Stress-Test Results Red-Flag Register Unknowns Register Required Retests Final Conclusion

The skill is built for traders who want a rigorous second opinion on backtest credibility rather than another system that simply celebrates a high Profit Factor or attractive equity curve.