Mitigating Backtest Overfitting: Multiple Testing & Walk-Forward Validation

If you torture the data long enough, it will confess to any trading strategy. In systematic quantitative research, backtest overfitting is the leading cause of strategies that exhibit stellar in-sample Sharpe ratios but collapse upon live capital deployment.

Here are the mathematical and empirical techniques we use to enforce statistical rigor across factor pipelines.

1. Controlling False Discovery Rates (FDR)

When testing hundreds of candidate alpha signals across cross-sectional equity universes, standard $p$-value thresholds ($p < 0.05$) are insufficient due to the multiple-testing problem.

Using the Benjamini-Hochberg procedure or Deflated Sharpe Ratio (DSR), we adjust significance thresholds based on the total number of trial variations evaluated:

$$\text{Target threshold: } P_{(k)} \le \frac{k}{m} \cdot \alpha$$

This ensures that the expected proportion of false positives remains strictly bounded.

2. Realistic Market Impact & Slippage Modeling

A paper strategy that rebalances frequently can look remarkably profitable until realistic execution frictions are introduced.

In our backtest simulators, we incorporate the Kyle-Obizhaeva square-root market impact law:

$$\text{Market Impact Cost } \sim \sigma \cdot \sqrt{\frac{V_{\text{order}}}{V_{\text{ADV}}}}$$

Where:

• $\sigma$ is the asset's annualized daily volatility.
• $V_{\text{order}}$ is the target trade volume.
• $V_{\text{ADV}}$ is the 20-day Average Daily Volume.

Adding turnover churn bands and discrete integer share constraints ensures backtest results translate into deployable institutional strategies.