QuantConnect Validation Skill (Phase 5)

Purpose: Walk-forward validation for Phase 5 robustness testing before deployment.

Progressive Disclosure: This primer contains essentials only. Full details available via qc_validate.py docs command.

---

When to Use This Skill

Load when:

• Running /qc-validate command
• Testing out-of-sample performance
• Evaluating strategy robustness
• Making deployment decisions

Tool: Use python SCRIPTS/qc_validate.py for walk-forward validation

---

Walk-Forward Validation Overview

Purpose: Detect overfitting and ensure strategy generalizes to new data.

Approach:

1. Training (in-sample): Develop/optimize on 80% of data
2. Testing (out-of-sample): Validate on remaining 20%
3. Compare: Measure performance degradation

Example (5-year backtest 2019-2023):

• In-sample: 2019-2022 (4 years) - Training period
• Out-of-sample: 2023 (1 year) - Testing period

---

Key Metrics

1. Performance Degradation

Formula: (IS Sharpe - OOS Sharpe) / IS Sharpe

Degradation	Quality	Decision
< 15%	Excellent	Deploy with confidence
15-30%	Acceptable	Deploy but monitor
30-40%	Concerning	Escalate to human
> 40%	Severe	Abandon (overfit)

Key Insight: < 15% degradation indicates robust strategy that generalizes well.

---

2. Robustness Score

Formula: OOS Sharpe / IS Sharpe

Score	Quality	Interpretation
> 0.75	High	Strategy robust across periods
0.60-0.75	Moderate	Acceptable but monitor
< 0.60	Low	Strategy unstable

Key Insight: > 0.75 indicates strategy maintains performance out-of-sample.

---

Quick Usage

Run Walk-Forward Validation

# From hypothesis directory with iteration_state.json
python SCRIPTS/qc_validate.py run --strategy strategy.py

# Custom split ratio (default 80/20)
python SCRIPTS/qc_validate.py run --strategy strategy.py --split 0.70

What it does:

1. Reads project_id from iteration_state.json
2. Splits date range (80/20 default)
3. Runs in-sample backtest
4. Runs out-of-sample backtest
5. Calculates degradation and robustness
6. Saves results to PROJECT_LOGS/validation_result.json

---

Analyze Results

python SCRIPTS/qc_validate.py analyze --results PROJECT_LOGS/validation_result.json

Output:

• Performance comparison table
• Degradation percentage
• Robustness assessment
• Deployment recommendation

---

Decision Integration

After validation, the decision framework evaluates:

DEPLOY_STRATEGY (Deploy with confidence):

• Degradation < 15% AND
• Robustness > 0.75 AND
• OOS Sharpe > 0.7

PROCEED_WITH_CAUTION (Deploy but monitor):

• Degradation < 30% AND
• Robustness > 0.60 AND
• OOS Sharpe > 0.5

ABANDON_HYPOTHESIS (Too unstable):

• Degradation > 40% OR
• Robustness < 0.5 OR
• OOS Sharpe < 0

ESCALATE_TO_HUMAN (Borderline):

• Results don't clearly fit above criteria

---

Best Practices

1. Time Splits

• Standard: 80/20 (4 years training, 1 year testing)
• Conservative: 70/30 (more OOS testing)
• Very Conservative: 60/40 (extensive testing)

Minimum OOS period: 6 months (1 year preferred)

---

2. Never Peek at Out-of-Sample

CRITICAL RULE: Never adjust strategy based on OOS results.

• OOS is for testing only
• Adjusting based on OOS defeats validation purpose
• If you adjust, OOS becomes in-sample

---

3. Check Trade Count

Both periods need sufficient trades:

• In-sample: Minimum 30 trades (50+ preferred)
• Out-of-sample: Minimum 10 trades (20+ preferred)

Too few trades = unreliable validation.

---

4. Compare Multiple Metrics

Don't just look at Sharpe:

• Sharpe Ratio degradation
• Max Drawdown increase
• Win Rate change
• Profit Factor degradation
• Trade Count consistency

All metrics should degrade similarly for robust strategy.

---

Common Issues

Severe Degradation (> 40%)

Cause: Strategy overfit to in-sample period

Example:

• IS Sharpe: 1.5 → OOS Sharpe: 0.6
• Degradation: 60%

Decision: ABANDON_HYPOTHESIS

Fix for next hypothesis: Simplify (fewer parameters), longer training period

---

Different Market Regimes

Cause: IS was bull market, OOS was bear market

Example:

• 2019-2022 (bull): Sharpe 1.2
• 2023 (bear): Sharpe -0.3

Decision: Not necessarily overfit, but not robust across regimes

Fix: Test across multiple regimes, add regime detection

---

Low Trade Count in OOS

Cause: Strategy stops trading in OOS period

Example:

• IS: 120 trades → OOS: 3 trades

Decision: ESCALATE_TO_HUMAN (insufficient OOS data)

---

Integration with /qc-validate

The /qc-validate command workflow:

1. Read iteration_state.json for project_id and parameters
2. Load this skill for validation approach
3. Modify strategy for time splits (80/20)
4. Run in-sample and OOS backtests
5. Calculate degradation and robustness
6. Evaluate using decision framework
7. Update iteration_state.json with results
8. Git commit with validation summary

---

Reference Documentation

Need implementation details? All reference documentation accessible via --help:

python SCRIPTS/qc_validate.py --help

That's the only way to access complete reference documentation.

Topics covered in --help:

• Walk-forward validation methodology
• Performance degradation thresholds
• Monte Carlo validation techniques
• PSR/DSR statistical metrics
• Common errors and fixes
• Phase 5 decision criteria

The primer above covers 90% of use cases. Use --help for edge cases and detailed analysis.

---

Related Skills

• quantconnect - Core strategy development
• quantconnect-backtest - Phase 3 backtesting (qc_backtest.py:**)
• quantconnect-optimization - Phase 4 optimization (qc_optimize.py:**)
• decision-framework - Decision thresholds
• backtesting-analysis - Metric interpretation

---

Key Principles

1. OOS is sacred - Never adjust strategy based on OOS results
2. Degradation < 15% is excellent - Strategy generalizes well
3. Robustness > 0.75 is target - Maintains performance OOS
4. Trade count matters - Need sufficient trades in both periods
5. Multiple metrics - All should degrade similarly for robustness

---

Example Decision

In-Sample (2019-2022):
  Sharpe: 0.97, Drawdown: 18%, Trades: 142

Out-of-Sample (2023):
  Sharpe: 0.89, Drawdown: 22%, Trades: 38

Degradation: 8.2% (< 15%)
Robustness: 0.92 (> 0.75)

→ DEPLOY_STRATEGY (minimal degradation, high robustness)

---

Version: 2.0.0 (Progressive Disclosure) Last Updated: November 13, 2025 Lines: ~190 (was 463) Context Reduction: 59%