Notebook Configuration Audit Pattern

Experiment Overview

Context

Problem: Training notebooks often have configuration cells at the top (e.g., LOOKBACK_DAYS = 1460), but later cells may have hardcoded values that contradict these settings (e.g., lookback_days=365).

Symptoms:

• Notebook behavior doesn't match documented configuration
• Different cells use different values for the same setting
• Comments say one thing, code does another

Verified Audit Workflow

Step 1: Extract All Notebook Cells

import json
with open('notebooks/training.ipynb') as f:
    nb = json.load(f)

# Search for specific patterns
for i, cell in enumerate(nb['cells']):
    src = ''.join(cell.get('source', []))
    if 'lookback' in src.lower():
        print(f'Cell {i}: {src[:200]}...')

Step 2: Search for Key Configuration Patterns

Common patterns to audit:

patterns = [
    # Lookback/timeframe settings
    'lookback', 'LOOKBACK', 'days=', 'years',
    '365', '252', '1460', '1000',
    'timeframe', 'TIMEFRAME', '1Hour', '1Day', '1Min',

    # Slippage/cost settings
    'slippage', 'cost', '0.001', '0.002', '0.0025',

    # Version references
    'v2.', 'v3.',

    # Multi-asset settings
    'crypto', 'equity', 'enabled'
]

for i, cell in enumerate(nb['cells']):
    src = ''.join(cell.get('source', []))
    for pattern in patterns:
        if pattern.lower() in src.lower():
            # Check if it's a hardcoded value vs config reference
            lines = src.split('\n')
            for j, line in enumerate(lines):
                if pattern.lower() in line.lower() and '=' in line:
                    if not line.strip().startswith('#'):
                        print(f'Cell {i}, Line {j}: {line.strip()[:100]}')

Step 3: Cross-Reference Configuration Definitions

Find where config values are defined vs used:

# Find definitions (cell with CONFIG_NAME = value)
definitions = {}
for i, cell in enumerate(nb['cells']):
    src = ''.join(cell.get('source', []))
    if 'LOOKBACK_DAYS =' in src or 'TIMEFRAME =' in src:
        definitions[i] = src

# Find usages (cells using config or hardcoded values)
usages = {}
for i, cell in enumerate(nb['cells']):
    src = ''.join(cell.get('source', []))
    if 'lookback_days=' in src:  # lowercase = function parameter
        usages[i] = src

Step 4: Fix Inconsistencies

Replace hardcoded values with config references:

# Fix: lookback_days=365 -> lookback_days=LOOKBACK_DAYS
cell_src = nb['cells'][17]['source']
fixed = []
for line in cell_src:
    if 'lookback_days=365,' in line:
        line = line.replace('lookback_days=365,', 'lookback_days=LOOKBACK_DAYS,')
    fixed.append(line)
nb['cells'][17]['source'] = fixed

# Save
with open('notebooks/training.ipynb', 'w') as f:
    json.dump(nb, f, indent=1)

Common Issues Found

Failed Attempts

Verification Checklist

After fixing, verify:

# 1. Check config definitions
print(f'LOOKBACK_DAYS = {LOOKBACK_DAYS}')  # Should be 1460

# 2. Check all usages reference config
# Search for hardcoded values that should use config:
grep -n "lookback_days=[0-9]" notebooks/training.ipynb

# 3. Check comments match config
# Search for year references:
grep -n "years" notebooks/training.ipynb

# 4. Run notebook smoke test
python -c "exec(open('notebooks/training.ipynb').read())"

Automation Script

Save this as scripts/audit_notebook_config.py:

#!/usr/bin/env python
"""Audit notebook for configuration inconsistencies."""
import json
import sys

def audit_notebook(path: str) -> list:
    """Find potential configuration issues in notebook."""
    issues = []

    with open(path) as f:
        nb = json.load(f)

    # Known config patterns to check
    checks = [
        ('lookback_days=365', 'Should use LOOKBACK_DAYS'),
        ('lookback_days=252', 'Should use LOOKBACK_DAYS'),
        ('(2 years)', 'Should say (4 years) if LOOKBACK_DAYS=1460'),
        ('slippage_cost_crypto=0.002', 'Should be 0.0025 per Alpaca fees'),
    ]

    for i, cell in enumerate(nb['cells']):
        src = ''.join(cell.get('source', []))
        for pattern, message in checks:
            if pattern in src:
                issues.append(f'Cell {i}: {message} (found: {pattern})')

    return issues

if __name__ == '__main__':
    path = sys.argv[1] if len(sys.argv) > 1 else 'notebooks/training.ipynb'
    issues = audit_notebook(path)
    if issues:
        print('Configuration Issues Found:')
        for issue in issues:
            print(f'  - {issue}')
        sys.exit(1)
    else:
        print('No configuration issues found.')

Related Skills

• private-function-shim-export: Related shim/import issues
• codebase-consolidation-pattern: Module organization patterns

References

• Commit 160ebff: Fixes for training.ipynb hardcoded values
• Cell 12: Configuration definitions (LOOKBACK_DAYS, TIMEFRAME)
• Cell 17: Symbol selection (was using hardcoded 365)
• Cell 30: Data loading (comment said 2 years)
• Cell 33: Training loop (print statement had wrong slippage)