# Thales Research Notebook

Dated notes in this folder: one file per day, named `YYYY-MM-DD.md`. When multiple things happen in a day they all land in the same file, each entry headed with `## HH:MM — title`. When a new day starts, create a new file.

- **Current State** (below) — updated in place; always reflects what is actually running in production.
- **Daily notes** — `YYYY-MM-DD.md`; append-only within a day.
- **Index** — at the bottom of this file, newest day first.

Earlier history not captured here is in `git log` and `results/diagnostics/`.

---

## Current State — last updated 2026-04-06

### Universe & Data
- **Universe**: Russell 1000 (`universe.source: russell1000`), min market cap $2B, min avg volume 500k
- **Price data**: Tiingo primary, yfinance fallback; 904 symbols loaded, 4.17M rows, range 2005-01-01 → ~2026-02-27 (stale)
- **Factors**: Fama-French 5 + momentum, `data/factors/ff5_daily.parquet`
- **Macro**: VIX, yield curve (T10Y2Y), unemployment claims (ICSA) in `data/macro/`
- **Fundamentals**: SEC EDGAR PiT (free), plus FMP for extended metrics. Used in meta-labeler features.
- **Survivorship-free**: opt-in via `--survivorship-free` flag; Wikipedia-scraped S&P 500 membership history in `data/constituents/`

### Production Strategy
- **Class**: `MomentumStrategy` (`src/thales/strategy/momentum.py`)
- **Signal**: 12-2 momentum (`lookback: 252`, `skip: 5`) blended with smoothness (`smoothness_window: 21`, `smoothness_weight: 0.5`)
- **Top-N**: 50 stocks
- **Weighting**: HRP (`strategy.construction.weighting: hrp`) — crisis insurance, see 2026-04-06 HRP ablation
- **Sizing**: Half-Kelly (`risk.sizing: half_kelly`, `kelly.fraction: 0.5`, `rolling_window_months: 24`, `min_observations: 8`)

### Portfolio Risk Controls
- Max single position: **10%**
- Max sector: **35%** (raised from 25% for momentum concentration)
- Target vol: **12%** annualized (`vol_target: 0.12`, buffer 1%)
- Max leverage: **3.0x**
- Drawdown kill-switch: **20%** (`drawdown_kill_switch: 0.20`)
- Kill-switch re-entry: **dispersion-regime aware** (`kill_switch_reentry: dispersion`), min 21 cash days
- Dynamic vol target: **off** (`dynamic_vol_target: false`)

### Execution
- **Broker**: Alpaca paper (account `[REDACTED:acct]`, $10K)
- **Schedule**: GitHub Actions cron Mon-Fri 14:35 UTC via `scripts/run_daily.py` → `thales run`
- **Mode**: Daily signal diff (`rebalance.mode: daily`) with Tuesday alpha rebalance (`rebalance_day: tuesday`)
- **Turnover cap**: 5% daily (`daily_turnover_cap: 0.05`)
- **Hysteresis**: entry_band 0, exit_band 10 (hold until rank drops below N+10)
- **Emergency exit**: bypass turnover cap at rank 200+ or after 3 zero-volume days
- **Execution timing**: T+1 open fills (`execution_timing: next_open`) — models overnight gap + intraday returns
- **TCA**: 3-component implementation shortfall → `data/processed/tca_log.jsonl`
- **Costs (backtest)**: flat 10 bps round-trip (`costs.model: flat`); impact model wired but disabled

### ML Application — **nothing live in production**

**1. Meta-labeler** (`src/thales/portfolio/meta_labeler.py`) — *disabled*
- Config: `meta_labeling.enabled: false`
- Trained artifact: `results/model/meta_labeler.pkl` (last trained 2026-03-31, 3040 samples, in-sample Sharpe 0.87)
- **Features (10)**: `xs_fracdiff_z, xs_vol_z, rvol_5_63, xs_adf_z, cs_dispersion, spy_ma200_dist, vix, xs_ebit_yield_z, xs_sue_z, xs_accruals_z` — `has_fundamentals: true`
- **Pipeline**: triple-barrier labels (AFML Ch 3), uniqueness weights (Ch 4), sequential bootstrap, ADF structural break filter, MDA importance, most recently PCA orthogonalization (Ch 8.4.2)
- **Hyperparams**: RF 200 trees, max_depth 4, barrier_mult 2.5, vol_span 20, label_horizon 42, purge_gap 42, embargo 5
- **Sizing mode**: `cross_sectional` (options: binary, continuous, relative_cdf, cross_sectional)
- **CPCV history**: PBO **100%** with OHLCV only → **100%** with AFML v2 → **100%** with SEC fundamentals. The **PCA orthogonalization** (commit `fbf0f37`) is the current frontier — **never CPCV'd**, represents the most plausible path to a breakthrough.
- **Stale warning**: inline comment in `settings.yaml` still says "needs PiT fundamentals" but fundamentals are already wired.

**2. LambdaRank** (`research/paused/ranker.py`, `RankerStrategy`) — *built, paused, archived 2026-05-26*
- Learning-to-rank gradient boosting as the primary stock-selection signal (not a meta-label gate)
- **6-path CPCV**: PBO **33.3%** — first ML model below baseline (commit `090fb7a`)
- **15-path CPCV**: PBO **50%** — matches baseline, does not beat it (commit `ee3f00e`)
- **Archive note**: moved from `src/thales/strategy/ranker.py` to `research/paused/ranker.py` on 2026-05-26 as part of the organization audit. Removed from `cli.py` strategy registry. To revive: move back into `src/thales/strategy/`, restore CLI registry entry, add tests, run through SHIP gate.

### Supporting ML / Research Infrastructure
- **Triple-barrier labeler**: `src/thales/preprocessing/labeler.py` *(deleted in cleanup 4e4c481; re-implementations live inside `meta_labeler.py`)*
- **Per-event features + Corwin-Schultz spread**: `src/thales/preprocessing/features.py` *(deleted in cleanup 4e4c481)*
- **Model store**: `src/thales/portfolio/model_store.py` (save/load trained RF)
- **Feature engineering**: `src/thales/features/indicators.py` (momentum variants, fracdiff, RVOL, dispersion, volume confirmation, FF3 residual momentum)
- **Fundamentals downloader**: `src/thales/data/fundamentals.py` (SEC EDGAR, free, PiT-aligned)

### Performance Snapshot (latest CPCV, 2026-04-06, HRP on, full 2005-2026 history)

| Metric | Value |
|---|---|
| PBO | **50.0%** |
| Observed Sharpe | 0.611 |
| Deflated Sharpe | 0.000 |
| Mean OOS Sharpe | +0.61 (std 0.20, range 0.35 – 1.01) |
| Mean OOS CAGR | +5.1% (range 2.6% – 9.7%) |
| Max OOS drawdown | 44.2% (crisis-straddling paths) |
| Positive OOS paths | **15/15** |
| IS-OOS Sharpe correlation | +0.02 |
| Artifact | `results/diagnostics/cpcv_2026-04-06T17-46-52.json` |

### Built but Disabled / Paused
| Component | Status | Config flag | Reason |
|---|---|---|---|
| Meta-labeling (bet sizer) | disabled | `meta_labeling.enabled: false` | PBO 100% across all feature sets so far; PCA variant untested |
| LambdaRank primary strategy | not deployed | `strategy.name: momentum` | PBO 50% matches baseline, doesn't beat it |
| Graduated SPY trend overlay | disabled | `overlay.enabled: false` | Biggest source of overfitting (+25% PBO in ablation) |
| Dispersion regime filter | disabled | `dispersion.enabled: false` | Built, not yet validated in production config |
| Dynamic vol target (VIX ratio) | disabled | `risk.dynamic_vol_target: false` | CPCV PBO 75%, didn't help |
| RVOL volume filter | disabled | `strategy.momentum.volume_filter.enabled: false` | Needs more walk-forward validation |
| FF3 residual momentum | disabled | `strategy.momentum.use_residual: false` | Costs ~1.7% CAGR, mechanism works but not a win |
| Fractional differentiation as primary signal | disabled | `strategy.momentum.use_fracdiff: false` | Cross-sectional ranking destroys stationarity advantage |
| VWAP signals | disabled | `strategy.momentum.use_vwap: false` | Not yet A/B tested |
| Value sleeve | removed | `strategy.construction.value_weight: 0.5` (unused) | No PBO or Sharpe benefit |
| Square-root impact cost model | available | `costs.model: flat` | Built (Almgren-Chriss), but flat 10 bps is the default |
| Survivorship-free mode | opt-in | CLI flag `--survivorship-free` | Off by default for backtests |

### Open Questions / Next Experiments
1. **CPCV the PCA-orthogonalized meta-labeler** (commit `fbf0f37` is the untested frontier — could break PBO 100% ceiling)
2. **LambdaRank tuning** — feature set expansion, loss function tweaks to push PBO below 50%
3. **Refresh price data** (`thales fetch`) — currently stale at Feb 27
4. **Validate dispersion regime filter** in production config — currently off
5. **TWAP order slicing** when AUM reaches ~$250K+

### Infrastructure
- **Python**: 3.13 venv
- **Tests**: 404 (CLAUDE.md count; includes 39 production integration tests in `tests/test_execution/test_production_integration.py`)
- **Parallel CPCV**: `thales cpcv --workers N` — 6 workers on M5 gives ~4 min wall time for 15 paths (~7.5× speedup)

---

## Daily Log Index

- **[2026-04-06](2026-04-06.md)** — HRP ablation (walk-forward vs CPCV disagree, HRP stays on); parallel CPCV validation; research notebook initialized