# Agents & Commands

## Agents
| Agent | Description |
|-------|-------------|
| `trading-ml` | Primary — coordinates all ML trading work |
| `db-explorer` | Database exploration and queries |
| `data-engineer` | Data loading, cleaning, preprocessing |
| `feature-designer` | Technical indicators and feature engineering |
| `model-trainer` | ML model training and evaluation |
| `backtester` | Strategy backtesting and performance analysis |
| `expert` | LSTM expert system — 16 parallel expert networks |
| `trade-planner` | Trade parameter calculation (entry, SL, TP, ladder) |

## Project Structure
```
AI_Strategy/
├── opencode.json              # opencode config
├── config.py                  # DB config, constants
├── main.py                    # CLI entry point
├── monitor.py                 # Hourly signal monitoring (scheduled)
├── train_experts_all.py       # Full training: 14 tickers × 3 TF (v3 blending)
├── run_walkforward.py         # Walk-forward validation (7 folds)
├── walkforward_cv.py          # Walk-forward CV for MoE v12.5
├── db/
│   ├── connection.py          # MySQL connection (nlbotinterface.ru:3306/bitcoin_tickers)
│   └── queries.py             # SQL queries
├── data/
│   └── loader.py              # Candle data loading via pandas
├── features/
│   ├── technical.py           # Technical indicators + FEATURE_COLS
│   ├── directional.py         # Directional signals
│   ├── context.py             # Candlestick patterns, context features
│   └── targets.py             # ATR-based SL/TP outcomes
├── models/
│   ├── train.py               # RF/XGBoost training (legacy)
│   ├── predict.py             # Inference (legacy)
│   ├── hybrid.py              # LSTM-encoder → RF (used by experts.py)
│   ├── lstm_model.py          # Standalone LSTM (100-period)
│   ├── experts.py             # 16 LSTM experts + ExpertEnsemble
│   ├── moe.py                 # MultiTimeframeMoE — production model
│   └── saved/                 # Model persistence directory
├── rl/
│   ├── environment.py         # TradingEnv — gym-like market simulator
│   ├── dqn.py                 # DQN agent
│   └── train.py              # Training loop
├── trade/
│   └── manager.py             # TradeManager — live execution (SL/TP, re-entry, weekend)
└── utils/
    ├── moex_tickers.py        # MOEX instrument list
    └── notifier.py            # Telegram/webhook notifications
```

## Code Organization Rules (agents MUST follow)

### 1. NO new standalone scripts
Never create new `.py` files at the project root for training, prediction, analysis, or testing. All new functionality must be added as:
- A **subcommand** in `main.py` CLI (for training/inference)
- A **module** in `models/`, `features/`, `data/`, `rl/`, or `utils/` (for reusable logic)
- A new **class/method** in an existing module (for incremental improvements)
- A **monitoring feature** in `monitor.py` (for signal monitoring)

### 2. Canonical scripts — extend, don't duplicate
These are the **only** allowed entry point scripts at the project root:

| Script | Purpose | How to extend |
|--------|---------|---------------|
| `main.py` | CLI entry point for training, inference, analysis | Add new argparse subcommands |
| `monitor.py` | Hourly signal monitoring for all tickers | Add new features inside the script |
| `train_experts_all.py` | Full training of ExpertEnsemble (14 tickers × 3 TF) | Modify for new experts/params |
| `run_walkforward.py` | Walk-forward validation | Modify for new evaluation modes |
| `walkforward_cv.py` | Cross-validation walk-forward for MoE v12.5 | Modify for new CV strategies |

Any new training/monitoring/analysis logic goes into `main.py` as a subcommand.

### 3. One-off experiments → use `/tmp/opencode/`
For temporary experiments, debugging, or data exploration, create files in `/tmp/opencode/`. These files are **not** committed and are automatically cleaned up. Never create permanent experiment scripts in the project root.

### 4. Predict/inference scripts
Do NOT create standalone `predict_*.py` scripts. Use instead:
- `main.py --ticker SBER --moe --load <model.joblib>` — **primary**: MoE fast inference
- `monitor.py` — for scheduled prediction on all tickers
- `main.py --experts --load` — for ad-hoc single-ticker inference (legacy ExpertEnsemble)

### 5. Test scripts
Do NOT create standalone `test_*.py` scripts at the project root. If tests are needed, add them to a `tests/` directory following pytest conventions.

### 6. Configuration changes
All thresholds, hyperparameters, and per-ticker settings go into `config.py`. Never hardcode them in new scripts. Technical indicator feature lists are in `features/technical.py`.

### 7. Model persistence
All models are saved to `models/saved/` with a consistent naming convention:
- `{ticker}_moe_v12_rr1x2.joblib` — **MoE** models (primary production format, RR 1:2)
- `{ticker}_{tf}_experts.joblib` — ExpertEnsemble models (legacy)
- `{ticker}_{tf}_directional.joblib` — Directional models (legacy)

**MoE naming**: lowercase ticker, no timeframe (multi-TF), suffix `_moe_v12_rr1x2`.

### 8. Code review before adding files
Before creating any new file, ask:
- Can this be a **subcommand** in `main.py`? → put it there
- Can this be a **method** in an existing module? → add it there
- Is this a **temporary experiment**? → put it in `/tmp/opencode/`
- Is this genuinely new infrastructure? → create a module in the appropriate subdirectory (`models/`, `features/`, etc.)

Violations: if an agent creates a standalone script at project root for a task that could be a subcommand, it will be reverted.

## Custom Commands
| Command | Usage | Description |
|---------|-------|-------------|
| `/train` | `/train SBER H1` | Train model |
| `/backtest` | `/backtest SBER H1 model.joblib` | Backtest strategy |
| `/features` | `/features list` | Manage features |
| `/data` | `/data load SBER H1` | Explore database |
| `/experts` | `/experts train SBER` | Train expert ensemble |
| `/trade` | `/trade SBER SELL 100000` | Plan trade (entry, SL, TP, ladder) |

## Project Commands (CLI)

### Production (MoE — primary)
| Command | Description |
|---------|-------------|
| `python main.py --ticker SBER --moe` | Train MoE (16 experts + XGBoost) |
| `python main.py --ticker SBER --moe --load model.joblib` | Fast inference — load MoE, predict on latest data |

### Legacy modes (may still work but not actively developed)
| Command | Description |
|---------|-------------|
| `python main.py --ticker SBER --timeframe H1` | Train binary classifier |
| `python main.py --ticker SBER --cv` | Cross-validation |
| `python main.py --ticker SBER --3class` | Train 3-class outcome classifier |
| `python main.py --ticker SBER --rl` | Train DQN reinforcement learning agent |
| `python main.py --ticker SBER --directional` | Directional probability model (long/short) |
| `python main.py --ticker SBER --experts` | Train parallel LSTM expert ensemble + RF |
| `python main.py --ticker SBER --experts --cascade` | Train cascaded experts (deprecated) |
| `python main.py --ticker SBER --importance` | Feature importance analysis |
| `python main.py --list` | List MOEX instruments |

### Other modes
| Command | Description |
|---------|-------------|
| `python main.py --ticker SBER --lstm` | Standalone LSTM (50-period) |
| `python main.py --ticker SBER --xgb` | Use XGBoost for binary classifier |
| `python main.py --ticker SBER --optimize` | Optuna hyperparameter optimization |
| `python main.py --ticker SBER --benchmark` | Walk-Forward CV benchmark |
| `python run_walkforward.py` | Walk-forward validation (7 folds) |
| `python walkforward_cv.py --all` | MoE walk-forward CV for all tickers |

---

## MoE Architecture (Production — v12)

### Overview
```
Окно 20 свечей (66 base features, OHLCV + технические индикаторы + MTF контекст)
                                │
               ┌────────────────┼────────────────┐
               ▼                ▼                ▼
          16 LSTM экспертов (9 regression + 7 binary)
               │                │                │
               └────────────────┼────────────────┘
                                │
        Каждый эксперт → 4 горизонта [3,5,8,13] × 2 (signal+confidence) = 8 признаков
        16 экспертов × 8 = 128 expert-признаков + 70 base = 198 признаков
                                │
                                ▼
              ┌─ XGBoost (Long Classifier) ──→ P(success|long)
              └─ XGBoost (Short Classifier) ──→ P(success|short)
                                │
                                ▼
                    Threshold [0.55 – 0.75]
                    WinRate > 35%, Min Trades ≥ 15
                                │
                                ▼
                    BUY / SELL / NEUTRAL
```

### Key Parameters
- **Window**: 20 свечей (LSTM sequence)
- **Horizons**: `[3, 5, 8, 13]` (4 targets per expert)
- **Base features**: ~70 (OHLCV + технические индикаторы + MTF контекст + временные признаки)
- **Expert signals per expert**: 8 (4 горизонтов × 2: signal + confidence)
- **Total XGBoost features**: 198 (70 base + 128 expert)
- **Horizon weights**: `[0.4, 0.3, 0.2, 0.1]` (короткие горизонты важнее)

### XGBoost Regularization
```python
max_depth=4, learning_rate=0.03, n_estimators=300,
subsample=0.7, colsample_bytree=0.7,
reg_alpha=0.1, reg_lambda=1.0, min_child_weight=5
```

### Threshold Search (`find_best_thr` in `models/moe.py`)
- Range: **0.55 – 0.75** (шаг 0.02)
- Filters: **WinRate > 35%** (точка безубыточности для RR 1:3) AND **≥ 15 сделок**
- Fallback: если ни один порог не прошёл → `threshold=0.55`, `net=0.0` (no-trade protection)
- TP = 6.0 × ATR, SL = 3.0 × ATR (RR 1:2)

### Expert Importance Analysis
После обучения MoE вычисляет вклад каждого эксперта в XGBoost:
- Суммирует `feature_importances_` всех 8 колонок каждого эксперта
- Выводит таблицу с Long/Short/Total для всех 16 экспертов + `BASE_FEATURES`
- Как правило, топ-5: momentum, trend_follow, osob_rsi, vol_dynamics, volume

---

## Expert Ensemble (`models/experts.py`)

### 16 Experts — Current State (v12)

All 16 experts use **multi-horizon** targets (4 horizons: 3, 5, 8, 13) with independent LSTM heads. No multiclass experts remain — `MULTICLASS_NAMES = set()`.

| # | Expert | Type | Hidden | Target Derivation | Horizon-aware |
|---|--------|:----:|:------:|-------------------|:-------------:|
| 1 | `TrendExpert` | regression | 24 | `(pdi - mdi) / max(adx, 1)` → [-1, +1]; сила и направление тренда | ❌ (descriptive) |
| 2 | `VolExpert` | binary | 12 | `ATR[i+h]/ATR[i] > 1.05` → expansion(1)/contraction(0) | ✅ |
| 3 | `MomentumExpert` | regression | 24 | Z-score `future_return_h` скользящим окном; обрезан до [-1, +1] | ✅ |
| 4 | `ReversalExpert` | binary | 16 | 1 = цена возвращается к EMA20 через h баров (отскок при `|dist| > 0.02`) | ✅ |
| 5 | `VolumeExpert` | regression | 12 | `volume_ratio` (объём / скользящее среднее) | ❌ (descriptive) |
| 6 | `SRExpert` | binary | 12 | 1 = цена вблизи кластеризованных уровней S/R (в пределах 0.03%) | ❌ (descriptive) |
| 7 | `BreakoutExpert` | regression | 12 | Успешность пробоя N-бар диапазона (Close / max(High[N:])) | ❌ (descriptive) |
| 8 | `PullbackMAExpert` | regression | 12 | Откат к EMA + продолжение тренда (сила отскока) | ❌ (descriptive) |
| 9 | `OSOBRSIExpert` | regression | 12 | RSI mean reversion; сила перекупленности/перепроданности | ❌ (descriptive) |
| 10 | `TrendFollowExpert` | regression | 16 | ADX + EMA alignment; сила трендового движения | ❌ (descriptive) |
| 11 | `VolumeAtExtremesExpert` | regression | 12 | Объём на локальных пиках/впадинах | ❌ (descriptive) |
| 12 | `MicrostructureExpert` | binary | 16 | 1 = будущее направление цены совпадает с текущим OFI (`|bulk| > 0.2`) | ✅ |
| 13 | `VolumeProfileExpert` | binary | 16 | 1 = `close[i+h] > poc_price[i+h]`; цена закрывается выше POC | ✅ |
| 14 | `OrderFlowExpert` | regression | 12 | Симулированный buy/sell delta + cum delta divergence | ❌ (descriptive) |
| 15 | `VolatilityDynamicsExpert` | regression | 12 | Parkinson/GK/YZ волатильность; направление изменения | ❌ (descriptive) |
| 16 | `MarketRegimeExpert` | binary | 20 | 1 = торгуемый режим (ADX>25 + DI spread>10) или (низкая волатильность + choppy < 61.8) | ✅ |

**Notes:**
- `regression` → MSE loss; `binary` → BCEWithLogitsLoss. **Focal Loss не используется.**
- "Horizon-aware" означает, что target сдвинут в будущее на `h` баров.
- "Descriptive" эксперты оценивают текущее состояние рынка (один target на все горизонты).

### Loss Functions (в `BaseExpert.train_expert`)
| target_type | Loss |
|:-----------:|:----:|
| `binary` | `nn.BCEWithLogitsLoss(reduction='none')` |
| `regression` | `nn.MSELoss(reduction='none')` |
| `multiclass` | `nn.CrossEntropyLoss()` (не используется — `MULTICLASS_NAMES` пуст) |

### Training Flow (MoE, `models/moe.py`)
1. **Load data** → `load_dataframe(ticker, 'H1')` + multi-timeframe features (D1, W1)
2. **Feature engineering** → `engineer_features()` + directional signals + curated context → ~70 base features
3. **Targets** → `create_targets(df)` — для каждого эксперта multi-horizon матрица `(n, H)`
4. **Train LSTM experts** → each expert: `SequenceDataset` → LSTM (20×input_size, hidden, 4 output heads) → early stopping (patience=8)
5. **Forward pass** → `predict_all(df)` → матрица `(n, 198)` = 70 base + 128 expert features
6. **Train XGBoost** → два отдельных классификатора (long/short) с регуляризацией
7. **Threshold search** → 0.55–0.75, WinRate > 35%, min 15 trades
8. **Save** → `{ticker}_moe_v12_rr1x2.joblib`

### Fast Inference
```bash
python main.py --ticker SBER --moe --load models/saved/sber_moe_v12_rr1x2.joblib
python monitor.py  # all tickers, models cached in memory
```
Loads pre-trained MoE → forward pass on newest data → XGBoost prediction → thresholded BUY/SELL/NEUTRAL. No LSTM retraining (~3 sec). Monitor preloads all 17 models in memory (~1.4s).

### Cascade Mode (DEPRECATED)
Cascade (последовательное соединение экспертов: trend → vol → momentum → reversal → volume → sr) помечен как устаревший. Параллельная архитектура показывает сопоставимые результаты при вдвое меньшем времени обучения. Флаг `--cascade` сохранён для обратной совместимости, но не рекомендуется к использованию.

---

## Legacy Models (not actively developed)

### Directional Probability Model
- **Features**: 8 directional signals + technical indicators + context
- **Targets**: `outcome_long` (1/0), `outcome_short` (1/0) — ATR-based SL/TP
- **Model**: MultiOutputClassifier (RandomForest) → P(success|long), P(success|short)
- **Signals**: rsi_signal, bb_signal, macd_signal, trend_signal, momentum_signal, volume_signal, directional_bias, signal_strength

### 3-Class Outcome Model
- **0** — No trade; **1** — Success (TP hit first, RR 1:2); **2** — Fail (SL hit first)

### DQN Reinforcement Learning
- **Actions**: HOLD (0), BUY (1), SELL (2)
- **Reward**: +2 for TP hit, -1 for SL hit
- **State**: 15 technical features + position info
- **ATR-based**: SL = 1.5×ATR, TP = 3.0×ATR

### Configuration (`config.py`)
```python
RL_CONFIG = {
    'n_epochs': 20, 'episode_size': 2000,
    'atr_mult_sl': 1.5, 'atr_mult_tp': 3.0,
    'hidden_dim': 128, 'lr': 1e-3, 'gamma': 0.99,
    'epsilon_start': 1.0, 'epsilon_end': 0.01, 'epsilon_decay_steps': 10000,
    'buffer_capacity': 50000, 'batch_size': 128, 'target_update': 200,
}
```

---

## Microstructure Computation Functions (`models/experts.py`)
| Function | Output | Description |
|----------|--------|-------------|
| `_compute_roll_spread()` | spread % | Roll (1984) implied bid-ask spread from price autocovariance |
| `_compute_corwin_schultz_spread()` | spread % | Corwin-Schultz (2012) HL-based spread estimator |
| `_compute_kyle_lambda()` | [-1,+1] z-score | Kyle's λ — price impact per unit volume |
| `_compute_amihud_illiq()` | [-1,+1] z-score | Amihud ILLIQ — \|r\|/(P·V) averaged |
| `_compute_garman_klass_vol()` | σ | GK volatility — 0.5·ln²(H/L) - (2ln2-1)·ln²(C/O) |
| `_compute_yang_zhang_vol()` | σ | YZ volatility — Rogers-Satchell + overnight |
| `_compute_volume_profile()` | dict | VAH%, VAL%, POC% from rolling 20-bar distribution |
| `_compute_order_flow_imbalance()` | [-1,+1] z-score | simulated buy/sell delta normalized |
| `_compute_cumulative_delta()` | [-1,0,+1] | cum delta divergence vs price |
| `_compute_structural_breaks()` | [-1,+1] | z-score of slope change between windows |