Add data ingestion (ophir ingest) and trading-best-practices skill

.claude/skills/trading-best-practices/SKILL.md

@@ -0,0 +1,114 @@
+---
+name: trading-best-practices
+description: >-
+  Industry best practices for building/modifying the ophir stock-trading agent.
+  Use whenever writing or changing code that ingests market data, generates
+  trading signals, sizes positions, decides buy/sell/hold, executes orders,
+  backtests a strategy, or puts an LLM in the trading loop. Enforces paper-first
+  / dry-run defaults, deterministic auditable execution, a risk gate + drawdown
+  kill-switch, look-ahead/survivorship-bias-free backtests, and LLM-safety rails.
+---
+
+# Trading best practices
+
+Apply these whenever you touch the trading agent. They exist to *not lose money and
+not fool yourself*. Full rationale: `ophir-bot/trading-agent-blueprint.md`.
+
+## Non-negotiable invariants
+- **Paper-first.** `mode="paper"`, `dry_run=True`, `allow_live=False` are the defaults.
+  Going live needs an explicit, loud opt-in. Refuse to start against a live endpoint
+  unless `allow_live` is set on purpose.
+- **Fail safe, not open.** On any ambiguity (stale data, disconnect, breached limit) the
+  default action is *do nothing*, never *trade anyway*.
+- **Broker is the source of truth.** Never trust local memory for positions/cash;
+  reconcile against the broker every cycle.
+- **Deterministic, idempotent execution path.** Same inputs → same orders. Deterministic
+  `client_order_id` (e.g. `f"{date}:{symbol}:{side}"`) so a retry/double-run can't
+  double-trade.
+- **Observability.** Every decision + order goes to an append-only audit trail; you must
+  be able to answer "why did it buy X on day Y?" months later.
+
+## Data & signal
+- **No look-ahead.** The signal for day T may only use data through T-1's close. Lag every
+  feature; never `.shift()` the wrong way.
+- **Staleness & quality checks before inference.** Verify last bar is recent, no gaps/dupes,
+  tz-consistent (ophir is tz-naive). Stale feed → skip the cycle.
+- **Split/dividend adjustment must be consistent** across a window (don't mix adjusted and
+  raw). Reuse ophir's `extract_features` contract — don't re-derive the 13 features.
+- Use **all three** ophir targets (`r_close`, `upside`, `downside`) for a risk-aware score,
+  not just `r_close`.
+
+## Position sizing & risk
+- **Volatility targeting is the default** sizing method (scale each name inversely to its
+  vol; scale the book to a target annual vol, e.g. 15%). Equal-weight is only a baseline.
+- **Fractional Kelly only** (¼–½). Full Kelly is too aggressive and very sensitive to
+  estimation error — overestimating edge → risk of ruin.
+- **Constraints:** per-name cap (≤5%), sector caps, gross/net exposure bounds, liquidity
+  cap (% of ADV), turnover budget + no-trade band to cut churn.
+- **Drawdown kill-switch is the single most important control** — halt new risk on a
+  peak-to-trough breach (e.g. 20%) and alert a human. Add a daily-loss limit.
+- **Risk Gate (pre-trade):** a hard checkpoint *after* portfolio construction and *before*
+  the OMS that can **veto / scale / halt**. Property to hold: no gate output can push the
+  book past any configured limit.
+
+## Execution
+- Use **`alpaca-py`**, never the deprecated `alpaca-trade-api`.
+- **`Decimal`, never `float`**, for share quantities and prices (float rounding → rejected
+  orders).
+- **Retries only on idempotent ops** (`tenacity`, backoff + jitter). A submit that times
+  out is resolved by *querying order status*, not blind resubmit.
+- **Reconcile vs broker truth** each cycle; handle partial fills (re-target residual) and
+  rejects (log + alert).
+- **Delta-reconcile, never liquidate-and-rebuy.** Sells first (free buying power), then buys.
+- Prefer **marketable-limit or MOC/LOC** over naked market orders for a daily strategy.
+- **Gate on the market calendar** (`pandas_market_calendars`); respect PDT (<$25k) and T+1
+  settlement (size against `buying_power`, not raw cash). Single-run lock prevents
+  scheduler double-fire.
+
+## Backtesting & validation
+- **Two engines:** `vectorbt` for fast signal/parameter sweeps; a small **event-driven loop
+  that reuses your signal/portfolio/risk modules** for execution realism (backtest == live
+  by construction).
+- **The three biases that destroy backtests:** look-ahead, **survivorship** (don't backtest
+  today's S&P 500 over history — use point-in-time membership), and **overfitting /
+  data-snooping** (out-of-sample holdout, walk-forward, deflated Sharpe).
+- **Model real costs:** commission/fees, half-spread, slippage (`base + k·size/ADV`), market
+  impact. A pre-cost edge that dies after costs is the norm.
+- **Validation:** walk-forward; **purged & embargoed CV** when labels overlap in time (your
+  multi-day horizon leaks under naive CV); touch the out-of-sample lockbox once.
+- **Benchmark vs SPY**, report Sharpe/Sortino/MaxDD/Calmar/turnover. Pin a **golden-file
+  backtest** as a regression test.
+
+## LLM in the trading loop (critical — this build lets the LLM pick)
+LLMs **hallucinate**, are **non-deterministic**, and have a **knowledge cutoff**. Research
+shows their trading decisions are unstable and over-sensitive to input noise. Therefore:
+- **Ground every claim in tool-fetched real data**, never model memory. Require citations
+  for any number or fact. No fabricated prices/figures.
+- **The LLM's picks MUST pass the deterministic Risk Gate** and stay **paper / `dry_run`**.
+  An LLM never sizes or places an order that bypasses the gate.
+- **Log the full rationale** (inputs, research, debate, final picks) to the audit trail.
+- **Reduce instability:** sample/seed deliberately, prefer selective consensus across
+  multiple runs, keep prompts structured. Treat LLM output as *advisory even when it
+  "decides."*
+- The **safest** pattern is LLM-advisory + human-gated. If you move toward production,
+  migrate decision authority back to the deterministic model + risk rules and keep the LLM
+  for research and reporting.
+
+## Anti-patterns (reject these)
+Look-ahead/survivorship/overfitting; ignoring costs; trusting local state over the broker;
+fire-and-forget orders; no kill-switch; secrets in git or shared paper/live keys; skipping
+paper burn-in; float share quantities; naive timezones; scheduler double-fire; trading on
+stale data; **letting an LLM place/size orders without a deterministic gate.**
+
+## Recommended libraries (2026)
+`alpaca-py` (broker) · `vectorbt` + `backtesting.py`/`nautilus_trader` (backtest) ·
+`quantstats`/`pyfolio-reloaded` (analytics) · `cvxpy`/`riskfolio-lib` (sizing) ·
+`pydantic-settings` (config) · `structlog` (logging) · `tenacity` (retries) ·
+`pandas_market_calendars` (calendar) · `hypothesis` (property tests) ·
+`sqlalchemy`+sqlite/postgres (state).
+
+## References
+- `ophir-bot/trading-agent-blueprint.md` (primary).
+- López de Prado, *Advances in Financial Machine Learning*; Chan, *Quantitative Trading*;
+  Clenow, *Trading Evolved*.
+- Backtest-bias & LLM-trading-risk web sources (2026) are catalogued in `ophir-bot/roadmap.md`.

CHANGELOG.md

@@ -7,6 +7,63 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.1] - 2026-06-10
+
+### Fixed
+
+- `OHLCMulitClassPredictor.forward` now zeros the response-region rows of
+  `feature_input` before the feature MLP. `r_close` / `upside` / `downside` are
+  both input features and targets, so the self-attending response tokens could
+  copy the answer instead of forecasting; at inference (future rows zeroed) the
+  model collapsed to an identical-per-ticker constant. The change is
+  shape-preserving (existing checkpoints still load); the model must be retrained
+  to benefit.
+
+## [0.3.0] - 2026-06-10
+
+### Added
+
+- Trading-agent prediction layer under `ophir.agent`: `config` (pydantic-settings
+  with paper / dry-run / allow-live defaults and a live-mode guard), `audit`
+  (structlog append-only JSON audit trail), and `predict` (a `Forecast` dataclass
+  with `predict_ticker` / `predict_many` / `rank`). `ophir.agent.feed` gains
+  `forecast_window_tensors`, which builds a forward-looking window (real history
+  plus zeroed future rows) for genuine forecasts.
+- CLI commands `ophir predict <SYMBOL>`, `ophir rank <SYMBOLS> [--top-k]`, and
+  `ophir train` (full-US-market trainer, `<2024` train / `>=2024` validation
+  split, fine-tune or from-scratch via `--finetune-from` / `--max-steps`).
+  `register.fetch_base_trainer` gains a `max_steps` argument.
+
+### Changed
+
+- Add `pydantic-settings` and `structlog` dependencies. Source `torch` from the
+  PyTorch CUDA 13.0 index and pin `torch<2.11` (flex-attention compilation
+  regresses on 2.11+); refresh `uv.lock` accordingly.
+
+## [0.2.1] - 2026-06-04
+
+### Added
+
+- `trading-best-practices` Claude Code skill
+  (`.claude/skills/trading-best-practices/SKILL.md`) capturing trading-system
+  best practices (paper-first defaults, a pre-trade risk gate + drawdown
+  kill-switch, look-ahead/survivorship-bias-free backtests, and LLM-in-the-loop
+  safety) to guide future trading-agent work. Repo tooling; no runtime impact on
+  the `ophir` package.
+
+## [0.2.0] - 2026-06-04
+
+### Added
+
+- `ophir ingest <SYMBOL> [--days N]` command and the `ophir.agent` ingestion
+  modules: fetch a ticker's daily OHLC from Yahoo Finance and persist it
+  model-ready in the existing parquet layout, reusing
+  `ophir.ticker.extract_features` / `extract_model_data`.
+  `ophir.agent.feed.latest_window_tensors` bridges the most recent window
+  to the model's `(S, 13)` / `(S, 3)` input tensors. Yahoo data is fetched
+  split/dividend-adjusted (`auto_adjust=True`); ophir's split back-adjustment
+  is skipped on this path to avoid double-adjustment. No GPU required.
+
 ## [0.1.7] - 2026-05-20
 
 ### Fixed
@@ -125,7 +182,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   value yields a rotation of π.
 - Model validation and minor fixes.
 
-[Unreleased]: https://github.com/kwcantrell/ophir/compare/v0.1.7...HEAD
+[Unreleased]: https://github.com/kwcantrell/ophir/compare/v0.3.1...HEAD
+[0.3.1]: https://github.com/kwcantrell/ophir/compare/v0.3.0...v0.3.1
+[0.3.0]: https://github.com/kwcantrell/ophir/compare/v0.2.1...v0.3.0
+[0.2.1]: https://github.com/kwcantrell/ophir/compare/v0.2.0...v0.2.1
+[0.2.0]: https://github.com/kwcantrell/ophir/compare/v0.1.7...v0.2.0
 [0.1.7]: https://github.com/kwcantrell/ophir/compare/v0.1.6...v0.1.7
 [0.1.6]: https://github.com/kwcantrell/ophir/compare/v0.1.5...v0.1.6
 [0.1.5]: https://github.com/kwcantrell/ophir/compare/v0.1.4...v0.1.5

README.md

@@ -30,7 +30,14 @@ cloud.
   return, and an Ollama-backed chat panel.
 - **Checkpoint / data-dir management** (`ophir.register`) — Lightning
   `Trainer` factories and checkpoint loaders.
-- **`ophir` CLI** (Typer) — `serve` and `register` subcommands.
+- **Data ingestion** (`ophir.agent.ingest`) — pull a ticker's daily OHLC from
+  Yahoo Finance into a model-ready dataset (`ophir ingest`), reusing the
+  `ophir.ticker` feature pipeline.
+- **Model prediction** (`ophir.agent.predict`) — load the trained checkpoint and
+  forecast a ticker's next 90 days, ranking candidates by predicted return
+  (`ophir predict` / `ophir rank`).
+- **`ophir` CLI** (Typer) — `serve`, `ingest`, `predict`, `rank`, and `register`
+  subcommands.
 
 ## Requirements
 
@@ -62,11 +69,19 @@ pip install .           # or: pip install -e .   (editable, for development)
 
 ```bash
 ophir serve [--port 7860] [--share/--no-share] [--debug/--no-debug]
+ophir ingest <SYMBOL> [--days 730]
+ophir predict <SYMBOL>
+ophir rank <SYMBOL> [<SYMBOL> ...] [--top-k 5]
 ophir register massive-key <KEY>
 ```
 
 - `ophir serve` launches the Gradio UI (`ophir.ui.serve`). `--share` exposes a
   public link; `--debug` (default on) launches Gradio in debug mode.
+- `ophir ingest <SYMBOL>` pulls ~2 years of daily OHLC from Yahoo Finance into
+  a model-ready parquet (no GPU required); `--days` overrides the lookback.
+- `ophir predict <SYMBOL>` forecasts the next 90 days with the trained model;
+  `ophir rank <SYMBOLS> [--top-k 5]` ranks several by predicted return (both need
+  a CUDA GPU + checkpoint).
 - `ophir register massive-key <KEY>` stores a [MASSIVE](https://pypi.org/project/massive/)
   API key (used for data fetching) under the package's `.ophir/` directory.
 
@@ -119,6 +134,7 @@ uv run --group docs sphinx-build -W -b html docs docs/_build/html
 | `src/ophir/ticker.py` | Stock data loading, splits, feature extraction, datasets. |
 | `src/ophir/register.py` | Trainer factories, checkpoint loaders, data dirs. |
 | `src/ophir/ui.py` | Gradio UI and local-LLM chat. |
+| `src/ophir/agent/` | Trading-agent layers (data ingestion, prediction) built on the model. |
 
 ## License
 

docs/api/index.rst

@@ -17,3 +17,9 @@ see :doc:`../architecture` for its design.
    ophir.training_models
    ophir.ticker
    ophir.register
+   ophir.agent
+   ophir.agent.config
+   ophir.agent.audit
+   ophir.agent.ingest
+   ophir.agent.feed
+   ophir.agent.predict

docs/cli.rst

@@ -31,6 +31,62 @@ Option         Default   Description
    checkpoint onto a CUDA device. A GPU, a checkpoint, network access, and a
    local Ollama server are therefore required for this command to run.
 
+``ophir ingest``
+----------------
+
+Ingest a ticker's daily OHLC from Yahoo Finance into a model-ready dataset
+(:func:`ophir.agent.ingest.ingest`).
+
+.. code-block:: bash
+
+   ophir ingest <SYMBOL> [--days INTEGER]
+
+============== ========= ============================================
+Option         Default   Description
+============== ========= ============================================
+``SYMBOL``     --        Ticker symbol, e.g. ``AAPL`` (required).
+``--days``     ``730``   Calendar days of history to fetch.
+============== ========= ============================================
+
+Writes ``<DATA_DIR>/days/stocks/symbol=<SYMBOL>/data.parquet`` in the layout
+:class:`ophir.ticker.StockHanlder` reads, reusing
+:func:`ophir.ticker.extract_features`. The default ~2 years covers the model's
+365-day window plus rolling-feature warmup. No GPU required.
+
+``ophir predict``
+-----------------
+
+Forecast a ticker's next 90 days with the trained model
+(:func:`ophir.agent.predict.predict_ticker`); ingests the ticker first if
+needed. Requires a CUDA GPU and a trained checkpoint.
+
+.. code-block:: bash
+
+   ophir predict <SYMBOL>
+
+============== ============================================
+Argument       Description
+============== ============================================
+``SYMBOL``     Ticker symbol to forecast (e.g. ``AAPL``).
+============== ============================================
+
+``ophir rank``
+--------------
+
+Forecast several tickers and print the top picks by predicted cumulative return
+(:func:`ophir.agent.predict.rank`). Requires a CUDA GPU and a trained checkpoint.
+
+.. code-block:: bash
+
+   ophir rank <SYMBOL> [<SYMBOL> ...] [--top-k INTEGER]
+
+============== ========= ============================================
+Option         Default   Description
+============== ========= ============================================
+``SYMBOL``     --        One or more ticker symbols to rank.
+``--top-k``    ``5``     Number of top picks to print.
+============== ========= ============================================
+
 ``ophir register massive-key``
 ------------------------------
 

docs/conf.py

@@ -20,8 +20,8 @@
 project = "ophir"
 author = "Kalen Cantrell"
 copyright = f"{datetime.now():%Y}, {author}"
-release = "0.1.7"
-version = "0.1.7"
+release = "0.3.1"
+version = "0.3.1"
 
 # -- General configuration ---------------------------------------------------
 extensions = [

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ophir"
-version = "0.1.7"
+version = "0.3.1"
 description = "BERT-style masked transformer for stock OHLC prediction, with a Gradio UI and local-LLM chat."
 readme = "README.md"
 authors = [
@@ -20,9 +20,11 @@ dependencies = [
     "pandas>=2.3.3",
     "plotly>=6.5.2",
     "pyarrow>=23.0.0",
+    "pydantic-settings>=2.14.1",
+    "structlog>=26.1.0",
     "tensorboard>=2.20.0",
     "tensorboardx>=2.6.4",
-    "torch>=2.10.0",
+    "torch>=2.10.0,<2.11",  # pinned: 2.11+ changes flex-attention and breaks the model
     "transformers>=4.57.6",
     "typer>=0.21.1",
     "yfinance>=1.3.0",
@@ -35,6 +37,16 @@ ophir = "ophir.cli:app"
 requires = ["uv_build>=0.9.26,<0.10.0"]
 build-backend = "uv_build"
 
+# The model requires CUDA, so torch is pulled from the PyTorch CUDA 13.0 index
+# (matches the RTX 4080 + driver 13.1). Change the cuXXX to match another host.
+[[tool.uv.index]]
+name = "pytorch-cu130"
+url = "https://download.pytorch.org/whl/cu130"
+explicit = true
+
+[tool.uv.sources]
+torch = { index = "pytorch-cu130" }
+
 [tool.ruff]
 target-version = "py312"
 line-length = 100
@@ -115,4 +127,4 @@ filterwarnings = [
     "default::DeprecationWarning",
     "default::PendingDeprecationWarning",
     "default::FutureWarning",
-]
+]

src/ophir/agent/__init__.py

@@ -0,0 +1,7 @@
+"""Clean-room trading-agent layers built on top of the ophir model.
+
+Subpackages here reuse ophir's model and feature contract (e.g.
+:mod:`ophir.ticker`) but keep trading/agent logic separate from the model
+code. The first layer is market-data ingestion (:mod:`ophir.agent.ingest`
+and :mod:`ophir.agent.feed`).
+"""