Skip to content

CONTEXT.md

Latest commit

 

History

History
264 lines (218 loc) · 11.3 KB

CONTEXT.md

File metadata and controls

264 lines (218 loc) · 11.3 KB
Raw

StockLens — Compact Context

For AI assistants: read this file first. It gives you ≥80% of the context needed to make safe edits. Drill into modules/*.md only when changing a specific module.

This file describes the current stable architecture. Historical changes (M2.x / F.x / H.x / G.x / B.x) live in ../CHANGELOG.md. Architectural decisions live in adr/.

One sentence

AI-driven post-close review system for A-share / HK / US equities: 9-source failover → 18 quant scoring models → LLM decision report → 10 push channels.

Positioning (ADR-0001): post-close only. All intra-day code paths have been deleted. Quotes are read from the stock_quote_snapshot table written by the analysis pipeline at end of run.

Tech stack

Python 3.9+ / FastAPI / SQLAlchemy (SQLite, WAL) / LiteLLM / pytest.

Iron rules (top-level)

  1. Code change must update docs in same commit. Sync map in ../CLAUDE.md.
  2. Source code is English-only. Comments / docstrings / log messages / string literals.
  3. models.py change requires Alembic migration in same commit.
  4. Sub-package import boundaries (ADR-0004): stocklens / api / bot may only import tickbridge.public.* and tickbridge.client.*. Internal tickbridge / querybus / pulsefan / quantcore / kvcache modules are off-limits to upper layers.
  5. No new module > 800 lines. Enforced by CI line-count check.

Repository layout

main.py                     Web entry. Starts FastAPI (+ optional Bot Stream).
api/                        FastAPI REST shell.
bot/                        DingTalk / Feishu Stream bots.
stocklens/                  Core business layer (post-close domain logic).
  pipeline/                 Pipeline orchestrator + 10 atomic stages.
  analyzer/                 LLM analyzer + prompt builder.
  agent/                    ReAct + multi-agent (5 agents incl. risk veto).
  services/                 Business services (scoring shell, finviz, bollinger,
                              tech indicators, history, system_config, ...).
  notification/             Stock-domain wrapper around pulsefan.
  search/                   Stock-domain wrapper around querybus.
  market/                   US macro + market env + profile.
  portfolio/                FIFO/avg cost, FX, snapshot replay.
  backtest/                 Thin shell over quantcore.backtest.
  storage/                  ORM models, Alembic, retention framework.
  repositories/             9 repos.
  data_service/             tickbridge-server HTTP adapter (only channel).
  config/                   pydantic-settings + Config dataclass + sub-configs.
  contracts/                Cross-layer Protocols & types.
  observability/            OTLP tracing + Prometheus metrics.
  utils/                    concurrency / cache_bootstrap / json_io / fail_open.

tickbridge-server/          ★ Independent Git repo, financial data daemon.
                              Only `tickbridge.public.*` and `tickbridge.client.*`
                              are importable from upper layers.
quantcore/                  ★ Independent Git repo. Pure quant compute (zero IO).
                              18 scoring models + indicators + stats + backtest.
querybus/                   ★ Independent Git repo. Search/news fan-out.
                              Single class: SearchProvider.
pulsefan/                   ★ Independent Git repo. Notification fan-out.
                              Single class: Notifier.
kvcache/                    ★ Independent Git repo. The cache.
                              Backed by cachetools + diskcache.

assets/                     Logos, icons, catalog (managed by AssetManager).
strategies/                 YAML strategy files.
templates/                  Jinja2 report templates.
alembic/                    DB migrations.
tests/                      pytest suite. Sub-packages have their own tests/.
docs/                       This file + modules/* + adr/* + dev_guide.

★ = independent Git repo, one-way dependency, releasable to PyPI (ADR-0007). The five ★ directories are gitignored in this repo and installed via <pkg> @ file://./<pkg> (see "Local development setup" below).

Local development setup

stocklens consumes five sibling repositories. Clone them next to (or inside, since they are gitignored) the stocklens working tree, then install editable:

git clone git@github.com:W-M-R/StockLens.git
cd stocklens
for pkg in tickbridge-server kvcache querybus pulsefan quantcore; do
  git clone "git@github.com:W-M-R/${pkg}.git" "${pkg/-server/}"  # tickbridge-server keeps its name
done
# tickbridge-server stays as 'tickbridge-server/'; the other four clone to
# kvcache/ querybus/ pulsefan/ quantcore/ (matching the file:// paths).
pip install -e .            # resolves the five `@ file://./<pkg>` deps

In production, swap the file:// entries in requirements.txt for tagged Git refs (e.g. quantcore @ git+https://github.com/W-M-R/quantcore.git@v0.1.0).

Core data flow

Web UI / REST API / Bot
  → Pipeline.run(stock_list)
    Phase 1  Shared prefetch       (PREFETCH pool)
             USMarketEnv + USMacro + MarketNews + Jin10
    Phase 2  Data collection       (DATA_FETCH pool)
             DataCollector → tickbridge daemon → SQLite upsert
    Phase 3  Real pipeline         (PIPELINE pool, enrich + LLM in parallel)
             enrich = quote → trend → tech indicators → intel → finviz →
                      ScoringEngine(18 models) → metadata report
             LLM    = PromptBuilder → LiteLLM Router → ResponseParser →
                      AnalysisResult
             (each enrich completion immediately submits its LLM task)
    → Report storage + Dispatcher → 10 push channels
    → [optional] Auto-backtest

Scoring formula

composite_score = macro 25 + fundamental 20 + valuation 20 +
                  technical+momentum 15 + distress 10 + sentiment 10

Total 100. Effectiveness validation pending — see Phase B in roadmap.

18 models live in quantcore/scoring/: Piotroski F-Score | Altman Z | Ohlson O | Rule of 40 | PEG | Beneish M | DuPont | Earnings Quality | Magic Formula | Dividend Safety | Insider Ownership | Options Sentiment | Momentum | Earnings Surprise | Short Interest | SCTR | Risk (Sharpe/Sortino/Beta) | Macro+Market Regime.

LLM configuration priority

LiteLLM YAML > LLM_CHANNELS multi-channel > legacy single-key (GEMINI / OPENAI / ANTHROPIC).

Embedded OSS libraries

Project principle (ADR-0005 sec 4): before adding any infra-class code, search PyPI first. Already embedded:

Need Library
Cache backend cachetools, diskcache, zstandard (auto-compress > 4 KiB)
HTTP client httpx (new code only — see ADR-0003)
HTTP retry tenacity
Circuit breaker pybreaker
Password / session passlib, itsdangerous
SSE sse-starlette
API rate limit slowapi
Boot env validation pydantic-settings
Sub-config validation pydantic.dataclasses.dataclass
JSON serialization orjson (via stocklens.utils.json_io)
asyncio loop uvloop
Fuzzy match rapidfuzz
LLM routing litellm
LLM JSON repair json-repair
Markdown render markdown2
Lint / format ruff
Pre-commit pre-commit
Coverage pytest-cov
Tracing opentelemetry + OTLP
Metrics prometheus-client

Concurrency

All ThreadPoolExecutor instances obtain worker counts from stocklens.utils.concurrency.get_pool_workers(name). Tunable via STOCKLENS_<NAME>_WORKERS / STOCKLENS_DEFAULT_WORKERS. 11 pools registered.

Cache namespaces

17 namespaces are registered in stocklens/utils/cache_bootstrap.py (boot-time kvcache.configure() call). Add namespaces there, not in business code. Includes 6 tb_* namespaces wrapping TickbridgeClient HTTP calls (L1+L2, persistent across pipeline restarts).

Cross-layer access rules

Domain Only-allowed entry point
Market data (price / OHLCV / chip / fundamental / index / Reddit / X) tickbridge.DataProvider or TickbridgeClient
Search / news querybus.SearchProvider
Notification / push pulsefan.Notifier
Pure quant compute (scoring / indicators / risk / correlation / backtest) quantcore.*
Cache kvcache.get_manager()

Direct imports of internal sub-modules of any of the above sub-packages are forbidden from stocklens/ / api/ / bot/. Guard in tests/unit/test_subpackage_boundaries.py (AST-scans all five packages).

Test layout & coverage

tests/                      Stocklens core (466 cases).
  unit/                     Root-level unit tests.
  unit/stages/              Pipeline stage units (48).
  unit/pipeline/            Orchestrator integration (mocked).
  unit/notification/        Sender contracts via pulsefan.
  unit/search/              querybus.providers contracts.
  unit/repositories/        In-memory SQLite repo tests.
  unit/tickbridge/          Failover + xueqiu field adapter.
querybus/tests/             Independent suite (31).
pulsefan/tests/             Independent suite (19).
kvcache/tests/              Independent suite (74, 80%+ coverage).

Total ~526 cases. Repo-wide line coverage 19.0%.

Under-covered hot-spots (target: 35% next quarter):

  • stocklens/portfolio/portfolio_service.py (1306 lines)
  • stocklens/pipeline/orchestrator.py (941 lines)
  • stocklens/analyzer/llm_analyzer.py

Key documents

Topic Doc
This file (architecture overview) docs/context.md
Module-level details docs/modules/<module>.md
Architectural decisions docs/adr/*.md
Change history CHANGELOG.md
Code → doc sync map CLAUDE.md
Dev guide / common commands docs/dev_guide.md
Data source matrix docs/datasource_matrix.md
Trading strategies docs/strategies.md
Data flow detail docs/data_flow.md

Open technical debt

Tracked here so newcomers can find them quickly. Detail in CHANGELOG.

  1. Test coverage 19% → 35% — three large business modules under-covered.
  2. portfolio_service.py 1306 / orchestrator.py 941 / settings.py 1161 exceed the 800-line soft limit. Splits scheduled but blocked on sub-config migration (B1).
  3. Scoring model validation — partial coverage (Phase B, 2026-05). See docs/scoring_validation.md. Only 2 of 18 models are price-derived and validatable from current DB. The remaining 16 (fundamental / valuation / macro / sentiment) need data backfill before validation. Of the 20 testable raw price signals, 8 have |t-stat| ≥ 3 at 20d, 12 are noise. Notable: sctr underperforms raw px_vs_ma200 (composite weighting suspect); ret_3m and rs_spy_60 are 0.98-collinear (double-counting in composite). Re-run via scripts/scoring_validation/{run_validation,analyze}.py.
  4. Config sub-config migration — 25+ call sites still use top-level flat access (config.realtime_source_priority) instead of config.datasource.realtime_source_priority.
  5. requests permanent coexistence — see ADR-0003. 7 modules in tickbridge-server stay on requests permanently. All new HTTP code must use httpx.