security: API key auth on REST endpoints (#179)

[RichardHightower]

Summary

Closes the critical-severity unauthenticated REST API surface raised in #179. Adds opt-in AGENT_BRAIN_API_KEY enforced via the X-API-Key header by a single verify_api_key FastAPI dependency wired at the router level on every data router (/index, /index/cache, /index/folders, /index/jobs, /query, /graph). Health stays open by design. Server, CLI, and MCP move together so enabling auth doesn't silently break the MCP layer (the gap the original issue missed).

Locked design decisions

1. Default mode — strict on non-loopback. Loopback + empty key emits a loud warning and starts; non-loopback + empty key refuses to start (sys.exit(2) so process managers can distinguish a misconfiguration from a crash). Preserves single-user dev UX; fail-fast for shared hosts.
2. Scope — server + CLI + MCP in one PR. CORS wildcard (allow_origins=["*"]) tightening filed as a separate follow-up per the issue's own carve-out.
3. Docs gating. When AGENT_BRAIN_API_KEY is set AND DEBUG=false, /docs, /redoc, and /openapi.json are also hidden — there's no point requiring auth for endpoints whose schema is publicly browsable. DEBUG=true keeps them open.

Surface touched

Server — agent_brain_server/config/settings.py adds AGENT_BRAIN_API_KEY: str = "". New agent_brain_server/api/security.py (≈40 LOC) holds verify_api_key (constant-time compare via secrets.compare_digest). 6 routers get dependencies=[Depends(verify_api_key)]. api/main.py factors out _build_app() + _check_api_key_startup_gate(); the startup gate runs in run() before uvicorn binds. runtime.py extends RuntimeState with api_key: str | None = None; write_runtime() chmods runtime.json to 0o600 since it may now carry a secret.

CLI — DocServeClient(api_key=...) and DocServeClient.from_httpx(api_key=...) inject X-API-Key into the underlying httpx.Client default headers. transport.open_client resolves via the new resolve_api_key() (env → runtime.json → config.json → None). agent-brain init generates secrets.token_urlsafe(32) into project-local .agent-brain/config.json (chmod 0o600); --no-api-key opts out. agent-brain start exports the config.json key as AGENT_BRAIN_API_KEY in the server subprocess unless already set externally.

MCP — _resolve_api_key(state_dir) with precedence AGENT_BRAIN_MCP_API_KEY env → AGENT_BRAIN_API_KEY env → runtime.json::api_key → config.json::api_key → None. open_backend_client injects the resolved key into both HTTP and UDS httpx.Client default headers.

Docs — .env.example (repo root + agent-brain-server) document the new variable. README quickstart calls out the auto-generated key, the --no-api-key opt-out, and the non-loopback fail-fast. docs/CHANGELOG.md [Unreleased] Security entry captures the full inventory and the BREAKING note for non-loopback users.

Tests added (43 new)

• agent-brain-server/tests/unit/api/test_security.py — 6 tests for verify_api_key in isolation (no-op when key empty, 401 missing / wrong, 200 correct, case-insensitive header).
• agent-brain-server/tests/unit/api/test_auth_enforcement.py — 20 tests covering structural (every gated router carries the dependency, health does not) and behavioral (401 without header, 200 with correct header) layers across all 6 gated routers + health exemption.
• agent-brain-server/tests/unit/api/test_startup_gate.py — 9 tests covering exit 2 on 0.0.0.0, warn on loopback, all three loopback aliases (127.0.0.1/localhost/::1), silent when key set, and the /docs gating matrix across (key set/unset × DEBUG true/false).
• agent-brain-server/tests/unit/test_runtime.py — 3 new tests (chmod 0o600, api_key round-trip, default-None contract preserves backwards compat).
• agent-brain-cli/tests/test_api_key_propagation.py — 11 tests covering DocServeClient + from_httpx header injection, resolve_api_key precedence (5 cases), end-to-end through open_client.
• agent-brain-cli/tests/test_init_api_key.py — 6 tests for init key generation, --no-api-key opt-out, config.json 0o600, cross-project key uniqueness, and the post-init / pre-start config.json fallback.
• agent-brain-mcp/tests/test_api_key_propagation.py — 8 tests for the MCP precedence chain (5 cases) and HTTP / UDS header injection.

Test plan

• task before-push from repo root — exit 0 (~3min, runs format + lint + typecheck + tests across all 4 packages including the DR-5 monorepo gate)
• task pr-qa-gate from repo root — exit 0, MCP coverage 91.89% (above 80% floor)
• Per-package counts: agent-brain-server 1304 passed / 28 skipped (was 1275 baseline); agent-brain-cli 433 passed (was 410 baseline); agent-brain-mcp 468 passed / 96 deselected (was 451 baseline)
• Smoke Feat/phase1 finalization and qa gate #1 — agent-brain-serve on default loopback, no key → starts with warning, curl /health and curl /query both 200 (no auth required path preserved)
• Smoke feat: Implement BM25 & Hybrid Retrieval #2 — set AGENT_BRAIN_API_KEY=xxx, restart → curl /query without header 401, with -H "X-API-Key: xxx" 200, /health 200 either way
• Smoke T001: Add rank-bm25 and llama-index-retrievers-bm25 dependencies #3 — API_HOST=0.0.0.0 agent-brain-serve without key → exits 2 with clear log line
• Smoke T002: Update doc-serve-server dependencies #4 — agent-brain query "test" end-to-end via CLI with runtime.json::api_key present → 200 (CLI auto-injects header)
• Smoke T003: Add BM25_INDEX_PATH to settings #5 — agent-brain-mcp --transport stdio with AGENT_BRAIN_API_KEY env set → search_documents tool succeeds against authed backend

Out of scope (filed as separate follow-ups)

• CORS wildcard tightening — api/main.py:687-693 uses allow_origins=["*"] with allow_credentials=True. Tracked separately per the issue's own carve-out.
• Authorization: Bearer / OAuth 2.1 — covered by future MCP v4 (MCP v4: OAuth 2.1 for remote Agent Brain instances #188). X-API-Key is the API-key idiom; Bearer semantics are for OAuth tokens.

Plan

Full design and verification plan: docs/plans/2026-06-05-issue-179-api-key-auth.md

Closes #179.

🤖 Generated with Claude Code