feat(mcp): implement MCP server per design doc (#153)

[RichardHightower]

Summary

Implement the native MCP server scoped in docs/plans/2026-mcp-server-design.md. This is the Phase-2 implementation companion to scoping issue #153.

Background

Issue #153 was filed deliberately small: "scope the MCP surface, name the tools, and sketch the entry point binary. Defer the actual implementation until UDS lands and the user reassesses." The strategic doc (docs/plans/2026-strategic-recommendations-integration.md:236-243) still positions MCP behind UDS in transport priority.

This ticket tracks the implementation work, gated on the owner's MCP-vs-UDS decision. It can sit deferred until that decision is made; the design is captured in the scoping doc so nothing is lost.

Scope (per design doc)

• New agent-brain-mcp/ Poetry package
• 4 tools: search, index, status, jobs (schemas derived from existing agent-brain-rag Pydantic models)
• 4 resources: agent-brain://status, agent-brain://folders, agent-brain://config (sanitized), agent-brain://jobs/{job_id}
• Dual backends: HTTP (default, lightweight) and embedded (optional [embedded] Poetry extra, sub-20ms target)
• --backend auto|http|embedded selector with no silent fallback
• --describe-surface flag for client introspection
• Schema-drift regenerate-and-compare test
• Import-isolation test (HTTP mode must not pull LlamaIndex/ChromaDB)
• Protocol-level smoke test (spawn over stdio, MCP initialize, list tools, call status)

Prerequisites

• Extract state-dir / base-URL resolver from agent-brain-cli/agent_brain_cli/config.py:328,376 into agent-brain-rag so CLI and MCP share one resolver. Update CLI to consume the extracted helper.

Out of scope (defer to follow-up tickets)

• MCP prompts
• Sampling / completion
• Streaming progress notifications
• Plugin auto-registration
• Wiring into root task before-push / task pr-qa-gate (opt-in via task install:mcp / task test:mcp only until the package is stable)

Decision gate

Do not start this work until the owner has reviewed UDS transport real-world characteristics and explicitly confirmed MCP should proceed. The design doc exists so the choice can be made informed; it does not commit to building.

Refs: #153

[RichardHightower]

Done. MCP server shipped and matured across v10.1 (v1: 7 tools/5 resources/6 prompts, stdio), v10.2 (v2: 16 tools, subscriptions, Streamable HTTP), v10.3 (v3: CLI-via-MCP + framework matrix). PyPI: agent-brain-ag-mcp.