feat: isolate connector deps via MCP subprocess architecture#24
Merged
Conversation
Each native connector (Snowflake, BigQuery) now runs as an isolated
MCP server subprocess launched by `uvx`, completely decoupled from the
core analytics-agent venv.
- Remove snowflake-connector-python, sqlalchemy-bigquery, google-cloud-bigquery
from core dependencies — core install is ~200MB lighter
- Delete engines/snowflake/ and engines/bigquery/ from core
- Add factory.ConnectorSpec: maps engine types to connector packages + env var
translation, builds the MCPQueryEngine stdio subprocess config transparently
- config.yaml syntax unchanged — `type: snowflake` and `type: bigquery` still work;
factory resolves them to `uvx analytics-agent-connector-{type}` subprocesses
- Each process in its own uvx-managed venv; two Snowflake engines = two processes,
one shared connector venv, separate credentials via env vars
- Add connectors/snowflake/ and connectors/bigquery/ — standalone MCP server packages
(FastMCP, 4 tools each) published separately to PyPI
- Fix _resolve_secrets in api/settings.py: use factory.get_secret_env_vars() instead
of engine class attribute (class no longer exists in core)
- Move test_bigquery_engine.py to connectors/bigquery/tests/
Closes #20
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Docker:
- ARG CONNECTORS="snowflake bigquery" in Dockerfile; pre-bakes connector
packages via `uv tool install` at image build time so they work offline
- Override: --build-arg CONNECTORS="snowflake" or CONNECTORS="" for bare image
Backend:
- GET /api/connectors/{type}/status — checks `uv tool list` for fast-path
- POST /api/connectors/{type}/install — runs `uv tool install` (idempotent, 120s timeout)
Frontend:
- ConnectorInstallStep: checks status on mount; if already installed (pre-baked
Docker image or previous UI install), calls onReady() immediately (fast path)
without showing anything to the user; otherwise shows install button + progress
- AddConnectionFlow: native transport plugins now route through install step
before the config form; mcp-stdio/mcp-sse plugins go straight to form
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
factory.py: build_mcp_config now uses shutil.which() to check if the connector binary is on PATH (placed there by `uv tool install`). Falls back to `uvx <package>` only if binary not found. This avoids uvx trying to resolve the package from PyPI when it was installed from a local path. Add README.md to both connector packages (required by hatchling build). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
MCPQueryEngine: store client reference on self after get_tools() to prevent GC from closing the stdio subprocess between tool calls. Remove incorrect __aenter__ usage (not supported in langchain-mcp-adapters >= 0.1.0). aclose() now just clears the references. Dockerfile: add /root/.local/bin to PATH so `shutil.which` finds connector binaries installed by `uv tool install` (which places them in ~/.local/bin, not in the venv). Note: grpcio (BigQuery dep) has SIGILL on arm64 Linux containers — use --platform linux/amd64 or deploy to x86 hosts. Both connectors verified working in x86 Docker against live BigQuery (customers, sales tables). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Cloud deployments (AWS, GCP, Azure) are x86_64. Explicitly declaring --platform=linux/amd64 in the FROM lines ensures: - CI builds produce the correct production-target image - Mac Apple Silicon developers get a consistent build (Rosetta emulation) - Connector packages (e.g. grpcio in BigQuery deps) use x86 binaries Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Hardcoding --platform=linux/amd64 is an anti-pattern (Docker lint rule FromPlatformFlagConstDisallowed). ARM adoption is accelerating — AWS Graviton4 now powers 33% of new EC2 workloads, Azure Ampere is GA. grpcio 1.78.0+ ships prebuilt ARM64 wheels so the SIGILL we hit locally was a Mac/Docker Desktop QEMU artifact, not a real ARM incompatibility. Multi-platform builds should use: docker buildx --platform linux/amd64,linux/arm64 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
'WHERE table_schema != INFORMATION_SCHEMA' counts tables across all databases the MySQL user can see (103 on a shared server). Replace with DATABASE() so the count reflects only the configured database (8). Also fixes quickstart.sh: token validation endpoint and macOS bash 3.2 empty array expansion. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The previous approach executed raw SQL (information_schema query) which is MySQL-specific. PostgreSQL, SQLite, and MCP engines would fall through to list_tables anyway. Drop the SQL path entirely — list_tables is already correct, scoped to the configured database, and works for every engine type. Unifies the message to 'tables accessible' in all cases. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…uery
SimpleFormShell: add 'json' field type — resizable textarea with inline
JSON validation, green ✓ / red ✗ status indicator, and pretty-print on
blur.
BigQueryForm: custom form component with Test button that calls
POST /api/connectors/bigquery/test before saving. Result appears inline
("Connected — 2 tables accessible" or error message).
Backend: POST /api/connectors/{type}/test — ephemeral test endpoint that
instantiates the engine from the supplied config+secrets without saving,
calls list_tables, and returns ok/message.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
MCP tools from langchain-mcp-adapters are async-only — invoke() raises "StructuredTool does not support sync invocation". Use ainvoke() instead. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Two fixes found while testing BigQuery Test button end-to-end in Docker:
1. cryptography<43: versions 43+ use hardware crypto instructions (SHA/AES
extensions) not available in Docker Desktop's ARM64 VM, causing SIGILL
during google.auth.crypt initialization. Pin to 42.x which uses portable
instructions. Not needed on real Linux ARM64 cloud hosts.
2. MCP tools return [{type:text, text:"[...]"}] content blocks, not a plain
string. Unwrap the first content block before parsing the table list so
the count reflects actual tables, not the number of content blocks (1).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
MCP tools return [{type:text, text:"...", id:"..."}] content blocks.
When execute_sql comes from an MCP connector, orjson.loads gives a list,
then result.get("columns") throws AttributeError → caught as is_error=True
→ red display even though the data is correct.
Unwrap the first content block text before the rest of the pipeline runs,
so all tool handling paths (execute_sql, list_tables, get_schema, etc.)
see a plain JSON string regardless of whether the tool is native or MCP.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Same Docker Desktop ARM64 SIGILL as bigquery — cryptography 43+ uses hardware crypto instructions not available in Docker Desktop's ARM64 VM. Pin to <43 so snowflake.connector imports cleanly on Mac dev builds. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…T support) The edit view shows a rich 5-tab auth section (Password / Private Key / SSO / PAT / OAuth); the add-new form had only a bare SSO button. Now consistent: - SnowflakeForm rewritten to embed SnowflakeAuthSection directly; the auth section's "Connect with Key" / "Sign in" button acts as the submit - factory.py: add private_key and pat_token to Snowflake env_map so both are forwarded as SNOWFLAKE_PRIVATE_KEY / SNOWFLAKE_PAT_TOKEN to the connector subprocess (was only in secret_env_vars, not env_map) - factory.py: add pat_token to secret_env_vars mapping Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Previous form showed all 5 config fields plus a full auth section — overwhelming. New design: - Name + Account at top (always required) - Auth method tabs (Password / Private Key / SSO / PAT / OAuth) second - Only the fields for the selected method appear below the tabs - Warehouse / Database / Schema collapsed into an ▶ Advanced section - Method-specific submit label: "Connect with Key", "Sign in with SSO", etc. Selecting Private Key shows: Account, Username, PEM textarea, Passphrase. That's all. Nothing else unless you open Advanced. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ngines
The test_connection endpoint was calling get_tools() (sync stub, returns [])
and list_tables.invoke() for all engine types. MCP engines need async:
- get_tools_async() to launch the subprocess and discover tools
- ainvoke() to call the tool
- MCP content block unwrapping (same as streaming.py and connectors.py)
Result: POST /api/settings/connections/{name}/test now returns
"Connected — N tables accessible" for MCP-backed connectors instead
of the generic "Engine connected" fallback.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
UI-created connections store private_key / password / pat_token directly in integrations.config (not in env vars). The status check only read env vars, so connections saved via the UI always showed Unconfigured. Now checks both env vars (legacy yaml flow) and conn_cfg (UI flow) for all three credential types, and PAT is now included in the has_creds test. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Add required_keys / credential_keys to ConnectorSpec and an is_configured(conn_cfg, sso_connected) method. The logic for deciding whether a connection is 'connected' or 'unconfigured' now lives on the spec itself, not scattered across four places in settings.py. settings.py drops all ad-hoc credential checks for snowflake/bigquery and delegates to _CONNECTOR_MAP[type].is_configured(). Adding a new connector type no longer requires touching settings.py for status logic. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Backend: - Add auth_method: str | None to ConnectionStatus — detects active method from conn_cfg (private_key / pat_token / password) and the SSO state. Frontend: - Connection.auth_method typed and wired to SnowflakeAuthSection.connectedAuth in the edit view — shows the "Connected as USER via Private Key" banner and pre-selects the correct tab instead of showing blank fields. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
When connectedAuth is set (existing connection with private key auth): - Username field is pre-filled with connectedAuth.username instead of blank - Private Key textarea shows '✓ Key saved — paste a new key above to rotate it.' when the field is empty and the active method is privatekey Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
ConnectorInstallStep was treating a 404 from /api/connectors/{type}/status
(returned for built-in plugins like DataHub) as "needs install", showing
the "Install connector" screen for platforms that are already embedded in
the core package. On any error (including 404), call onReady() immediately
to proceed straight to the connection config form.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Each engine connection card now has the same master toggle switch as context platform cards. Toggling off hides the data source from the agent's available engines (stored in disabled_connections). The toggle reuses the existing toggleConnection / saveToolToggles infrastructure. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The click handler was passing !has (current state) instead of has (the enable target). Fixed: passing has=true enables, has=false disables. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…dropdown Backend: GET /api/engines now reads disabled_connections from the settings table and excludes those engines — so the chat dropdown only shows enabled data sources. Frontend: toggleConnection in connectionSettings store refreshes the engine list via listEngines() after saving, so the dropdown updates immediately when a data source is toggled on or off in Settings. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Key file map: replace stale engines/snowflake reference with factory.py, engines/mcp/engine.py, api/connectors.py, and connector packages - Integrations section: rewritten for MCP subprocess model — ConnectorSpec, is_configured(), env var forwarding, credential storage in config - "Adding a new query engine": 5-step guide for the connector package pattern (FastMCP server, ConnectorSpec, _KNOWN_TOOLS, frontend plugin, Dockerfile) - Common pitfalls: remove stale resolver.py/Snowflake notes, add connector dep isolation rule and generic type coercion guidance Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- settings.py (config.yaml Snowflake branch): password variable was removed during OO refactor but still referenced in the yaml-source password field; re-add reading from env + conn_cfg - settings.py (DB Snowflake branch): same fix in the yaml-source block - connectors.py: sort imports (stdlib before local) - __init__.py, factory.py, streaming.py: ruff format Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes all heavy connector dependencies (Snowflake, BigQuery/GCP) from the core
analytics-agentpackage. Each native connector now runs as an isolated MCP server subprocess launched viauvx, with its own venv and deps.What changed
Architecture
snowflake-connector-python,sqlalchemy-bigquery,google-cloud-bigqueryremoved from core depsengines/snowflake/andengines/bigquery/deleted from coreconnectors/snowflake/andconnectors/bigquery/— standalone FastMCP server packagesfactory.py:ConnectorSpecdataclass maps engine types → subprocess commands + env var translation;config.yamlsyntax unchanged (type: snowflakestill works)ConnectorSpec.is_configured()— OO status check replaces scattered if/elif blocks insettings.pyDocker
ARG CONNECTORS="snowflake bigquery"— pre-bakes connector packages viauv tool installat build time (offline-capable)ENV PATH="/app/.venv/bin:/root/.local/bin:$PATH"so installed binaries are discoverableUI
ConnectorInstallStep: checksGET /api/connectors/{type}/statusbefore showing config form; fast-path if already installed (pre-baked Docker or prior UI install)/api/enginesfilters bydisabled_connectionsauth_methodfield in connection status so the edit view pre-selects the correct auth tab and shows "✓ Key saved"CI
helm.yml:helm lint+ port consistency check + hook ordering check +kind-basedhelm installintegration testBugs fixed (found during testing)
streaming.py: unwrap MCP content blocks[{type:text,text:"..."}]before parsing — was marking execute_sql results as errors (red display)get_tools_async()+ainvoke()for MCP enginescryptography<43pin — 43+ uses CPU instructions not available in Docker Desktop's ARM64 VMquickstart.sh: token validation endpoint (/api/v2/graphql→/api/graphql) + bash 3.2 empty array fixTest plan
uv sync— no snowflake/bigquery packages in core venvjust startworks afteranalytics-agent bootstrapdocker build && docker run→/healthreturns 200,/api/enginesreturns configured engineshelm lint && helm template | <checks>pass in CI🤖 Generated with Claude Code