+adapter: a proposal for multi-adapter support

[colin-k-rogers]

Preface

In the beginning… there were no adapters and there was much sorrow: how was dbt to manage the wide world of warehousing? Then there were many adapters and there was much rejoicing. Yet there remained a nagging voice in the back of our heads: why do I have to pick just one? Why must my dbt project be confined to a single adapter? For a spell the answer was obvious: storage and compute, while separated within a warehouse, were still inextricably tied. But the wheel continues to turn, and now shared catalogs can give multiple adapters access to the same physical tables.

Summary

A dbt project can execute different nodes in the DAG with different adapters. A node selects an adapter with +adapter; without that configuration, it uses the active profile target's default adapter.

The initial implementation supports cross-adapter dependencies that cross a DuckDB boundary. A model on one adapter can ref() a model on the other when the producer creates a physical table in a declared shared catalog. The producer may be:

• A SQL table model on either adapter.
• A catalog-backed SQL incremental model on the non-DuckDB adapter.

Python models, views, ephemeral models, and other resource types are not supported as cross-adapter producers. A view can still be the consumer: for example, a DuckDB view may ref() a shared Iceberg table produced by Databricks. The restriction applies to the object being reached across the adapter boundary, not to the consuming model's materialization.

DuckDB-origin incremental producers remain out of scope because DuckDB's full-refresh path renames intermediate and target relations, which attached Iceberg REST catalogs do not support safely.

Each adapter has its own parse, compile, and run context. dbt binds target, adapter, ref(), and source() to the adapter of the current consumer node. When adapters share a catalog, dbt renders the three-part relation name that is valid for the consumer.

Adapter-aware config resolution now covers models, seeds, snapshots, functions, and data tests. The broader design can extend beyond DuckDB boundaries and to additional producer resource types as portable catalog semantics become available.

Example

Here, stg_orders executes on Databricks and materializes an Iceberg table registered in Unity Catalog. orders_summary executes on DuckDB and calls ref('stg_orders').

The solid edge is the dbt dependency: there is still one ref() from the DuckDB node to the Databricks node. The dotted path explains physical access. Databricks writes the shared table, and DuckDB reads that same table through the Unity Catalog connection declared in catalogs.yml.

orders_summary may itself be a table, view, or incremental model. What matters at the boundary is that stg_orders is a supported physical producer.

Config

Profile

The active target declares each adapter that a node may select and marks one connection as the default.

analytics:
  target: dev
  outputs:
    dev:
      - type: databricks
        default: true
        host: dbc-example.cloud.databricks.com
        http_path: /sql/1.0/warehouses/example
        catalog: analytics
        schema: dbt_colin
        token: "{{ env_var('DATABRICKS_TOKEN') }}"
      - type: duckdb
        path: local.duckdb

Existing single-adapter profile mappings remain valid.

Node adapter

models:
  analytics:
    local:
      +adapter: duckdb

Adapter selection can be supplied through project config (+adapter), schema config, or inline config(adapter=...). Disabled nodes may name an adapter that is not configured in the active profile; enabled nodes must select a configured adapter.

Shared catalog

Catalog-backed producers identify a logical catalog_name. catalogs.yml defines how each supported adapter reaches that catalog.

catalogs:
  - name: shared
    type: unity
    table_format: iceberg
    config:
      databricks:
        file_format: delta
        use_uniform: true
      duckdb:
        endpoint: https://dbc-example.cloud.databricks.com/api/2.1/unity-catalog/iceberg
        warehouse: shared
        catalog_database: duck_shared

A model that should be reachable across adapters declares the catalog:

models:
  analytics:
    staging:
      +catalog_name: shared

Parsing

Parsing uses two stages for nodes that select a non-default adapter:

1. The discovery pass determines the effective adapter.
2. dbt re-renders project/schema config and then authoritatively replays the SQL with that adapter's Jinja environment and target context.

The authoritative pass supplies inline config(), hooks, macro dispatch, dependencies, rendered SQL, database, schema, aliases, and quoting. Parse-time adapter side effects from discovery are replaced rather than accumulated. If the adapter changes between discovery and authoritative rendering, parsing fails instead of iterating indefinitely.

During parsing, dbt verifies that:

• Every enabled node's selected adapter exists in the active profile.
• A model edge that crosses a DuckDB boundary uses a supported producer.
• A cross-adapter model producer declares catalog_name for a shared catalogs.yml v2 entry.
• The named catalog supports both the producer adapter and the consumer adapter.
• The requested catalog bridge is supported.
• Deferred relations satisfy the same producer materialization and language rules.

For the initial bridge, supported producers are SQL tables and non-DuckDB SQL incremental models. Views and ephemeral models are adapter-local definitions rather than portable catalog tables, and Python producers remain unverified.

Sources are handled separately because they do not execute on a producer adapter. A catalog-backed source() resolves the namespace required by the consuming adapter; an ordinary source retains its authored relation.

Catalog-backed models currently perform a full parse because partial-parse invalidation does not yet account for catalogs.yml changes safely. Partial-parse state does preserve each recorded relation's adapter identity and quoting.

Compilation and Run Context

An adapter is attached to a node context rather than to the invocation as a whole. Every compile and run context binds ref() and source() to the current consumer adapter, including hooks.

The task context owns an AdapterStore containing the adapters declared by the active profile. For each node, runtime selects:

• The adapter instance.
• An adapter-configured Jinja environment.
• A base context containing the adapter-specific target, database, and schema.
• The relation cache and worker connection for that adapter.

These contexts may be cached by adapter type but are not shared across adapters. Schema creation, relation caches, materialization, hooks, and execution all use the node adapter. DuckDB worker sessions initialize the shared catalogs they need before executing nodes.

A cross-adapter ref() keeps normal dbt DAG semantics. Only the physical spelling of the producer relation changes for the consumer. Deferral also preserves this routing: dbt infers or uses the deferred producer adapter, validates the catalog bridge, and renders the deferred relation in the consumer's namespace.

For incremental producers, the incremental model still executes entirely on its producer adapter. The consumer sees only the final catalog-backed table. This is currently supported when the producer is not DuckDB; DuckDB-origin incremental full refresh remains unsupported because of its rename-based swap flow.

Proposed Implementation

The implementation is separated so each change can be reviewed independently:

1. Parse relations with each node's adapter
2. Reset and safely load catalog state
3. Resolve catalog-backed refs and sources for consumers
4. Route runtime work through node adapters
5. Initialize DuckDB worker sessions
6. Preserve adapter routing through deferral
7. Render project and schema config with each node's adapter
8. Render inline config and hooks with each node's adapter
9. Support incremental cross-adapter producers

All nine changes are currently draft PRs. The series is represented as linked drafts because GitHub cannot use a branch in a fork as the base branch of a PR against dbt-labs/dbt.