|
| 1 | + .. Licensed to the Apache Software Foundation (ASF) under one |
| 2 | + or more contributor license agreements. See the NOTICE file |
| 3 | + distributed with this work for additional information |
| 4 | + regarding copyright ownership. The ASF licenses this file |
| 5 | + to you under the Apache License, Version 2.0 (the |
| 6 | + "License"); you may not use this file except in compliance |
| 7 | + with the License. You may obtain a copy of the License at |
| 8 | +
|
| 9 | + .. http://www.apache.org/licenses/LICENSE-2.0 |
| 10 | +
|
| 11 | + .. Unless required by applicable law or agreed to in writing, |
| 12 | + software distributed under the License is distributed on an |
| 13 | + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 14 | + KIND, either express or implied. See the License for the |
| 15 | + specific language governing permissions and limitations |
| 16 | + under the License. |
| 17 | +
|
| 18 | +============= |
| 19 | +DuckDB on AWS |
| 20 | +============= |
| 21 | + |
| 22 | +`DuckDB <https://duckdb.org/>`__ is an in-process analytical database. It can read and write data in |
| 23 | +Amazon S3 directly, which makes it an affordable way to run transforms that do not justify the scale |
| 24 | +of a distributed cluster. |
| 25 | + |
| 26 | +DuckDB is not an AWS service, so the generic hook, operator and ``duckdb`` connection type live in the |
| 27 | +:doc:`DuckDB provider <apache-airflow-providers-duckdb:index>`. What this provider adds is the AWS |
| 28 | +auth. Using an Airflow AWS connection to create the DuckDB secret that grants access to S3. |
| 29 | +This means there is no credential wiring needed in the Dag. |
| 30 | + |
| 31 | +You run :class:`~airflow.providers.duckdb.operators.duckdb.DuckDBExecuteQueryOperator` and point it |
| 32 | +at a ``duckdb_aws`` connection when the SQL should reach S3. |
| 33 | + |
| 34 | +Prerequisite Tasks |
| 35 | +------------------ |
| 36 | + |
| 37 | +.. include:: ../_partials/prerequisite_tasks.rst |
| 38 | + |
| 39 | +DuckDB support is an optional extra |
| 40 | + |
| 41 | +.. code-block:: bash |
| 42 | +
|
| 43 | + pip install 'apache-airflow-providers-amazon[duckdb]' |
| 44 | +
|
| 45 | +.. note:: |
| 46 | + |
| 47 | + S3 access needs DuckDB's ``httpfs`` and ``aws`` extensions. This provider installs them if they |
| 48 | + are missing by default. If your workers should not download anything at runtime, set |
| 49 | + ``autoinstall_extensions=False`` and make the extensions available yourself, either |
| 50 | + in an ``extension_directory`` or baked into your image. |
| 51 | + |
| 52 | +.. _howto/connection:duckdb_aws: |
| 53 | + |
| 54 | +Run a DuckDB query against Amazon S3 |
| 55 | +==================================== |
| 56 | + |
| 57 | +Two connections are involved: |
| 58 | + |
| 59 | +* A **DuckDB on AWS** (``duckdb_aws``) connection which configures DuckDB itself, i.e.: *which DuckDB database and with what settings*. |
| 60 | + Its id is what you pass to |
| 61 | + :class:`~airflow.providers.duckdb.operators.duckdb.DuckDBExecuteQueryOperator` as ``conn_id``, and its |
| 62 | + type is also what selects :class:`~airflow.providers.amazon.aws.hooks.duckdb.AwsDuckDBHook`. |
| 63 | +* An **AWS** (``aws``) connection says *which AWS identity*, i.e.: keys or role, region, endpoint, etc. The DuckDB |
| 64 | + connection points at it through ``aws_conn_id``, which defaults to ``aws_default``. |
| 65 | + |
| 66 | +Credentials stay in the usual AWS connection. The ``duckdb_aws`` connection can be empty (though it must exist) and often |
| 67 | +will be with DuckDB's ability to use in-memory databases. Create it, name it whatever you like, and leave every field blank |
| 68 | +to get an in-memory database. |
| 69 | + |
| 70 | +Given that connection, the hook loads the ``httpfs`` and ``aws`` extensions and creates the S3 secret, |
| 71 | +so the task supplies only SQL: |
| 72 | + |
| 73 | +.. exampleinclude:: /../../amazon/tests/system/amazon/aws/example_duckdb.py |
| 74 | + :language: python |
| 75 | + :dedent: 4 |
| 76 | + :start-after: [START howto_operator_aws_duckdb] |
| 77 | + :end-before: [END howto_operator_aws_duckdb] |
| 78 | + |
| 79 | +Pass the id explicitly. With no ``conn_id`` the operator uses ``duckdb_default`` and the vendor-neutral |
| 80 | +hook, which has no S3 access. |
| 81 | + |
| 82 | +The DuckDB connection carries the database and the engine settings; leave its ``Database path`` empty for |
| 83 | +an in-memory database, which suits a task that reads from S3 and writes back to it. Everything |
| 84 | +:class:`~airflow.providers.amazon.aws.hooks.duckdb.AwsDuckDBHook` accepts can be set in the connection |
| 85 | +``extra``, and ``hook_params`` on the operator overrides it per task. Set ``memory_limit`` and |
| 86 | +``threads`` explicitly: DuckDB otherwise sizes itself from the resources it detects on the host. |
| 87 | + |
| 88 | +.. code-block:: python |
| 89 | +
|
| 90 | + hook_params = {"memory_limit": "2GB", "threads": 4} |
| 91 | +
|
| 92 | +.. note:: |
| 93 | + |
| 94 | + ``aws_conn_id`` resolves on whether the key is present, not on its value, because ``None`` is a |
| 95 | + valid choice for it rather than the absence of one. Set it to ``null`` in the ``extra`` (or |
| 96 | + ``None`` in ``hook_params``) to use no Airflow AWS connection at all and let the AWS SDK find |
| 97 | + credentials in the environment, such as environment variables or an instance role. |
| 98 | + |
| 99 | +Using the hook directly |
| 100 | +======================= |
| 101 | + |
| 102 | +For work that is not a single statement (e.g. chaining several queries against one database, or returning |
| 103 | +results to Python) you may use the hook directly. Opening one connection for several statements also |
| 104 | +avoids paying connection setup and extension loading more than once per task: |
| 105 | + |
| 106 | +.. code-block:: python |
| 107 | +
|
| 108 | + @task |
| 109 | + def compare_regions(): |
| 110 | + hook = AwsDuckDBHook(aws_conn_id="aws_default") |
| 111 | + with hook.get_conn() as conn: |
| 112 | + conn.execute("CREATE TABLE sales AS SELECT * FROM read_parquet('s3://my-bucket/sales.parquet')") |
| 113 | + return conn.execute("SELECT region, SUM(revenue) FROM sales GROUP BY region").fetchall() |
| 114 | +
|
| 115 | +How credentials reach DuckDB |
| 116 | +============================ |
| 117 | + |
| 118 | +DuckDB does not use ``boto3``. The hook therefore issues a ``CREATE SECRET`` statement that tells |
| 119 | +DuckDB how to authenticate, and the ``credential_strategy`` parameter selects how: |
| 120 | + |
| 121 | +``credential_chain`` (default) |
| 122 | + Issues ``CREATE SECRET (TYPE s3, PROVIDER credential_chain)``, which resolves credentials through |
| 123 | + the AWS SDK inside DuckDB's ``aws`` extension. Nothing secret is written into SQL text and the SDK |
| 124 | + refreshes expiring credentials itself, so a long query does not fail partway through when a set of |
| 125 | + temporary credentials expires. |
| 126 | + |
| 127 | + From DuckDB 1.4 onward the chain is resolved when the secret is created rather than when S3 is |
| 128 | + first read. If nothing can be resolved, the hook logs a warning and opens the connection anyway, |
| 129 | + so a query that does not use S3 still runs; one that does will fail on its own. A ``config`` |
| 130 | + secret that cannot be created raises instead, since explicit credentials failing is a |
| 131 | + misconfiguration. |
| 132 | + |
| 133 | +``config`` |
| 134 | + Resolves credentials through |
| 135 | + :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook` and writes them into the secret |
| 136 | + explicitly. This also works with a container-assigned role, because boto3 resolves that on the |
| 137 | + Airflow side, but the credentials are frozen when the connection is opened rather than refreshed. |
| 138 | + Needed when the Airflow connection carries static keys that the AWS SDK running inside |
| 139 | + DuckDB cannot see. The credentials become part of the SQL statement, so prefer |
| 140 | + ``credential_chain`` where it works. |
| 141 | + |
| 142 | +``none`` |
| 143 | + Creates no secret. Use for a DuckDB database that never touches S3. |
| 144 | + |
| 145 | +.. code-block:: python |
| 146 | +
|
| 147 | + DuckDBExecuteQueryOperator( |
| 148 | + task_id="query", |
| 149 | + conn_id="duckdb_aws_default", |
| 150 | + sql="SELECT COUNT(*) FROM read_parquet('s3://my-bucket/data.parquet')", |
| 151 | + hook_params={"aws_conn_id": "aws_static_keys", "credential_strategy": "config"}, |
| 152 | + ) |
| 153 | +
|
| 154 | +Non-default S3 endpoints |
| 155 | +======================== |
| 156 | + |
| 157 | +Set ``s3_endpoint_url``, or ``endpoint_url`` in the AWS connection extra, to reach an S3-compatible |
| 158 | +service or an S3 interface VPC endpoint. The hook translates it into the secret's ``ENDPOINT``, |
| 159 | +``URL_STYLE`` and ``USE_SSL`` clauses, selecting path-style addressing because virtual-host-style |
| 160 | +bucket names generally do not resolve against such endpoints. |
| 161 | + |
| 162 | +.. note:: |
| 163 | + |
| 164 | + The S3 secret is the only channel through which Airflow configuration reaches DuckDB. A DuckDB |
| 165 | + session with no secret ignores the Airflow connection entirely — including its region and endpoint |
| 166 | + falling back to whatever DuckDB's own settings and environment provide. |
| 167 | + |
| 168 | + DuckDB's ``httpfs`` extension reads credentials from the standard |
| 169 | + ``AWS_ACCESS_KEY_ID`` / ``AWS_SECRET_ACCESS_KEY`` / ``AWS_SESSION_TOKEN`` environment |
| 170 | + variables, so it can authenticate where those are set. What its HTTP client does not implement is |
| 171 | + the ECS container credential provider, so a container-assigned task role (often how a managed Airflow |
| 172 | + worker receives credentials) is invisible to it and S3 access fails with an opaque |
| 173 | + ``HTTP 403``. Both ``credential_chain`` and ``config`` handle that case; the difference is whether |
| 174 | + DuckDB or Airflow does the resolving, and therefore whether the credentials refresh. |
| 175 | + |
| 176 | +Reference |
| 177 | +========= |
| 178 | + |
| 179 | +* `DuckDB S3 API support <https://duckdb.org/docs/stable/extensions/httpfs/s3api>`__ |
| 180 | +* `DuckDB secrets manager <https://duckdb.org/docs/stable/configuration/secrets_manager>`__ |
0 commit comments