You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5487c08
Browse filesBrowse the repository at this point in the historyBrowse files
Add Azure Blob Storage support to the DataFusion object storage layer (#73374)
* Add Azure Blob Storage support to the DataFusion object storage layer
DataFusion credential resolution only handled AWS and GCS, so a
wasb-backed DataSourceConfig raised Unknown connection type instead of
working, even though DataFusion's storage layer supports Azure. Add a
StorageType.AZURE, an AzureObjectStorageProvider using DataFusion's
MicrosoftAzure, and a wasb credential branch resolving the connection's
shared key, SAS token, or Azure AD service principal, falling back to
ambient auth when none are set.
The service principal fields (client_id, client_secret, tenant_id)
must all be set together or not at all -- DataFusion's binding panics
on a partial combination -- so they're only forwarded when the
connection has all three (tenant_id extra, login, and password).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Remove unused WasbHook install guard from DataFusion Azure credentials
The wasb credential branch never called WasbHook -- it only imported it
to detect whether apache-airflow-providers-microsoft-azure was installed,
unlike the aws branch which actually calls AwsGenericHook. Since Azure
credentials come entirely from the generic Connection object, this guard
and its extra dependency are unnecessary, matching how the GCS branch
already works without a provider-specific import.
* Reject unsupported Azure identity fields in DataFusion credential resolution
DataFusion's Azure binding has no equivalent for a raw connection
string, a specific managed/workload identity, or a URL-form SAS
token, so these connection extra fields were silently dropped and
the code fell back to whatever ambient credentials were available.
That can authenticate as a different identity than the one the
connection was configured for. Raise a clear error instead, matching
the same guard already used for unsupported GCS identity fields.
* Fix ambiguous object-storage list in LLMSchemaCompareOperator docs
Missing comma made "Azure Blob Storage Parquet" read as one item
instead of two separate examples in the supported source list.
* Document Azure Blob Storage support in the Analytics operator docs
The Analytics operator docs listed Azure as supported in the intro
line but not in the Supported Storage Systems list, still flagged it
under "not yet supported", and had no usage example or credential
docs, unlike S3 and GCS.
* Fix DataFusion Azure account resolution ignoring connection host
The wasb credential branch took the storage account name unconditionally
from the connection's login field. For service-principal auth, login holds
the client_id, not the account name, so a connection whose host correctly
named the storage account still had the object store target the client_id
GUID instead. A connection with no login at all (key/SAS auth) silently
fell back to the AZURE_STORAGE_ACCOUNT_NAME environment variable.
Resolve the account from host first, matching WasbHook's own behavior, and
fall back to login only when host is empty. Also format the storage type
name consistently with .value (avoids drifting str-Enum formatting across
Python versions) and move the sas_token parsing import to the top of the
file.
* Document that DataFusion's Azure storage only recognizes az:// URIs
abfs:// and abfss:// embed the storage account in the URI itself, a
different shape from az:// that engine.py does not parse; using them
currently fails with an unhelpful "Unsupported storage type" error with
no hint that Azure support exists under a different scheme.
* Fix silent identity/endpoint mismatches in DataFusion Azure credentials
Four issues surfaced by review, each verified against the real datafusion
binding and object_store source:
- An empty connection (no host or login -- the shape of the wasb_default
connection airflow db creates) sent the literal string "None" as the
account instead of omitting it, breaking the fallback to
AZURE_STORAGE_ACCOUNT_NAME that worked before the host-priority fix.
- A partial service-principal config (tenant_id set but login or password
missing) silently fell through to ambient auth, or sent the client
secret as a shared key, instead of raising -- authenticating as a
different identity than the one configured.
- The account name kept only the first DNS label, so a sovereign-cloud
host (*.chinacloudapi.cn, *.usgovcloudapi.net) or the Azurite emulator's
loopback URL was silently redirected to the public
*.blob.core.windows.net account of the same name; the binding has no
endpoint override to correct this after the fact.
- DataFusion's MicrosoftAzure always reads AZURE_* worker environment
variables before overlaying the connection's credential, and checks an
environment-derived access key or workload-identity token ahead of the
connection's SAS token or client secret. In the common AKS
workload-identity deployment, this meant the connection's credential was
silently ignored in favor of the pod's identity. The binding has no way
to skip its own environment read, so this can only be caught, not fixed.
* Read the legacy extra__<conn_type>__ prefix for wasb and GCS credentials
WasbHook and GoogleBaseHook still fall back to fields written as
extra__wasb__<name> / extra__google_cloud_platform__<name> -- the spelling
older Airflow connection UIs used for custom extra fields. This code only
looked at the bare key, so a connection created that way had its sas_token
or connection_string silently invisible: the unsupported-field guard never
saw a prefixed connection_string, and a prefixed sas_token fell through to
ambient auth instead of being used.
* Trim two overlong inline comments in Azure credential resolution
Same facts, fewer words -- the mechanism and consequence were spelled out
twice in slightly different phrasing.
* Simplify the Azure env-precedence docs paragraph
Same facts, plainer words -- "outrank", "ahead of", and "consulted last"
weren't adding anything a simpler verb wouldn't.
* Attribute AZURE_USE_AZURE_CLI to object_store, not Airflow
Names the actual library so it's clear this is DataFusion's underlying
binding's behavior, not an Airflow-defined setting, and contrasts it
directly with WasbHook's automatic az login to explain why it's worth
calling out at all.
* Fix docs spell-check failure on British spelling "honours"
The docs spellcheck job uses an American-English dictionary; "honors" is
already the spelling used elsewhere in this same doc tree.
* Simplify wasb credential resolution per review
Collapse the legacy-field lookup into one dict.get() call, and use the
walrus operator so tenant_id/sas_token are only computed where they're
actually checked, instead of unconditionally ahead of the branch. Also
consolidate the three shared-key test variants (password, shared_access_key
extra, account_key extra) into one parametrized test, picking up account_key
coverage that was previously untested.
* Fix Azure Blob Storage credential precedence gaps in DataFusion
The env-var precedence guard was missing two real object_store aliases
(AZURE_STORAGE_MASTER_KEY and the AZURE_CLIENT_ID/SECRET/TENANT_ID
triple, including the AKS workload-identity webhook's own env vars)
that silently outrank a connection's SAS token, while AZURE_STORAGE_SAS_KEY
never can and was raising for nothing.
The account-resolution check also rejected a non-public-cloud host even
when the worker had already set AZURE_STORAGE_ENDPOINT to fix exactly
that, so the documented workaround never actually worked.
Finally, the SAS-token branch was checked ahead of the shared-key extra,
the opposite of WasbHook.get_conn's order -- a connection with a current
shared_access_key but an expired SAS token would work through the hook
and fail here.
Co-Authored-By: Claude <noreply@anthropic.com>
* Move Azure env-var precedence constants next to their only caller
They were declared next to the unrelated _AZURE_PUBLIC_SUFFIX constant,
~180 lines from _get_credentials, the only place that reads them --
inconsistent with the file's existing convention of keeping a constant
next to its sole user.
Co-Authored-By: Claude <noreply@anthropic.com>
* Make the Azure credential env-var guard aware of which tier it protects
The flat guard list raised for env vars that could never actually
outrank a connection's own credential in object_store's precedence
chain -- most notably AZURE_FEDERATED_TOKEN_FILE against a plain
shared-key connection, which blocks the single most common Azure
Kubernetes deployment shape (the workload-identity webhook injects it
into every labelled pod) even though the connection's own key always
wins there.
Also closes a gap in the sovereign-cloud/endpoint-override check: an
Azurite-style host (bare hostname or host:port, with no login) fell
through an unrelated Active-Directory-ID fallback and silently
resolved to the literal string "None" instead of raising.
Co-Authored-By: Claude <noreply@anthropic.com>
* Consolidate Azure env-precedence tests into one parametrized case per tier
Each credential tier (access_key, service_principal, SAS) had a
"rejects when outranked" test and a separate "ignores when it can't be
outranked" test that only differed in which env vars were set and
whether the call was expected to raise. Folding both into a single
should_raise-parametrized test per tier keeps the same coverage with
far less repetition.
Co-Authored-By: Claude <noreply@anthropic.com>
* De-nest the endpoint-override check in _resolve_wasb_account
Left over from incrementally adding the emulator-host colon check;
the suffix check and the endpoint-override check were always meant
to be one combined condition, not a nested if.
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix inaccurate "dead" claim about AZURE_STORAGE_SAS_KEY in test docstring
object_store's resolve_sas_token() does read this variable into
self.sas_key; it's just checked after sas_query_pairs within that same
step, so it never wins once the connection sets sas_query_pairs. "Dead"
overstated that as unused entirely, which could mislead a future reader
into thinking the safety argument doesn't depend on precedence order.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
0 commit comments