Repository navigation
Add OIDC workload identity for Dag processor session credentials - #74700
ephraimbuddy wants to merge 6 commits into
Conversation
boring-cyborg
Bot
added
area:API
8b5d792 to
1b68b46
Compare
1b68b46 to
f95cb44
Compare
| NonemptyString = Annotated[str, Field(min_length=1)] | ||
|
|
||
|
|
||
| class WorkloadIdentityConfig(BaseModel): |
There was a problem hiding this comment.
This takes a single issuer, and grants are keyed on bare sub. Kubernetes issuers are per cluster, so Dag processors in a second cluster (per-team clusters in a multi-team setup, say) can't be trusted by the same API server. Should the top level be a list of issuer entries, each with its own grants, before the option ships and the shape becomes something we have to keep? If single issuer is intended, saying so in the option description would help.
There was a problem hiding this comment.
Single issuer is intended for now. The option description and the docs now say so, including that Dag processors whose identities come from different issuers, such as separate Kubernetes clusters, can't all use one API server. Several issuers can be added later without breaking this shape, for example by also accepting a list of these objects, each with its own grants.
| def __init__(self, config: WorkloadIdentityConfig, *, client: httpx.AsyncClient): | ||
| if config.audience in conf.getlist("execution_api", "jwt_audience"): | ||
| raise ValueError("Workload identity must use a dedicated audience") | ||
| if unknown := frozenset().union(*config.grants.values()) - get_configured_bundle_names(): |
There was a problem hiding this comment.
Since the core lifespan enters every sub-app lifespan in one AsyncExitStack, a grant naming an unconfigured bundle stops the whole API server, not just this exchange. Renaming a bundle and forgetting the grant would take down the UI and the Task Execution API that workers heartbeat against. Is that intended, or would logging the error and returning 503 from the exchange be enough?
There was a problem hiding this comment.
Not intended; that blast radius is too wide. A grant naming an unconfigured bundle no longer stops startup. The verifier logs an error naming both settings and refuses only that subject's exchanges with a 403. I went with 403 rather than 503 so processors don't spend their retry budget on an error only a config change can fix. Malformed trust config (the JSON shape, or reusing the runtime audience) still fails startup, since it is wrong from the moment it's written. Covered by test_unconfigured_bundle_grant_refuses_only_its_subject.
0f527b0 to
f56f9d2
Compare
Dag processor integrations currently need a trusted provisioner to write and rotate an Airflow session token on disk. Deployments that already provide renewable workload identities should not need a second credential file or an Airflow signing key on the processor. An opt-in identity exchange lets these integrations keep Airflow session credentials in memory while retaining the existing bundle grants and Job ownership model. Stable sessions avoid retiring active Jobs when the platform rotates an identity, and bounded lifetimes keep issued credentials within the authority of that identity. This is a follow-up to the Dag processor API foundation. The existing file-based provisioning path remains supported; wiring the standard Dag processor command to the HTTP client is separate work.
A grant naming a Dag bundle the API server does not have can only be fixed by correcting the configuration and restarting. Reporting it as a retryable 503 on every exchange made each processor spend its retry budget on an error that could never clear, and the cause only surfaced on the processor side. Components that host the in-process Execution API, such as the Dag processor manager and the triggerer, can never serve the exchange. They still loaded its trust settings from shared configuration, so a component with a narrower bundle list than the API server would have failed to start. A token with an unknown signing key forces a JWKS refresh, and anyone can send one. While that refresh waited on a slow identity provider, it also held up exchanges whose keys were already cached.
The Task SDK client moved to httpx2 while this change was in review, and its responses, timeouts and errors do not interoperate with httpx objects. The session token provider is built on that client, so it has to use httpx2 types for its timeouts and error handling to keep working. The API server's JWKS fetcher still uses httpx, as the rest of its token validation does.
The opt-out stopped the reissue middleware from re-validating the external workload identity as if it were an Airflow token. The middleware now only renews tokens that Airflow's own JWT check has already validated, and the exchange never runs that check, so the opt-out no longer prevents anything.
After a failed cold-cache JWKS fetch the API server waits 10 seconds before fetching again, and the SDK's default backoff retried every registration attempt inside that window more than half the time. A brief identity provider outage therefore failed registration, although the 503 told the client to retry. An identity supplier that raises its own exception type, or a JWKS endpoint that returns something other than a JSON object, turned a recoverable condition into failed requests, even for keys that were already cached. A grant naming an unconfigured Dag bundle stopped the whole API server, taking down the UI and the API that workers heartbeat against, when only the subject holding that grant needs to be refused. Catching type errors at the route also reported verifier bugs as invalid identities, when only older PyJWT releases raise them for malformed time claims.
Processors must stop trusting keys withdrawn by their issuer and recover from temporary identity-provider failures without exhausting retries during its cooldown. Authentication must also remain the boundary for incoming trace context.
f56f9d2 to
287d7f9
Compare
Allow Dag processor integrations to exchange an OIDC workload identity for short-lived Airflow session tokens kept only in memory. The API server validates the identity and configured bundle grants; existing file-based provisioning remains supported.
This builds on #74370 and adds API and client support. Wiring the standard
airflow dag-processorcommand to this client remains separate work.Performance check for the follow-up fixes (previous PR head
a051dcd460versus0f527b0cc1): medians from five alternating runs in Breeze, using the same environment and an unchanged control.The control changed by 0.19%; no regression was measured. Concurrent test load caused sample variation, so these results do not establish precise production speedups.
Was generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-6) following the guidelines