Repository navigation
Run Dag processors without metadata database access - #74233
Closed
ephraimbuddy wants to merge 5 commits into
Closed
ephraimbuddy wants to merge 5 commits into
ephraimbuddy wants to merge 5 commits into
Conversation
boring-cyborg
Bot
added
area:API
ephraimbuddy
force-pushed
the
dag-processor-token-auth
branch
2 times, most recently
from
October 5, 2026 20:17
98a1573 to
8b59c47
Compare
ephraimbuddy
changed the title
1 task done
ephraimbuddy
force-pushed
the
dag-processor-token-auth
branch
from
October 14, 2026 11:19
245aa9d to
3884aff
Compare
Dag processors need to publish results and reconcile source changes through the API before database credentials can be removed from their deployment. Keep the opt-in command, recovery behavior and deployment integration together on top of the scoped authentication foundation.
The Task SDK client raises httpx2 exceptions and requires matching transport and timeout objects. Integrate apache#74728 so isolated processors can recover from API failures and retain structured error reasons, including in the manager and job runner paths introduced by this branch.
An invalid access_control made the API server fail the whole publication with a server error, so users saw nothing about why the file's Dags stopped updating. FAB now reports it as a validation error, which the API server can tell apart from a database failure. Server errors that persist after retries are usually outages rather than bad content, so reporting them as import errors was misleading. The import error that replaces a refused result could itself exceed the publication limit when the refused result carried large metadata, and the default database-backed processor loaded an API client stack it never uses.
Rejected inventories and stale claim state could prevent later priority requests from running, and equal callback payloads could remove the wrong claimed request. Parse-time API calls also timed out after one second even when the configured limit was longer. Parse and inventory checkpoint keys could collide. Supported older FAB releases surfaced invalid access control as server failures, while custom container UIDs and verbose CLI runs did not behave consistently.
A bundle whose refresh kept failing held its callbacks indefinitely, and in API mode those parked callbacks used up the claim limit for every bundle. Files with callbacks or priority requests also lost their place at the front of the queue while their bundle waited for a refresh. Publications slower than a second were never acknowledged even when the server committed them, a refused acknowledgment blocked every later one, and a client that disconnected mid-upload surfaced as a server error. The API server also accepted Dag ids, tags and file locations its tables cannot hold, let a bundle adopt another team's orphaned legacy Dag, and locked every active Dag in a bundle on each inventory. The default processor still imported the HTTP client for every parse-time request. The security model and configuration reference still described processors as needing database access and Job retirement as immediate.
ephraimbuddy
force-pushed
the
dag-processor-token-auth
branch
from
October 15, 2026 09:05
3884aff to
9438455
Compare
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by the seven-PR native GitHub stack. The replacement activation PR is #74794; checking out that branch includes all prerequisites and supports complete processor execution without metadata database access.
Merge order: #74788 → #74789 → #74790 → #74791 → #74793 → #74794 → #74796. Each PR contains its layer diff and dependency links.
The original branch and discussion remain available here. Original description follows.
Depends on #74370 for the credentials, Job lifecycle and client foundation. Review that PR first; this PR adds processor integration and publication.
This PR builds on and reuses substantial parts of Kaxil Naik's implementation in #67878.
Closes #67878.
Run the standalone Dag processor through the Execution API without giving it metadata database credentials or server keys. This is an opt-in mode of the normal
airflow dag-processorcommand; the existing database-backed mode remains the default.Provision the bundle catalog with
airflow dag-processor --sync-bundles-onlyand issue a separate session token for each processor on a trusted host. Set[dag_processor] execution_api_token_fileand[core] execution_api_server_urlon the processor.The processor uses HTTP for Job registration, heartbeats and completion, bundle metadata and secrets, parse-time requests, serialized results and source text, and callback and priority-request delivery. Short-lived parsing tokens are scoped to a Job, bundle, file and attempt. The API server does not read the processor's source files or construct its bundle providers.
Complete inventories and accepted results drive reconciliation. Bundle revisions reject superseded snapshots, and transactional receipts recover lost publication acknowledgments. The scheduler performs bounded cleanup for inactive or unassigned bundles, stale warnings and expired processor Jobs. Callback and priority requests remain claimed until acknowledged and become recoverable after their owner retires, including when switching back to direct mode. Callback effects can repeat after a crash before acknowledgment.
The migrations add Job identities, one publication checkpoint table, and nullable revision/claim fields on existing tables. Contracts are versioned and Task SDK models are regenerated. API mode has local liveness and API readiness probes, with Helm and Compose probe selection.
Draft status: keep this opt-in while deployment and performance/recovery validation continue. Enabling the setting does not remove shared database credentials, Fernet keys or signing material, and does not block database network access. Those deployment changes remain necessary for isolation. No newsfragment is included.
Was generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-6) following the guidelines