Repository navigation
Parse native Go Dags and show each Dag's own source - #74341
Merged
jason810496 merged 10 commits intoOct 8, 2026
Merged
jason810496 merged 10 commits into
jason810496 merged 10 commits into
Conversation
1 task done
jason810496
force-pushed
the
feature/task-sdk/executable-dag-importer
branch
2 times, most recently
from
October 7, 2026 03:50
a3155b1 to
c0baa0e
Compare
jason810496
marked this pull request as ready for review
October 7, 2026 03:53
jason810496
requested review from
amoghrajesh,
ashb and
uranusjr
as code owners
October 7, 2026 03:53
jason810496
force-pushed
the
feature/task-sdk/executable-dag-importer
branch
from
October 7, 2026 09:22
c0baa0e to
ef89f5d
Compare
phanikumv
reviewed
Oct 7, 2026
kaxil
reviewed
Oct 7, 2026
henry3260
approved these changes
Oct 7, 2026
henry3260
left a comment
Contributor
There was a problem hiding this comment.
Just want to help unblock the technical side here, not a formal review.
1 task done
jason810496
force-pushed
the
feature/task-sdk/executable-dag-importer
branch
from
October 7, 2026 16:36
6a45233 to
fc67f86
Compare
jason810496
commented
Oct 7, 2026
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Thanks for the review.
The source region now holds the embedded files back to back, and the metadata indexes them through `sources`, `dag_source_paths` and `entrypoint_path`, replacing `source`. Add readers that verify each region and resolve a Dag to its own file, falling back to the entrypoint. Open and verify a bundle once so the metadata and source reads share the same file handle.
The importer claims any file that ends with the AFBNDL01 magic, so a bundle needs no extension, and shows each Dag's embedded source file in the Code view. ExecutableCoordinator now builds the parse command for such a file from its verified metadata and supervisor schema version.
Explain that the Dag processor parses every bundle binary once an ExecutableCoordinator is configured, that the Code view shows each Dag's own source file, and that handler-only binaries should be listed in .airflowignore until the Go runtime answers the Dag-parse request.
A bundle that fails verification now raises with the reason, such as a binary SHA-256 mismatch or an unknown footer version.
The bundle index is cached per file identity and only the requested source is read and hashed, so Dag count no longer multiplies the work.
A file that cannot be opened is not claimed by can_handle, so only a bundle that fails verification is kept.
Mirror the Java and TypeScript pages: with several ExecutableCoordinators, each Dag bundle needs a mapping.
The spec quoted a notice text that the UI does not show, so it now describes the behaviour.
The patch now checks the call signature of the function it replaces.
A task runs a native Dag file by its bundle-relative path, so the importer claims a file by its extensionless name and checks the bundle trailer only while listing a bundle.
jason810496
force-pushed
the
feature/task-sdk/executable-dag-importer
branch
from
October 8, 2026 02:39
fc67f86 to
98ee060
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md)Why
ExecutableCoordinatorhad no Dag importer, so Airflow could not parse native Go Dags, and an executable bundle embedded a single source file for all of its Dags.How
An executable bundle now embeds one source file per native Dag plus the entrypoint, back to back in the existing source region. The
AFBNDL01trailer is unchanged (footer_ver = 1), and the manifest indexes the files:ExecutableDagImporterclaims a file by itsAFBNDL01trailer, not its name, so a Go binary with no extension is parsed.get_source_code(definition, dag_id=None)follows Show each native Dag's own source in the Code tab #74249: a Dag's own file, the entrypoint for an unmapped Dag, and a notice when the bundle embeds no source.As in Parse native Java Dags and show their source #74036,
safe_modedoes not apply and every bundle is parsed, including bundles that only register task handlers.What
executable/_bundle_reader.py:read_bundle_source,read_bundle_entrypoint_sourceandread_bundle_language. Each region's bounds and sha256 are checked.ExecutableCoordinator: opens and verifies a bundle once for its metadata and sources. It runs[<bundle>]with the bundle's supervisor schema version both to parse the bundle and to run a task of a native Go Dag, so that task runs the binary its Dag was parsed from.ExecutableDagImporter, registered inCOORDINATOR_DAG_IMPORTERS.executable-bundle-spec.rstandairflow-metadata.schema.json:sourceis replaced byentrypoint_path,dag_source_pathsandsources.go.rst: how native Go Dags are parsed and run. A binary that only registers task handlers is parsed too and finds no Dags, so the Quick start shows how to keep task-handler binaries out of parsing with.airflowignore, asjava.rstdoes:Not in scope
dag_ids in the bundle metadata. Launching it without that inventory is left to a follow-up.Was generative AI tooling used to co-author this PR?