Repository navigation
Go SDK: embed each native Dag's source file in the bundle - #74342
Merged
jason810496 merged 11 commits intoOct 8, 2026
Merged
Conversation
1 task done
jason810496
force-pushed
the
feature/go-sdk/per-dag-source
branch
2 times, most recently
from
October 7, 2026 03:50
c559e14 to
f52d769
Compare
jason810496
marked this pull request as ready for review
October 7, 2026 03:54
jason810496
requested review from
guan404ming,
henry3260,
pierrejeambrun and
uranusjr
October 7, 2026 03:54
1 task done
airflow.Dag records the file that called it, and --airflow-metadata lists it per native Dag under dag_source_files so the packer can embed the file. Task handlers add nothing.
The packer reads dag_source_files from the binary's manifest and embeds the entrypoint plus the file of every native Dag once, in a stable order. The manifest drops source and gains entrypoint_path, dag_source_paths and a sources index. A Dag whose file is outside the module maps to the entrypoint with a warning.
inspect --source cuts the source region by the manifest's sources index and prints each file under its own header. An index that does not fit the region is an error.
The Go SDK README, the Go language SDK page and the example Justfile now say a bundle embeds the entrypoint plus the source file of each native Dag.
Serve looked tasks up among the task handlers only, so every task of a native Go Dag would be reported as not registered once Airflow runs native Go Dags.
jason810496
force-pushed
the
feature/go-sdk/per-dag-source
branch
from
October 7, 2026 09:18
f52d769 to
bce7e6b
Compare
kaxil
reviewed
Oct 7, 2026
henry3260
approved these changes
Oct 7, 2026
henry3260
left a comment
Contributor
There was a problem hiding this comment.
Just want to help unblock the technical side here, not a formal review.
This was referenced Oct 7, 2026
The module root itself now counts as inside the module, so a main.go at the top of a plain go mod init project packs.
Dag IDs such as 2024, on or null were written unquoted and read back as int, bool or None by a YAML parser.
The summary showed dags=0 for a bundle with only airflow.Dag Dags, so it now reports task handler and native Dags separately.
The error now says to repack the bundle with this airflow-go-pack instead of only describing the mismatch.
The old test called rejectOutputAlias directly, so removing the call in runPack for per-Dag files kept the suite green.
Go records the file that calls airflow.Dag, so a Dag built in a factory maps to the factory's file.
jason810496
force-pushed
the
feature/go-sdk/per-dag-source
branch
from
October 7, 2026 16:36
5e59e03 to
0e71e05
Compare
jason810496
commented
Oct 7, 2026
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Thanks for the review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md)executable-bundle-spec.rstand reads it.Why
A Go bundle embedded only the
func mainfile, so a native Dag declared in another file or package had no source of its own to show in the Code view. Also,Servelooked tasks up among the task handlers only, so the tasks of a native Go Dag could not run.How
airflow.Dagrecords the file that calls it, and--airflow-metadatareports it per native Dag:airflow-go-packembeds the entrypoint and each Dag's file once, as paths relative to the module root. A Dag declared in a file outside the module maps to the entrypoint, with a warning.What
airflow.Dagrecords its caller's file.Serve --airflow-metadatalists the files underdag_source_files; task handlers add nothing. The files come through a newDagSourceListerinterface, separate from the task-handlerEnumerableBundle.airflow-go-pack:entrypoint_path,dag_source_pathsandsourcesin place ofsource, in a stable order, so identical inputs give a byte-identical bundle;-trimpathbuild paths;airflow-go-pack inspect --sourceprints each embedded file under its path.Servefinds the tasks of Dags built withairflow.Dagas well as task handlers, so a task of a native Go Dag runs instead of being reported as not registered.go-sdk/internal/bundle/doc.gosays so.go-sdk/README.md,go.rstand the exampleJustfile.Not in scope
dagsinventory ofdag_ids andtask_ids in the bundle metadata is unchanged, and a mixed-language task still finds its bundle through it. Dropping it, as ADR-0014 proposes, is left to a follow-up, which then only deletesEnumerableBundleand its use.--airflow-metadata, now also to learn each Dag's source file.Was generative AI tooling used to co-author this PR?