Skip to content

Go SDK: embed each native Dag's source file in the bundle - #74342

Merged
jason810496 merged 11 commits into
apache:mainfrom
jason810496:feature/go-sdk/per-dag-source
Oct 8, 2026
Merged

jason810496 merged 11 commits into
apache:mainfrom
jason810496:feature/go-sdk/per-dag-source

Conversation

@jason810496

@jason810496 jason810496 commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Why

A Go bundle embedded only the func main file, so a native Dag declared in another file or package had no source of its own to show in the Code view. Also, Serve looked tasks up among the task handlers only, so the tasks of a native Go Dag could not run.

How

  • airflow.Dag records the file that calls it, and --airflow-metadata reports it per native Dag:

    // dags/reports.go
    func Reports() *airflow.DagRef {
        dag := airflow.Dag("reports") // recorded as dags/reports.go
        dag.Task(publish)
        return dag
    }
  • airflow-go-pack embeds the entrypoint and each Dag's file once, as paths relative to the module root. A Dag declared in a file outside the module maps to the entrypoint, with a warning.

    entrypoint_path: "example/bundle/main.go"
    dag_source_paths:
      orders: "example/bundle/main.go"
      reports: "example/bundle/dags/reports.go"
    sources:
      - {path: "example/bundle/main.go", offset: 0, length: 1532, sha256: "..."}
      - {path: "example/bundle/dags/reports.go", offset: 1532, length: 811, sha256: "..."}

What

  • airflow.Dag records its caller's file. Serve --airflow-metadata lists the files under dag_source_files; task handlers add nothing. The files come through a new DagSourceLister interface, separate from the task-handler EnumerableBundle.
  • airflow-go-pack:
    • writes entrypoint_path, dag_source_paths and sources in place of source, in a stable order, so identical inputs give a byte-identical bundle;
    • accepts absolute and -trimpath build paths;
    • packs a bundle that only has native Dags.
  • airflow-go-pack inspect --source prints each embedded file under its path.
  • Serve finds the tasks of Dags built with airflow.Dag as well as task handlers, so a task of a native Go Dag runs instead of being reported as not registered. go-sdk/internal/bundle/doc.go says so.
  • Docs: go-sdk/README.md, go.rst and the example Justfile.

Not in scope

  • The dags inventory of dag_ids and task_ids in the bundle metadata is unchanged, and a mixed-language task still finds its bundle through it. Dropping it, as ADR-0014 proposes, is left to a follow-up, which then only deletes EnumerableBundle and its use.
  • The packer still runs the binary with --airflow-metadata, now also to learn each Dag's source file.

Was generative AI tooling used to co-author this PR?

@jason810496 jason810496 self-assigned this Oct 7, 2026
@jason810496
jason810496 force-pushed the feature/go-sdk/per-dag-source branch 2 times, most recently from c559e14 to f52d769 Compare October 7, 2026 03:50
@jason810496
jason810496 marked this pull request as ready for review October 7, 2026 03:54
airflow.Dag records the file that called it, and --airflow-metadata
lists it per native Dag under dag_source_files so the packer can embed
the file. Task handlers add nothing.
The packer reads dag_source_files from the binary's manifest and embeds
the entrypoint plus the file of every native Dag once, in a stable
order. The manifest drops source and gains entrypoint_path,
dag_source_paths and a sources index. A Dag whose file is outside the
module maps to the entrypoint with a warning.
inspect --source cuts the source region by the manifest's sources index
and prints each file under its own header. An index that does not fit
the region is an error.
The Go SDK README, the Go language SDK page and the example Justfile now
say a bundle embeds the entrypoint plus the source file of each native
Dag.
Serve looked tasks up among the task handlers only, so every task of a native Go Dag
would be reported as not registered once Airflow runs native Go Dags.
@jason810496
jason810496 force-pushed the feature/go-sdk/per-dag-source branch from f52d769 to bce7e6b Compare October 7, 2026 09:18
Comment thread go-sdk/cmd/airflow-go-pack/sources.go Outdated
Comment thread go-sdk/cmd/airflow-go-pack/pack.go Outdated
Comment thread go-sdk/cmd/airflow-go-pack/pack.go
Comment thread go-sdk/cmd/airflow-go-pack/inspect.go
Comment thread go-sdk/cmd/airflow-go-pack/sources_test.go Outdated
Comment thread go-sdk/airflow/dag.go

@henry3260 henry3260 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to help unblock the technical side here, not a formal review.

The module root itself now counts as inside the module, so a main.go at the top of a plain go mod init project packs.
Dag IDs such as 2024, on or null were written unquoted and read back as int, bool or None by a YAML parser.
The summary showed dags=0 for a bundle with only airflow.Dag Dags, so it now reports task handler and native Dags separately.
The error now says to repack the bundle with this airflow-go-pack instead of only describing the mismatch.
The old test called rejectOutputAlias directly, so removing the call in runPack for per-Dag files kept the suite green.
Go records the file that calls airflow.Dag, so a Dag built in a factory maps to the factory's file.
@jason810496
jason810496 force-pushed the feature/go-sdk/per-dag-source branch from 5e59e03 to 0e71e05 Compare October 7, 2026 16:36

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread go-sdk/cmd/airflow-go-pack/sources.go Outdated
Comment thread go-sdk/cmd/airflow-go-pack/pack.go Outdated
Comment thread go-sdk/cmd/airflow-go-pack/pack.go
Comment thread go-sdk/cmd/airflow-go-pack/inspect.go
Comment thread go-sdk/cmd/airflow-go-pack/sources_test.go Outdated
Comment thread go-sdk/airflow/dag.go
@jason810496
jason810496 merged commit 1921597 into apache:main Oct 8, 2026
91 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants