Skip to content

Parse native Go Dags and show each Dag's own source - #74341

Merged
jason810496 merged 10 commits into
apache:mainfrom
jason810496:feature/task-sdk/executable-dag-importer
Oct 8, 2026
Merged

jason810496 merged 10 commits into
apache:mainfrom
jason810496:feature/task-sdk/executable-dag-importer

Conversation

@jason810496

@jason810496 jason810496 commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Why

ExecutableCoordinator had no Dag importer, so Airflow could not parse native Go Dags, and an executable bundle embedded a single source file for all of its Dags.

How

  • An executable bundle now embeds one source file per native Dag plus the entrypoint, back to back in the existing source region. The AFBNDL01 trailer is unchanged (footer_ver = 1), and the manifest indexes the files:

    entrypoint_path: "example/bundle/main.go"
    dag_source_paths:
      orders: "example/bundle/main.go"
      reports: "example/bundle/dags/reports.go"
    sources:
      - {path: "example/bundle/main.go", offset: 0, length: 1532, sha256: "..."}
      - {path: "example/bundle/dags/reports.go", offset: 1532, length: 811, sha256: "..."}
  • ExecutableDagImporter claims a file by its AFBNDL01 trailer, not its name, so a Go binary with no extension is parsed.

  • get_source_code(definition, dag_id=None) follows Show each native Dag's own source in the Code tab #74249: a Dag's own file, the entrypoint for an unmapped Dag, and a notice when the bundle embeds no source.

  • As in Parse native Java Dags and show their source #74036, safe_mode does not apply and every bundle is parsed, including bundles that only register task handlers.

What

  • executable/_bundle_reader.py: read_bundle_source, read_bundle_entrypoint_source and read_bundle_language. Each region's bounds and sha256 are checked.

  • ExecutableCoordinator: opens and verifies a bundle once for its metadata and sources. It runs [<bundle>] with the bundle's supervisor schema version both to parse the bundle and to run a task of a native Go Dag, so that task runs the binary its Dag was parsed from.

  • ExecutableDagImporter, registered in COORDINATOR_DAG_IMPORTERS.

  • executable-bundle-spec.rst and airflow-metadata.schema.json: source is replaced by entrypoint_path, dag_source_paths and sources.

  • go.rst: how native Go Dags are parsed and run. A binary that only registers task handlers is parsed too and finds no Dags, so the Quick start shows how to keep task-handler binaries out of parsing with .airflowignore, as java.rst does:

    # task_handler_bundle_name names a bundle that holds only handler binaries
    echo '*' > /opt/airflow/go-task-handlers/.airflowignore   # glob, the default
    echo '.' > /opt/airflow/go-task-handlers/.airflowignore   # regexp
    
    # handler binaries in the Dag bundle itself, kept in bin/
    echo 'bin/*' >> /opt/airflow/dags/.airflowignore           # glob
    echo '^bin/' >> /opt/airflow/dags/.airflowignore           # regexp

Not in scope

  • A mixed-language task still finds its bundle through the dag_ids in the bundle metadata. Launching it without that inventory is left to a follow-up.

Was generative AI tooling used to co-author this PR?

@boring-cyborg boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk kind:documentation labels Oct 6, 2026
@jason810496 jason810496 self-assigned this Oct 7, 2026
@jason810496
jason810496 force-pushed the feature/task-sdk/executable-dag-importer branch 2 times, most recently from a3155b1 to c0baa0e Compare October 7, 2026 03:50
@jason810496
jason810496 marked this pull request as ready for review October 7, 2026 03:53
@jason810496
jason810496 force-pushed the feature/task-sdk/executable-dag-importer branch from c0baa0e to ef89f5d Compare October 7, 2026 09:22
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/_dag_importer.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/_dag_importer.py
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/_dag_importer.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst
Comment thread task-sdk/docs/executable-bundle-spec.rst Outdated
Comment thread task-sdk/tests/task_sdk/coordinators/executable/test_coordinator.py

@henry3260 henry3260 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to help unblock the technical side here, not a formal review.

@jason810496
jason810496 force-pushed the feature/task-sdk/executable-dag-importer branch from 6a45233 to fc67f86 Compare October 7, 2026 16:36

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/_dag_importer.py
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst
Comment thread task-sdk/docs/executable-bundle-spec.rst Outdated
Comment thread task-sdk/tests/task_sdk/coordinators/executable/test_coordinator.py
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/_dag_importer.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/_dag_importer.py Outdated
The source region now holds the embedded files back to back, and the
metadata indexes them through `sources`, `dag_source_paths` and
`entrypoint_path`, replacing `source`. Add readers that verify each
region and resolve a Dag to its own file, falling back to the
entrypoint. Open and verify a bundle once so the metadata and source
reads share the same file handle.
The importer claims any file that ends with the AFBNDL01 magic, so a
bundle needs no extension, and shows each Dag's embedded source file in
the Code view. ExecutableCoordinator now builds the parse command for
such a file from its verified metadata and supervisor schema version.
Explain that the Dag processor parses every bundle binary once an
ExecutableCoordinator is configured, that the Code view shows each Dag's
own source file, and that handler-only binaries should be listed in
.airflowignore until the Go runtime answers the Dag-parse request.
A bundle that fails verification now raises with the reason, such as a binary SHA-256 mismatch or an unknown footer version.
The bundle index is cached per file identity and only the requested source is read and hashed, so Dag count no longer multiplies the work.
A file that cannot be opened is not claimed by can_handle, so only a bundle that fails verification is kept.
Mirror the Java and TypeScript pages: with several ExecutableCoordinators, each Dag bundle needs a mapping.
The spec quoted a notice text that the UI does not show, so it now describes the behaviour.
The patch now checks the call signature of the function it replaces.
A task runs a native Dag file by its bundle-relative path, so the importer claims a file by its extensionless name and checks the bundle trailer only while listing a bundle.
@jason810496
jason810496 force-pushed the feature/task-sdk/executable-dag-importer branch from fc67f86 to 98ee060 Compare October 8, 2026 02:39
@jason810496
jason810496 merged commit 46d3c59 into apache:main Oct 8, 2026
108 checks passed
@jason810496
jason810496 deleted the feature/task-sdk/executable-dag-importer branch October 8, 2026 04:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants