Skip to content

Go SDK: answer the Dag-parsing request from Serve - #74391

Merged
jason810496 merged 6 commits into
apache:mainfrom
FrankYang0529:airflow-go-sdk-answer-dag-parse-request
Oct 7, 2026
Merged

jason810496 merged 6 commits into
apache:mainfrom
FrankYang0529:airflow-go-sdk-answer-dag-parse-request

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Implement Go SDK: Answer the Dag-parsing request from Serve.

  • Serve answers a DagFileParseRequest with one DagFileParsingResult. The result holds the Dags declared with airflow.Dag that the bundle registered, in registration order. The result leaves out the Dag of a task handler, because that Dag comes from a Python Dag file that is parsed separately.
  • If a Dag fails to serialize, the result gets an import error that names that Dag.

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Signed-off-by: PoAn Yang <payang@apache.org>
The stderr excerpt is unrelated to Go Dag parsing, so this PR leaves it out.
Serve takes a bundle.Bundle again, and the Dag serializer is an optional interface the bundle can implement.
The runtime replies on the request id and exits without waiting for an acknowledgement, as the TS runtime and ADR 0003 do.
The panic becomes the Dag's error, and the stack now reaches the Dag processor log through the default slog logger.
The comments now state the behavior in one or two lines.
@jason810496
jason810496 force-pushed the airflow-go-sdk-answer-dag-parse-request branch from 52eb151 to e84f9b3 Compare October 7, 2026 08:02
@jason810496
jason810496 marked this pull request as ready for review October 7, 2026 08:02
@jason810496
jason810496 merged commit a2cf328 into apache:main Oct 7, 2026
90 checks passed
@jason810496

Copy link
Copy Markdown
Member

Pushed a few follow-up commits on top:

  • Dropped the task-sdk and core stderr-excerpt changes, so this PR is Go SDK only. They can go in a separate PR if needed.
  • Removed bundle.Registry. execution.Serve takes bundle.Bundle again and checks for bundle.DagSerializer in the parse branch. The adapter is now coordinatorSource{*taskHandlerMap; dags *dagMap}, the same shape as manifestSource in Go SDK: embed each native Dag's source file in the bundle #74342.
  • The parse result now replies on the request id and exits, the same as the TS runtime and ADR 0003. The ack wait and its test are removed.
  • parseDags returns an empty result when the bundle has no Dags, and logs Received Dag parse request / Parse-mode response at info, the same as TS.
  • serializeRecovering now logs the panic stack.
  • Trimmed the comments added in this PR.
  • Native Dag task lookup in LookupTask will be fixed in Go SDK: embed each native Dag's source file in the bundle #74342.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants