Skip to content

Cover native Go Dags in the go_sdk e2e test - #74411

Merged
jason810496 merged 4 commits into
apache:mainfrom
jason810496:feature/go-sdk/native-dag-e2e
Oct 9, 2026
Merged

jason810496 merged 4 commits into
apache:mainfrom
jason810496:feature/go-sdk/native-dag-e2e

Conversation

@jason810496

@jason810496 jason810496 commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Why

The go_sdk e2e mode only covers Python stub Dags with Go task handlers, so nothing checks that a Dag authored in Go is parsed, stored and run end to end.

What

  • go-sdk/example/native/: a native Go Dag example. go_native_pipeline uses TaskFlow Inputs with task results and Literals, nested task groups with a labelled group edge, If, Switch, and the Variable, Connection and XCom clients. go_native_trigger uses TriggerDagRun, and go_native_report is declared in report.go.

    extract := dag.TaskGroup("extract")
    regions := extract.TaskGroup("regions")
    north := regions.Task(countRows, airflow.TaskSpec{TaskID: "north"}, airflow.Inputs(airflow.Literal("north"), airflow.Literal(3)))
    summed := extract.Task(summarize, airflow.Inputs(north, south))
    dag.If(anyRows, airflow.Inputs(summed)).Then(loaded).Else(empty)
    dag.Switch(p.pick, airflow.Inputs(summed)).Case(p.daily).Case(p.weekly)
  • The go_sdk mode packs the native bundle into the Dags folder, so the Dag processor parses it with ExecutableCoordinator, and keeps the task handler bundle out of parsing with .airflowignore.

  • test_go_sdk_native_dag.py checks the parsed Dags and their edges, the run's task states (the untaken If and Switch sides are skipped), the XComs, the Variable and Connection reads, and the per-Dag source in the Code view.

  • go_native_trigger runs its TriggerDagRun task in the Go runtime (Go SDK: run TriggerDagRun tasks in the Go runtime #74417), and the test checks the triggered go_native_report run and its conf.


Was generative AI tooling used to co-author this PR?

@boring-cyborg boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:e2e-tests area:go-sdk area:task-sdk kind:documentation labels Oct 7, 2026
@jason810496
jason810496 force-pushed the feature/go-sdk/native-dag-e2e branch from 8d265ac to 00b3fa2 Compare October 7, 2026 16:04
@jason810496
jason810496 force-pushed the feature/go-sdk/native-dag-e2e branch 4 times, most recently from 6f52a66 to 727e256 Compare October 8, 2026 02:41
@jason810496
jason810496 marked this pull request as ready for review October 8, 2026 04:04
@jason810496 jason810496 self-assigned this Oct 8, 2026
@jason810496
jason810496 force-pushed the feature/go-sdk/native-dag-e2e branch from 33f3bcb to c0bc9a2 Compare October 8, 2026 09:57
@jason810496
jason810496 requested a review from phanikumv October 8, 2026 09:58
Shows Dags built with airflow.Dag, with task groups, If, Switch and TriggerDagRun, in a binary the Dag processor parses. The e2e test packs and runs it.
Pack the native example into the Dags folder so the Dag processor runs it to parse, and give that service the worker's USER and HOME. Keep the handler-only bundle out of parsing with an .airflowignore.
Parse, run and inspect the example native Dags through the stock stack. The TriggerDagRun case is a strict xfail until the Go runtime has a trigger runner.
@jason810496
jason810496 force-pushed the feature/go-sdk/native-dag-e2e branch from c0bc9a2 to bb10191 Compare October 9, 2026 08:35
@jason810496
jason810496 merged commit 785d82d into apache:main Oct 9, 2026
95 checks passed
@jason810496
jason810496 deleted the feature/go-sdk/native-dag-e2e branch October 9, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants