Skip to content

Java SDK: Trigger another Dag's run from a native Java Dag - #74440

Merged
jason810496 merged 12 commits into
jason/java-sdk/native-dag-branchfrom
jason/java-sdk/native-dag-trigger-dag-run
Oct 9, 2026
Merged

jason810496 merged 12 commits into
jason/java-sdk/native-dag-branchfrom
jason/java-sdk/native-dag-trigger-dag-run

Conversation

@jason810496

@jason810496 jason810496 commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Merge order:

  1. Java SDK: Pack each native Dag's source file into the bundle JAR #74096 — Pack each native Dag's source file into the bundle JAR
  2. Java SDK: Pick one of two tasks with a condition #74438 — Pick one of two tasks with a condition
  3. Java SDK: Choose one of several tasks with a branch #74439 — Choose one of several tasks with a branch
  4. Java SDK: Trigger another Dag's run from a native Java Dag #74440 — Trigger another Dag's run from a native Java Dag (current one)

Why

A native Java Dag could not start another Dag's run. TriggerDagRunOperator is Python, and a Java bundle has no Python Dag file to declare one in, so a Java Dag that fans out to another Dag needed a Python counterpart for that one task.

What changes

A TriggerDagRun passed to dag.task declares a task with no Java body; the Java runtime runs it as TriggerDagRunOperator does. Its settings are named as Python names the operator's parameters.

// Interface style.
dag.task("trigger_downstream",
    new TriggerDagRun("downstream_etl")
        .config("wait_for_completion", true)
        .config("poke_interval", Duration.ofSeconds(30))
        .config("conf", Map.of("rows", 2)));
// Annotation style: an ordinary @Builder.Task method that returns a TriggerDagRun.
// The serialized Dag has to carry what is triggered, so the method runs when the Dag
// is built rather than when the task runs, and therefore takes no parameters.
@Builder.Task(id = "trigger_downstream")
public TriggerDagRun triggerDownstream() {
  return new TriggerDagRun("downstream_etl").config("wait_for_completion", true);
}
  • The task serializes as Python's own TriggerDagRunOperator — its template fields, UI colour and extra link — plus a dag_dependencies entry, so the UI draws the dependency between the two Dags. Python writes only an operator's template fields and leaves the rest to the Dag file; a Java Dag has no such file, so every setting the author made is written too.
  • The runtime records the triggered run's link and ID as XComs, skips or fails on a run that already exists, and waits for completion either by polling or, with deferrable, by deferring to the Python triggerer's DagStateTrigger and resuming from the event it fires. This is the first Java runtime to emit DeferTask and a skipped TaskState.

One shared fixture line

scripts/ci/lang_sdk_serialization/test_dags.yaml now gives the shared trigger Dag a real logical date instead of a Jinja template, because the SDK takes one as a date-time. #74417 makes the same change for the Go SDK; whichever lands second drops that line, and until then the Go serialization conformance check reports that single difference.


Was generative AI tooling used to co-author this PR?

@jason810496
jason810496 added this pull request to stack #74441 October 8, 2026 05:01
@jason810496
jason810496 removed this pull request from stack #74441 October 8, 2026 05:06
@jason810496
jason810496 added this pull request to stack #74442 October 8, 2026 05:06
@jason810496
jason810496 force-pushed the jason/java-sdk/native-dag-trigger-dag-run branch from 3fe0637 to af569b2 Compare October 8, 2026 05:33
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread scripts/ci/lang_sdk_serialization/test_dags.yaml Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/execution/TriggerRunner.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/execution/Serde.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/execution/Serde.kt Outdated
A native Java Dag could not start another Dag's run. Airflow's
TriggerDagRunOperator is Python, and a Java bundle has no Python Dag file to
declare one in, so a Java Dag that fans out to another Dag needed a Python
counterpart for that one task.

A `TriggerDagRun` passed to `dag.task` declares a task that has no Java body
to run; the Java runtime runs it as `TriggerDagRunOperator` does. Its
settings are named as Python names the operator's parameters:

```java
dag.task("trigger_downstream",
    new TriggerDagRun("downstream_etl")
        .config("wait_for_completion", true)
        .config("conf", Map.of("rows", 2)));
```

An annotated Dag declares one from an ordinary `@Builder.Task` method that
returns a `TriggerDagRun`. The serialized Dag has to carry what is
triggered, so that method runs when the Dag is built rather than when the
task runs, and therefore takes no parameters.

The task serializes as Python's own `TriggerDagRunOperator`: its template
fields, its UI colour and extra link, and a `dag_dependencies` entry, so the
Airflow UI draws the dependency between the two Dags. Python writes only an
operator's template fields and keeps the rest in the Dag file; a Java Dag
has no such file, so every setting the author made is written too.

The runtime records the triggered run's link and ID as XComs, skips or fails
on a run that already exists, and waits for completion either by polling or,
with `deferrable`, by deferring to the Python triggerer's DagStateTrigger and
resuming from the event it fires. That makes this the first Java runtime to
emit DeferTask and a skipped TaskState.

`scripts/ci/lang_sdk_serialization/test_dags.yaml` now gives the shared
trigger Dag a real logical date instead of a Jinja template, because the SDK
takes one as a date-time. The Go SDK is making the same change in #74417;
whichever lands second drops that one line, and until then the Go
serialization conformance check reports that single difference.
@jason810496
jason810496 force-pushed the jason/java-sdk/native-dag-trigger-dag-run branch from af569b2 to 93f3da6 Compare October 8, 2026 11:46
@jason810496
jason810496 force-pushed the jason/java-sdk/native-dag-trigger-dag-run branch from 93f3da6 to c4a82f6 Compare October 8, 2026 14:46

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread scripts/ci/lang_sdk_serialization/test_dags.yaml Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/execution/TriggerRunner.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/execution/Serde.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/execution/Serde.kt Outdated

@guan404ming guan404ming left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@jason810496
jason810496 marked this pull request as ready for review October 9, 2026 06:30
@jason810496
jason810496 merged commit c5f5043 into main Oct 9, 2026
101 checks passed
@jason810496
jason810496 deleted the jason/java-sdk/native-dag-trigger-dag-run branch October 9, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants