Skip to content

Java SDK: Pack each native Dag's source file into the bundle JAR - #74096

Merged
jason810496 merged 10 commits into
mainfrom
jason/java-sdk/pack-dag-sources
Oct 9, 2026
Merged

jason810496 merged 10 commits into
mainfrom
jason/java-sdk/pack-dag-sources

Conversation

@jason810496

@jason810496 jason810496 commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Merge order:

  1. Java SDK: Group a native Dag's tasks with task groups #74230 — Group a native Dag's tasks with task groups
  2. Java SDK: Serialize native Dags to DagSerialization v3 #71190 — Serialize native Dags to DagSerialization v3
  3. Java SDK: Pack each native Dag's source file into the bundle JAR #74096 — Pack each native Dag's source file into the bundle JAR (current one)

Why

Airflow can parse native Java Dags (#74036), and its Code view needs the Dag's source. A bundle JAR carried at most one hand-picked file, so a Dag declared in another class had no source of its own. This follows #73723 for TypeScript: pack each Dag's source file once, map every Dag to its file, and always keep the entrypoint as a fallback.

What changes

  • The Gradle plugin's new packDagSources task packs the source of mainClass and of each class that declares a Dag in Java into the bundle JAR, and indexes them in sources.json. Nothing to configure.
  • To learn which class declares each Dag, the task runs mainClass once at build time, under a timeout. If that run fails or does not finish, the build logs a warning and packs only the mainClass source.
  • Only the JARs a bundle is assembled from, jar and shadowJar, carry the sources and the Airflow-Java-SDK-Sources attribute, so a sources or javadoc JAR is not mistaken for a bundle.
  • A Dag built by a @Builder.Dag class maps to that class, not to the generated builder. Task handlers for a Python Dag are not listed.
  • .java, .kt and .scala sources all work.

The bundle JAR now holds:

META-INF/MANIFEST.MF
  Main-Class: org.apache.airflow.example.ExampleBundleBuilder
  Airflow-Java-SDK-Sources: META-INF/airflow/sources.json
META-INF/airflow/sources.json
META-INF/airflow/sources/org/apache/airflow/example/ExampleBundleBuilder.java
META-INF/airflow/sources/org/apache/airflow/example/nativedag/AnnotationExample.java
META-INF/airflow/sources/org/apache/airflow/example/nativedag/InterfaceExample.java
{
  "entrypoint_path": "org/apache/airflow/example/ExampleBundleBuilder.java",
  "dag_source_paths": {
    "java_native_annotation_example": "org/apache/airflow/example/nativedag/AnnotationExample.java",
    "java_native_interface_example": "org/apache/airflow/example/nativedag/InterfaceExample.java"
  }
}

Was generative AI tooling used to co-author this PR?

Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/PackDagSources.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/PackDagSources.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
…request

DagDef remembers the outermost class that built it, and a generated
@Builder.Dag builder names the annotated class instead of itself.
Server accepts --describe-sources <file> to write each Java-declared Dag's
declaring class and return without connecting, for the Gradle plugin.
The packDagSources task runs mainClass in describe mode, finds each Dag's
source file through the class file's SourceFile attribute, and packs each
unique file with a sources.json mapping under META-INF/airflow/. The JAR
manifest names it with Airflow-Java-SDK-Sources.
Every `Jar` task gained a dependency on `packDagSources`, so a sources or
javadoc JAR carried the Dag sources and claimed the
`Airflow-Java-SDK-Sources` attribute that the Python side picks a bundle
JAR out of a directory by. Only `jar` and `shadowJar` carry them now.

The describe run had no time bound, so a `main` that does not route
through `Server.create(args)` and starts serving instead held the build
open with no way out. It now runs under a timeout and falls back to
packing the entrypoint's source alone, which is what every other
describe failure already does.

Say in `serve` and `serveAsync` that a `--describe-sources` server
returns without connecting, and record in `sources.json` that its paths
are relative to the source directory, which is also where they sit under
`META-INF/airflow/sources/`.
…layout

The Java importer now passes the Dag ID to the source lookup, so the Code
view reads dag_source_paths instead of always showing the entrypoint.

The source locator falls back to a unique match on the SourceFile name
when the package does not match the directory, using an index built only
when the package path misses. An unresolvable entrypoint or Dag class is
now logged as a warning.

The describe task is no longer up to date after a failed run, so a
transient failure does not stick until a source changes. Its hint now
points at Server.create(args), which is what answers --describe-sources.

Add a shadowJar test, correct the comment on which JARs carry the sources
payload, and document that a Dag maps to the class that constructed it.
@jason810496
jason810496 force-pushed the jason/java-sdk/pack-dag-sources branch from 5c11685 to 10a6954 Compare October 8, 2026 03:58
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/PackDagSources.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Server.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/PackDagSources.kt Outdated

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/PackDagSources.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Server.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/AirflowSdkPlugin.kt Outdated
Comment thread java-sdk/plugin/src/main/kotlin/org/apache/airflow/sdk/plugin/PackDagSources.kt Outdated
@jason810496
jason810496 merged commit 2bda192 into main Oct 9, 2026
112 checks passed
@jason810496
jason810496 deleted the jason/java-sdk/pack-dag-sources branch October 9, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants