Repository navigation
Java SDK: Pack each native Dag's source file into the bundle JAR - #74096
Merged
Merged
Conversation
jason810496
force-pushed
the
jason/java-sdk/pack-dag-sources
branch
from
October 2, 2026 15:50
b6f5359 to
0ac76fd
Compare
This was referenced Oct 2, 2026
jason810496
force-pushed
the
jason/java-sdk/pack-dag-sources
branch
4 times, most recently
from
October 5, 2026 11:30
df353a7 to
546db86
Compare
1 task done
jason810496
force-pushed
the
jason/java-sdk/pack-dag-sources
branch
3 times, most recently
from
October 6, 2026 03:05
ecdfaf9 to
7c5001f
Compare
jason810496
marked this pull request as ready for review
October 6, 2026 03:27
jason810496
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jscheffl,
potiuk and
uranusjr
as code owners
October 6, 2026 03:27
jason810496
force-pushed
the
jason/java-sdk/pack-dag-sources
branch
4 times, most recently
from
October 7, 2026 07:44
5286d08 to
2f89e80
Compare
kaxil
reviewed
Oct 7, 2026
jason810496
requested review from
choo121600,
ephraimbuddy,
jedcunningham and
vatsrahul1001
as code owners
October 7, 2026 17:46
jason810496
force-pushed
the
jason/java-sdk/pack-dag-sources
branch
from
October 8, 2026 02:36
0cb996d to
5c11685
Compare
…request DagDef remembers the outermost class that built it, and a generated @Builder.Dag builder names the annotated class instead of itself. Server accepts --describe-sources <file> to write each Java-declared Dag's declaring class and return without connecting, for the Gradle plugin.
The packDagSources task runs mainClass in describe mode, finds each Dag's source file through the class file's SourceFile attribute, and packs each unique file with a sources.json mapping under META-INF/airflow/. The JAR manifest names it with Airflow-Java-SDK-Sources.
Every `Jar` task gained a dependency on `packDagSources`, so a sources or javadoc JAR carried the Dag sources and claimed the `Airflow-Java-SDK-Sources` attribute that the Python side picks a bundle JAR out of a directory by. Only `jar` and `shadowJar` carry them now. The describe run had no time bound, so a `main` that does not route through `Server.create(args)` and starts serving instead held the build open with no way out. It now runs under a timeout and falls back to packing the entrypoint's source alone, which is what every other describe failure already does. Say in `serve` and `serveAsync` that a `--describe-sources` server returns without connecting, and record in `sources.json` that its paths are relative to the source directory, which is also where they sit under `META-INF/airflow/sources/`.
…layout The Java importer now passes the Dag ID to the source lookup, so the Code view reads dag_source_paths instead of always showing the entrypoint. The source locator falls back to a unique match on the SourceFile name when the package does not match the directory, using an index built only when the package path misses. An unresolvable entrypoint or Dag class is now logged as a warning. The describe task is no longer up to date after a failed run, so a transient failure does not stick until a source changes. Its hint now points at Server.create(args), which is what answers --describe-sources. Add a shadowJar test, correct the comment on which JARs carry the sources payload, and document that a Dag maps to the class that constructed it.
jason810496
force-pushed
the
jason/java-sdk/pack-dag-sources
branch
from
October 8, 2026 03:58
5c11685 to
10a6954
Compare
uranusjr
reviewed
Oct 8, 2026
uranusjr
reviewed
Oct 8, 2026
uranusjr
reviewed
Oct 8, 2026
uranusjr
reviewed
Oct 8, 2026
uranusjr
reviewed
Oct 8, 2026
uranusjr
reviewed
Oct 8, 2026
uranusjr
reviewed
Oct 8, 2026
This was referenced Oct 8, 2026
jason810496
added this pull request to stack #74442
October 8, 2026 05:06
jason810496
commented
Oct 8, 2026
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Thanks for the review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Airflow can parse native Java Dags (#74036), and its Code view needs the Dag's source. A bundle JAR carried at most one hand-picked file, so a Dag declared in another class had no source of its own. This follows #73723 for TypeScript: pack each Dag's source file once, map every Dag to its file, and always keep the entrypoint as a fallback.
What changes
packDagSourcestask packs the source ofmainClassand of each class that declares a Dag in Java into the bundle JAR, and indexes them insources.json. Nothing to configure.mainClassonce at build time, under a timeout. If that run fails or does not finish, the build logs a warning and packs only themainClasssource.jarandshadowJar, carry the sources and theAirflow-Java-SDK-Sourcesattribute, so a sources or javadoc JAR is not mistaken for a bundle.@Builder.Dagclass maps to that class, not to the generated builder. Task handlers for a Python Dag are not listed..java,.ktand.scalasources all work.The bundle JAR now holds:
{ "entrypoint_path": "org/apache/airflow/example/ExampleBundleBuilder.java", "dag_source_paths": { "java_native_annotation_example": "org/apache/airflow/example/nativedag/AnnotationExample.java", "java_native_interface_example": "org/apache/airflow/example/nativedag/InterfaceExample.java" } }Was generative AI tooling used to co-author this PR?