Repository navigation
Java SDK: answer TaskHandlerParseRequest with registered handlers - #74317
Draft
jason810496 wants to merge 7 commits into
Conversation
The vendored schema.json was refreshed only when airflowSupervisorSchemaVersion changed, by downloading the published file. A version is published only at release (2026-10-30 returns 404), so messages added during a release cycle, such as TaskHandlerParseRequest, never reach the Java models. The Gradle task :sdk:syncSupervisorSchema now copies the Task SDK snapshot when it declares the configured version, and downloads the published file as before otherwise. The copy runs only when asked: java-sdk/gradlew -p java-sdk :sdk:syncSupervisorSchema, then commit the file. Code generation reads only the committed schema.json, guarded by a read-only :sdk:checkSupervisorSchemaVersion that fails when the file is not at the configured version, so a later change to the snapshot never rewrites it from a build step or the ktlint hook. The prek hook stays a version-only check and runs the task only when the configured version changes. The refreshed file adds the task handler messages and main's changes since the last vendoring (DagSourceCode, UpdateDagRunNote, retry_reason, multi_team, more required fields); the Java SDK builds and its tests pass on it unchanged.
The Dag processor will ask a Java bundle how each task handler binds its arguments, but Java parameter names do not survive compilation and the generated task class reads its arguments by position only. The annotation processor now gives every class it generates for a @Builder.TaskHandler method a static AIRFLOW_TASK_PARAMS field: each flat data parameter's name and declared type (a TypeRef when it has type arguments, primitives kept primitive), or the TaskInput type. @Builder.Dag task classes are unchanged, because the tasks of a Dag declared in Java are never task handlers.
The Dag processor checks each stub task argument against the JSON Schema of the handler param it binds to. buildValueSchema maps a declared Java type to the values it accepts, in the vocabulary @task.stub emits for Python annotations: only primitives exclude null, collections, arrays and maps describe their items, a POJO is an object, and types that decode from more than one JSON shape (byte[], Object, Optional, Date) say nothing. Enum constants come sorted, because Class.getFields has no defined order and the same JAR should answer the same on every JVM. It describes the declared type, not every value the decoder coerces, as pydantic does for its lax mode. Nothing calls it yet.
The Dag processor asks the runtime a stub task would run which task handlers it registers, and a Java bundle fails on that first frame as an unexpected message. Server now replies with one TaskHandlerParsingResult listing every handler registered with @Builder.TaskHandler or Bundle.register(dagId, taskId, cls), keyed by Dag id in registration order, so the answer depends only on the JAR. The tasks of a Dag declared in Java are never task handlers and are left out, and Bundle.register(dagId, taskId, cls) now rejects a class the annotation processor generated for a @Builder.Dag task, which carries a GeneratedDagTask marker, telling the author to declare a task handler with @Builder.TaskHandler. Flat data parameters bind positional with their names; a TaskInput, as the only data parameter or through InputTask, binds named, with @ArgName fields matched exactly; a task that reads no argument binds named with no params, so the Dag processor only warns about an argument passed to it. The reply is a plain map, because the generated models cannot hold task_handlers, and the SDK's tests check its keys against the vendored supervisor schema. The runtime waits for the parent's acknowledgement before it exits.
The example, the e2e test bundle, the Scala Spark example and the k8s example registered the Java tasks of Dags that a Python file owns as a Dag declared in Java. The task handler parse reports only task handler registrations, so the Dag processor would find no handler for those stub tasks, and once the Java SDK answers native Dag parsing each such Dag would be a second definition of the Python Dag. They now register those tasks as task handlers. Execution already falls back to task handlers, so runs do not change.
The probe from #73974 needs each coordinator to find the artifact a stub task runs and to start its runtime. JavaCoordinator finds the JAR with the scan that execute_task uses, main_class included, and returns the resolved path of the JAR that sets the Main-Class a task would run. The scan can stop on another JAR, such as the airflow-sdk JAR that gives a thin JAR its schema version, so _JarInfo now records that path. The probe runs a task's own command, so it answers for the classpath and Main-Class a worker would run.
The coordinator and SDK tests each fake one side, so nothing showed that a JAR built with the Gradle plugin answers the way the Dag processor parses it. The test builds java-sdk/example as a thin bundle from a copy of the SDK sources, finds it as a stub task of java_annotation_example would, and probes it. The airflow-sdk JAR sets the schema version and the example's JAR the Main-Class, so the found JAR is the latter. It needs a JDK, so it runs only with AIRFLOW_LANG_SDK_REAL_PROBE_TESTS=1.
jason810496
added this pull request to stack #74318
October 6, 2026 02:35
This was referenced Oct 6, 2026
jason810496
force-pushed
the
jason/core-taskhandler-refactor/08-java-sdk-task-handler-parse
branch
from
October 6, 2026 02:44
94a8e22 to
9037ea2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom to top), on native #74037 (stack #74170): #73973, #73974, #73975, #74317, #73976, #74067, #74135, #74140, #74141, #73971, #74030, #74031, #74032, #74136, #74137, #74138, #74139
A Java bundle now answers #73974's probe with every handler registered through
@Builder.TaskHandlerorBundle.register(dagId, taskId, cls)and how its params bind, andJavaCoordinatorcan find and start the JAR a stub task would run. Until now a Java bundle failed on aTaskHandlerParseRequestas an unexpected first frame. Nothing calls the probe yet: #74135 does.answers:
Bundle.register(dagId, taskId, cls)now rejects a class generated for a@Builder.Dagtask and points to@Builder.TaskHandler.positionalwith their Java names. ATaskInputisnamed, with@ArgNamefields matching exactly. A handler that reads no argument isnamedwith empty params.-parameters, so the annotation processor gives each generated@Builder.TaskHandlerclass anAIRFLOW_TASK_PARAMSfield with its params' names and declared types.value_schemadescribes the declared Java type in the vocabularybuild_arg_bindingsemits; only primitives exclude null. It does not widen for the decoder's lenient coercions, such as"5"into along.task_handlers, and a test checks it against the vendored schema. The runtime waits for the parent to acknowledge it before exiting.:sdk:syncSupervisorSchemanow copies the Task SDK snapshot when it declares the configured version. It runs only when asked and the build only checks the version, so a later Task SDK schema change neither rewrites the file nor failsktlint. This refresh also brings in main's other schema changes since the last vendoring._find_task_handler_artifactis the worker's own scan (main_classincluded) and returns the JAR that sets the Main-Class, and_build_parse_task_handler_commandis a task's own command.Compatibility: a JAR built from main's Java SDK before this change reports
2026-10-30but cannot answer, so its probe fails and #74135 logs a warning. Rebuilding fixes it.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines