Repository navigation
Cover native TypeScript and Java Dags in the compose e2e - #73847
Draft
jason810496 wants to merge 28 commits into
Draft
jason810496 wants to merge 28 commits into
jason810496 wants to merge 28 commits into
Conversation
jason810496
added this pull request to stack #73848
September 28, 2026 13:20
This was referenced Sep 28, 2026
jason810496
changed the title
jason810496
force-pushed
the
jason/lang-sdk-e2e/07-java-sdk
branch
from
September 28, 2026 15:57
a943bc2 to
daaccdf
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/08-native-e2e
branch
2 times, most recently
from
September 29, 2026 01:25
f5e5047 to
ef6f31f
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/07-java-sdk
branch
from
September 29, 2026 01:26
daaccdf to
2547205
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/08-native-e2e
branch
from
September 29, 2026 01:29
ef6f31f to
92d2b90
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/07-java-sdk
branch
from
September 29, 2026 01:29
2547205 to
39c93a6
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/08-native-e2e
branch
from
September 29, 2026 13:18
92d2b90 to
dedaa30
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/07-java-sdk
branch
from
September 29, 2026 13:18
39c93a6 to
196ed3a
Compare
jason810496
removed this pull request from stack #73848
October 1, 2026 03:02
jason810496
force-pushed
the
jason/lang-sdk-e2e/08-native-e2e
branch
from
October 1, 2026 03:03
dedaa30 to
1468d5e
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/07-java-sdk
branch
from
October 1, 2026 03:03
196ed3a to
3d6e6a5
Compare
1 task done
jason810496
added this pull request to stack #74005
October 1, 2026 03:04
1 task done
jason810496
force-pushed
the
jason/lang-sdk-e2e/07-java-sdk
branch
from
October 1, 2026 06:20
3d6e6a5 to
e2459c7
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/08-native-e2e
branch
from
October 1, 2026 06:20
1468d5e to
00fac11
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/07-java-sdk
branch
from
October 8, 2026 06:34
cba80f3 to
01dee95
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/08-native-e2e
branch
from
October 8, 2026 06:34
de018a0 to
c76d1de
Compare
Put the example bundle in the Dag bundle as well, add a root-less "ts-native" Node coordinator that claims it there, and give the Dag processor node. The existing "ts" coordinator still runs every TypeScript task, so the native Dag test needs no gate anymore.
The Code view must show the bundle's entry module, and trigger_downstream, run by the TypeScript runtime, must trigger typescript_example with its conf and link to that run.
The JVM takes user.home from /etc/passwd, so when the host UID is one the JDK image defines (1000 is its "ubuntu" user), publishToMavenLocal wrote into the container and the example builds could not find the SDK. Name the mounted repository with maven.repo.local instead of relying on HOME.
A dedicated bundle declares two Dags in Java, one with the interface API and one with annotations, and routes every task to a "java-native" queue. Its JAR goes into the Dag bundle, where a root-less Java coordinator parses it in the Dag processor and runs its tasks. The test checks the edges, the queues, the run, the XComs and the Dag source the JAR embeds.
The Java and TypeScript e2e suites now parse native Dags through the Dag processor, the Dag bag and the coordinators' Dag importers, so changes to those sources select both jobs. The Java native-Dag bundle selects the Java job.
wait_for_dag_run now polls through get_dag_run, and the annotation Dag takes its queue from the interface Dag's constant.
setup-lang-sdk-test packed the TypeScript bundle and built its worker image for every lang-SDK run, so the Go and Java k8s job, which provisions no Node toolchain, failed during setup. Build and upload them only with --ts-sdk-native-dag-test (RUN_TS_SDK_NATIVE_DAG_K8S_TESTS), the variable the native TypeScript k8s test is gated on.
Drop the shared Dag processing files (dagbag, manager, processor) from the trigger list, since most of their changes are off the Lang-SDK path and the canary run still covers them. Add serialized_objects.py, which holds the checks every native Dag goes through.
The example's airflow-ts-pack bin points at the SDK's dist/, so on a clean checkout pnpm skipped the bin link and the pack failed. Follow the compose e2e sequence on both the host and container paths, and let the container user enable corepack under a writable HOME. The docstring no longer claims CI provisions Node for this test.
The example inherited the root tsconfig's paths alias, so esbuild bundled the SDK from ../src instead of the shipped dist/, and the e2e tests stopped covering the published build. Clearing paths in the example also fixes its TS6059 typecheck errors.
The Dag processor already dispatches parse requests, so name the real gaps instead: the bundle reaches only worker pods, the ts-sdk coordinator is limited to that bundle, and the Dag processor image has no Node. Also fix the gating env var the README named.
Say the Dag processor needs node to parse the native Dag, link to the parsing docs, and show how to trigger it.
The test asserts the skipmixin XCom, not a clear, so the old name read as if the clear path were covered.
Java: java-native loses dag_bundle_name and only parses, picked for the Dags folder by [sdk] dag_bundle_to_coordinator among the four JavaCoordinators. Its queue routes to java-jdk, so native tasks run on the coordinator the queue names. TypeScript: ts is the only NodeCoordinator, so it parses the native Dag without the ts-native coordinator. The artifact bundles keep their .airflowignore so they are not parsed as native Dag files.
dag_bundle_name no longer limits which Dag bundle is parsed. The other two reasons still apply.
The java-native queue routes to the java-jdk coordinator, and the java-native coordinator only parses the JAR.
The test called listBundleNativeDags, which main's TypeScript SDK does not have. bundleDags returns the same Dags, keyed by dag_id.
Adding ts-task-handlers to the comment that lists the task handler Dag bundles made its first line 114 characters long, over the 110 that yamllint allows.
Renames the combined Go + Java system test to mixed-language and adds TypeScript to it, adds one native Dag test per language SDK (Java and TypeScript; Go is skipped, native Dag parsing has no Go support yet), and drops the TypeScript-only gate: every lang-SDK artifact now builds unconditionally. Every coordinator queue now shares one runtime image (prod + JRE + Node) and one worker pod template. A coordinator downloads its own task_handler_bundle_name bundle directly from S3, the same way any Python Dag bundle works, so the init container and stage_artifacts.py staging step are gone -- restoring the Go binary's execute bit after an S3 download is now ExecutableCoordinator's own job (see the executable-bundle-exec-bit PR this depends on). Every task-handler bucket carries an .airflowignore so the Dag processor never tries to parse a packed artifact as a Dag; the two native buckets carry none, since parsing them is the point. With one coordinator per language, no [sdk] dag_bundle_to_coordinator entry is needed. Also updates the k8s CI job to provision Node, and the breeze docs and unit tests to match. Co-Authored-By: Claude <noreply@anthropic.com>
It's a plain .airflowignore-style pattern file, not source, same as the existing .airflowignore entry. Co-Authored-By: Claude <noreply@anthropic.com>
…ag e2e Both java_native_e2e (interface API) and java_native_annotation_e2e (annotations) now also wire a task group, an If, a Switch and a trigger_downstream task onto their extract -> transform -> load chain, against a new third Dag, java_native_target_e2e, that the trigger starts. The Code view test now expects each Dag's own declaring source file rather than always the bundle's main class, since the bundle embeds one source per Dag. Co-Authored-By: Claude <noreply@anthropic.com>
TestNativeJavaDagOnKubernetes now follows java_native_e2e's task group, If, Switch and trigger_downstream, un-pausing java_native_target_e2e first since trigger_downstream waits for the run it starts. Co-Authored-By: Claude <noreply@anthropic.com>
dag.triggerDagRun(spec) is not a method; a trigger is a task built by the standalone triggerDagRun(spec) and placed with dag.task(taskId, trigger). dag.if and dag.switch likewise take the deciding handler itself and place it, not an already-placed TaskRef, so has_rows and pick_cadence now pass their handler straight to dag.if/dag.switch with an explicit taskId. Also use datetime.UTC instead of the deprecated timezone.utc alias. Co-Authored-By: Claude <noreply@anthropic.com>
Rebasing onto a main that had independently grown set_variable, get_tasks, get_dag_source and get_dag_run let both copies through as a clean, silent merge. Keep one of each. Co-Authored-By: Claude <noreply@anthropic.com>
The bundle now embeds one source file per Dag, so the Code view shows native.ts, the file that actually declares typescript_native_example, not main.ts's Bundle registration. Co-Authored-By: Claude <noreply@anthropic.com>
trigger_downstream pushes the triggered run's ID under the trigger_run_id key, not return_value, matching the TS and Go runtimes. Co-Authored-By: Claude <noreply@anthropic.com>
tsconfig.json's path alias only covered the bare package import. Including example/src in typecheck (for the native Dag work) now also compiles main's own HITL example files, which import apache-airflow-ts-sdk/hitl; add that and the /coordinator subpath to both tsconfig.json and vitest.config.ts so the compiler and the runner keep agreeing. Also follow TaskBody's trigger -> operator rename in the native Dag example test. Co-Authored-By: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom to top): #74042, #74035, #74043, #74036, #74037, #73845, #73846, #73847
jason/lang-sdk-e2e/07-java-sdk. Only the commits after it are new here.Why
The layers below parse a native Lang-SDK Dag with its coordinator's runtime. This covers that path end to end in the compose e2e for TypeScript and Java: the Dag processor parses each Dag, and its tasks run on the coordinator their queue routes to.
How
ts-sdk/example/src/native.ts), its vitest, the compose test, and the k8s provisioning.triggerDagRunruns in TypeScript, defers toDagStateTriggerin the Python triggerer, and resumes in TypeScript.RUN_TS_SDK_NATIVE_DAG_K8S_TESTS. The k8s setup does not let the Dag processor parse the bundle yet: its image has no Node.js, and the bundle is staged only into worker pods.breeze k8s setup-lang-sdk-testbuilds the TypeScript bundle and worker image only with--ts-sdk-native-dag-test, so the Go and Java k8s job needs no Node toolchain.node, soNodeCoordinatorparses the native Dag there. The test checks the Code view, the triggeredtypescript_examplerun (run_type,conf, "Triggered DAG" link), and thattrigger_downstreamdeferred once and resumed on thetypescriptqueue.airflow-e2e-tests/java-native-bundle/, declares one Dag with the interface API and one with annotations. Its JAR goes into the Dags folder;[sdk] dag_bundle_to_coordinatorpicksjava-nativeto parse it, and the Dag processor now runs on theairflow-java-workerimage. The test checks edges, queues, runs, XComs, and the Code view.maven.repo.local, sopublishToMavenLocalno longer writes inside the container when the host UID is 1000.Was generative AI tooling used to co-author this PR?