Skip to content

Parse native TypeScript Dags with the Node coordinator - #73844

Merged
jason810496 merged 3 commits into
jason/lang-sdk-e2e/02-importer-registryfrom
jason/lang-sdk-e2e/05-node-dag-importer
Oct 1, 2026
Merged

jason810496 merged 3 commits into
jason/lang-sdk-e2e/02-importer-registryfrom
jason/lang-sdk-e2e/05-node-dag-importer

Conversation

@jason810496

@jason810496 jason810496 commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Stack (bottom to top): #73841, #74004, #73842, #73843, #73844, #73845, #73846, #73847

Part of the native Dag e2e stack, related to #71929. Builds on the Java importer layer (new commits only). Merge after #73442 and #73445, which make the TypeScript runtime answer the Dag-parsing request and add triggerDagRun.

Why

The earlier layers let a coordinator parse native Dags with its own runtime, and JavaCoordinator opts in. This makes NodeCoordinator opt in, so Airflow parses the Dags declared in packed *.min.mjs TypeScript bundles.

What changes

A NodeCoordinator without bundles_root now parses the TypeScript bundles in the Dag bundles it serves, and the same entry runs their tasks:

[sdk]
coordinators = {
  "ts": {
    "classpath": "airflow.sdk.coordinators.node.NodeCoordinator",
    "kwargs": {"node_executable": "/usr/local/bin/node"}
  }
}
queue_to_coordinator = {"typescript": "ts"}
  • Only files that end in .min.mjs and start with the header airflow-ts-pack writes are parsed, and safe_mode does not change this check. A file that cannot be read is kept, so its parse reports the error.
  • The parse runs node <bundle> with the supervisor schema version from the bundle metadata. A bundle that fails its integrity check gets an import error.
  • A coordinator with bundles_root still parses nothing, as before.
  • The TypeScript runtime is unchanged here. The TS SDK layer above makes it answer the parse request.
  • typescript.rst documents the setup, that the Dag processor needs Node.js, that a Dag with a cycle is an import error, and the Code view.

Decision left open by #71929

#71929 does not settle what get_source_code returns for a native Dag. Here the Code view shows the bundle's entry module for each of its Dags, since a Dag's source is stored per file. A short notice replaces a source that cannot be read.

Limitations

  • A bundle that only registers TaskHandler objects is parsed too when it sits in a served Dag bundle, because its metadata does not say whether it declares Dags. Each parse launches node and finds no Dags, so the docs suggest keeping such bundles under bundles_root.
  • dag_policy and task_policy do not run on a native Dag. airflow dags test, tasks test, tasks render and tasks list refuse it, and airflow dags reserialize does not store the Dags of *.min.mjs bundles. The docs say so.
  • The intro and the first Limitations bullet of typescript.rst still describe only stub Dags. This stack leaves them, since Update the TypeScript SDK docs for Dags declared in TypeScript #73875 rewrites that page.

How to test

uv run --project task-sdk pytest task-sdk/tests/task_sdk/coordinators/node

The native e2e layer at the top of the stack, above the SDK layers, parses a native TypeScript Dag end to end and checks its source and a triggered run.


Was generative AI tooling used to co-author this PR?

@boring-cyborg boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk kind:documentation labels Sep 28, 2026
@jason810496
jason810496 added this pull request to stack #73848 September 28, 2026 13:20
@jason810496 jason810496 changed the title jason/lang sdk e2e/05 node dag importer Parse native TypeScript Dags with the Node coordinator Sep 28, 2026
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/04-java-dag-importer branch from 4619343 to a80facc Compare September 28, 2026 15:57
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/05-node-dag-importer branch from b441444 to 3ccfffc Compare September 28, 2026 15:57
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/04-java-dag-importer branch from a80facc to d62246f Compare September 29, 2026 01:25
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/05-node-dag-importer branch from 3ccfffc to a9d2d19 Compare September 29, 2026 01:26
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/04-java-dag-importer branch from d62246f to f8acd1d Compare September 29, 2026 01:29
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/05-node-dag-importer branch from a9d2d19 to 1705d22 Compare September 29, 2026 01:29
@jason810496
jason810496 removed this pull request from stack #73848 October 1, 2026 03:02
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/04-java-dag-importer branch from f8acd1d to ceb6b6a Compare October 1, 2026 03:03
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/05-node-dag-importer branch from 1705d22 to 7443b46 Compare October 1, 2026 03:03
@jason810496
jason810496 added this pull request to stack #74005 October 1, 2026 03:04
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/04-java-dag-importer branch from ceb6b6a to ca832b6 Compare October 1, 2026 06:20
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/05-node-dag-importer branch from 7443b46 to a8c1db7 Compare October 1, 2026 06:20
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/04-java-dag-importer branch from ca832b6 to 5e8a705 Compare October 1, 2026 12:41
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/05-node-dag-importer branch from a8c1db7 to 8e95f87 Compare October 1, 2026 12:41
A NodeCoordinator configured without bundles_root now hands out a Dag
importer, so the Dag bundles it serves have their *.min.mjs bundles parsed
by running them with node. Only files that start with the airflow-ts-pack
layout header are parsed. The Dag source shown for such a Dag is the
bundle's entry module, as the bundle embeds it, or a short notice when
there is none to show.
Describe the coordinator configuration that parses native Dags, what the
Dag processor needs for it, what the Code view shows, the cost of
handler-only bundles, and that cluster policies and some CLI commands do
not apply to a native Dag.
@jason810496
jason810496 removed this pull request from stack #74005 October 1, 2026 13:18
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/05-node-dag-importer branch from 8e95f87 to fe68315 Compare October 1, 2026 13:18
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/04-java-dag-importer branch from 5e8a705 to 8ffd19e Compare October 1, 2026 13:18
Base automatically changed from jason/lang-sdk-e2e/04-java-dag-importer to jason/lang-sdk-e2e/02-importer-registry October 1, 2026 13:18
@jason810496
jason810496 merged commit fe68315 into jason/lang-sdk-e2e/02-importer-registry Oct 1, 2026
@jason810496
jason810496 deleted the jason/lang-sdk-e2e/05-node-dag-importer branch October 1, 2026 13:18
@jason810496
jason810496 restored the jason/lang-sdk-e2e/05-node-dag-importer branch October 1, 2026 13:36
@jason810496

Copy link
Copy Markdown
Member Author

Replaced by #74037. GitHub marked this PR as merged into a stack branch when the stack was reordered; nothing was merged into main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant