@@ -22,31 +22,40 @@ Non-Python Task SDKs
2222
2323|experimental |
2424
25- Airflow Dags are always defined in Python, but individual task *implementations * can be written in other
26- languages. When a task runs, Airflow's worker calls out the target language to execute the task logic, which
27- communicates the result back. The Dag author uses a lightweight Python *stub * to declare where the task lives;
28- everything else, including the actual business logic, Airflow API calls, and any library dependencies, lives
29- in a non-Python implementation.
25+ Airflow Dags and tasks can be written in languages other than Python. A language SDK is used in one of two
26+ ways:
27+
28+ * **Native Dags. ** The whole Dag, with its schedule, tasks and dependencies, is declared in the SDK's language.
29+ See :ref: `language-sdks/native-dags `.
30+ * **Mixed-language tasks. ** A Python Dag declares the tasks and their dependencies, and the SDK implements some
31+ of those tasks. See :ref: `language-sdks/mixed-language-tasks `.
32+
33+ Either way, a task runs on a *coordinator *, which starts the language's runtime for the task and relays
34+ messages between it and Airflow. See :ref: `language-sdks/coordinator-config `.
3035
3136.. list-table :: Available language SDKs
3237 :header-rows: 1
33- :widths: 15 40 15 30
38+ :widths: 15 35 15 15 20
3439
3540 * - Language
3641 - Coordinator class
3742 - Min. runtime
43+ - Native Dags
3844 - Guide
3945 * - JVM languages (e.g. Java)
4046 - :class: `task-sdk:airflow.sdk.coordinators.java.JavaCoordinator `
4147 - JRE 17
48+ - Yes
4249 - :doc: `java `
4350 * - Go
4451 - :class: `task-sdk:airflow.sdk.coordinators.executable.ExecutableCoordinator `
4552 - None (native binary)
53+ - Not yet
4654 - :doc: `go `
4755 * - TypeScript
4856 - :class: `task-sdk:airflow.sdk.coordinators.node.NodeCoordinator `
4957 - Node.js 22
58+ - Yes
5059 - :doc: `typescript `
5160
5261.. toctree ::
@@ -56,36 +65,38 @@ in a non-Python implementation.
5665 go
5766 typescript
5867
59- How it works
60- ------------
68+ .. _language-sdks/native-dags :
69+
70+ Native Dags
71+ -----------
72+
73+ A native Dag is declared entirely in the SDK's language: its schedule, its tasks, their options and the
74+ dependencies between them. No Python file is involved.
6175
62- The execution model has three moving parts.
76+ The SDK builds the Dag into an artifact, and the artifact goes into a Dag bundle like any other Dag file. The
77+ Dag processor runs the artifact with the language's runtime to read its Dags, so the Dag processor needs that
78+ runtime and the ``[sdk] `` coordinator configuration. From there, Airflow treats the Dag like any Python Dag:
79+ the scheduler plans its runs, it shows up in the UI, and each task runs on the coordinator its queue maps to,
80+ from the same Dag bundle.
6381
64- **Stub tasks in the Dag **
65- The Dag file declares tasks using :func: `@task.stub <airflow.sdk.task.stub> `. A stub is a normal Airflow
66- task from the scheduler's perspective. It participates in dependencies, retries, pools, and all other
67- task-level features exactly like other ``@task ``-decorated Python functions. The only difference is that
68- the worker does not execute the Python code inside the function definition; instead it delegates execution
69- to a *coordinator *.
82+ The **Code ** view in the Airflow UI shows the source the SDK embeds in the artifact.
7083
71- **Coordinators **
72- A coordinator is a Python object registered in the ``[sdk] coordinators `` configuration. This is considered
73- a part of an Airflow worker. When the worker picks up a stub task, it looks up the coordinator mapped to
74- that task's specified ``queue ``, and uses the coordinator to execute the task. The coordinator is
75- responsible for managing the target language's runtime, forwarding messages from, and relaying results back
76- to Airflow. All coordinators extend
77- :class: `task-sdk:airflow.sdk.execution_time.coordinator.BaseCoordinator `.
84+ For TypeScript, see :ref: `typescript-sdk/native-dag `.
7885
79- **Language runtime **
80- The coordinator calls out one short-lived runtime per task instance. In most cases, this would be a
81- subprocess of an executable implemented in a non-Python language. The runtime receives messages from the
82- worker to identify the workload, executes the task, and communicates through the coordinator as a proxy
83- back to the worker process.
86+ .. _language-sdks/mixed-language-tasks :
87+
88+ Mixed-language tasks
89+ --------------------
90+
91+ A Python Dag declares a task that another language implements as a *stub task *. The scheduler sees a normal
92+ task: it takes part in dependencies, retries, pools and every other task-level feature like any other
93+ ``@task `` function. When a worker picks it up, the worker does not run the Python function body; it hands the
94+ task to the coordinator its queue maps to, which runs the implementation in the target language.
8495
8596.. _language-sdks/stub-tasks :
8697
8798Stub tasks
88- ----------
99+ ~~~~~~~~~~
89100
90101A stub task is declared with the :func: `@task.stub <airflow.sdk.task.stub> ` decorator. Since it is still a
91102Python task declaration, every parameter available on a normal Dag or task applies. Task dependencies are also
@@ -129,16 +140,22 @@ documentation on how to do this correctly.
129140
130141.. note ::
131142
132- For a Dag containing stub tasks, the **Code ** view in the Airflow UI shows only the Python Dag
133- file — including the stub declarations — as the Dag's source. The non-Python implementation source
134- is not displayed anywhere in the UI; consult your project repository or the build artifact shipped
135- in the bundle to inspect it. This is an intentional architecture decision, not a bug.
143+ For a Dag containing stub tasks, the **Code ** view in the Airflow UI shows only the Python Dag file,
144+ including the stub declarations, as the Dag's source. The non-Python implementation source is not
145+ displayed anywhere in the UI; consult your project repository or the build artifact shipped in the bundle
146+ to inspect it. This is an intentional architecture decision, not a bug.
136147
137148.. _language-sdks/coordinator-config :
138149
139150Coordinator configuration
140151-------------------------
141152
153+ A coordinator is a Python object registered in the ``[sdk] coordinators `` configuration, and runs as part of an
154+ Airflow worker. When the worker picks up a task, it looks up the coordinator mapped to the task's ``queue ``,
155+ and uses it to run the task. The coordinator starts one short-lived runtime per task instance, usually a
156+ subprocess of an executable written in the target language, and relays messages between that runtime and the
157+ worker. All coordinators extend :class: `task-sdk:airflow.sdk.execution_time.coordinator.BaseCoordinator `.
158+
142159Coordinators are registered in ``airflow.cfg `` (or via environment variables) under ``[sdk] ``.
143160
144161``coordinators ``
0 commit comments