Skip to content

Commit eb91ce4

Browse files
committed
Update the TypeScript SDK docs for Dags declared in TypeScript
Describe the TypeScript SDK as it will be once the native Dag authoring surface and native Dag parsing land: declaring a whole Dag in TypeScript, and implementing mixed-language tasks, from one guide.
1 parent a8a6257 commit eb91ce4

5 files changed

Lines changed: 637 additions & 557 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst‎

Lines changed: 49 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -22,31 +22,40 @@ Non-Python Task SDKs
2222

2323
|experimental|
2424

25-
Airflow Dags are always defined in Python, but individual task *implementations* can be written in other
26-
languages. When a task runs, Airflow's worker calls out the target language to execute the task logic, which
27-
communicates the result back. The Dag author uses a lightweight Python *stub* to declare where the task lives;
28-
everything else, including the actual business logic, Airflow API calls, and any library dependencies, lives
29-
in a non-Python implementation.
25+
Airflow Dags and tasks can be written in languages other than Python. A language SDK is used in one of two
26+
ways:
27+
28+
* **Native Dags.** The whole Dag, with its schedule, tasks and dependencies, is declared in the SDK's language.
29+
See :ref:`language-sdks/native-dags`.
30+
* **Mixed-language tasks.** A Python Dag declares the tasks and their dependencies, and the SDK implements some
31+
of those tasks. See :ref:`language-sdks/mixed-language-tasks`.
32+
33+
Either way, a task runs on a *coordinator*, which starts the language's runtime for the task and relays
34+
messages between it and Airflow. See :ref:`language-sdks/coordinator-config`.
3035

3136
.. list-table:: Available language SDKs
3237
:header-rows: 1
33-
:widths: 15 40 15 30
38+
:widths: 15 35 15 15 20
3439

3540
* - Language
3641
- Coordinator class
3742
- Min. runtime
43+
- Native Dags
3844
- Guide
3945
* - JVM languages (e.g. Java)
4046
- :class:`task-sdk:airflow.sdk.coordinators.java.JavaCoordinator`
4147
- JRE 17
48+
- Yes
4249
- :doc:`java`
4350
* - Go
4451
- :class:`task-sdk:airflow.sdk.coordinators.executable.ExecutableCoordinator`
4552
- None (native binary)
53+
- Not yet
4654
- :doc:`go`
4755
* - TypeScript
4856
- :class:`task-sdk:airflow.sdk.coordinators.node.NodeCoordinator`
4957
- Node.js 22
58+
- Yes
5059
- :doc:`typescript`
5160

5261
.. toctree::
@@ -56,36 +65,38 @@ in a non-Python implementation.
5665
go
5766
typescript
5867

59-
How it works
60-
------------
68+
.. _language-sdks/native-dags:
69+
70+
Native Dags
71+
-----------
72+
73+
A native Dag is declared entirely in the SDK's language: its schedule, its tasks, their options and the
74+
dependencies between them. No Python file is involved.
6175

62-
The execution model has three moving parts.
76+
The SDK builds the Dag into an artifact, and the artifact goes into a Dag bundle like any other Dag file. The
77+
Dag processor runs the artifact with the language's runtime to read its Dags, so the Dag processor needs that
78+
runtime and the ``[sdk]`` coordinator configuration. From there, Airflow treats the Dag like any Python Dag:
79+
the scheduler plans its runs, it shows up in the UI, and each task runs on the coordinator its queue maps to,
80+
from the same Dag bundle.
6381

64-
**Stub tasks in the Dag**
65-
The Dag file declares tasks using :func:`@task.stub <airflow.sdk.task.stub>`. A stub is a normal Airflow
66-
task from the scheduler's perspective. It participates in dependencies, retries, pools, and all other
67-
task-level features exactly like other ``@task``-decorated Python functions. The only difference is that
68-
the worker does not execute the Python code inside the function definition; instead it delegates execution
69-
to a *coordinator*.
82+
The **Code** view in the Airflow UI shows the source the SDK embeds in the artifact.
7083

71-
**Coordinators**
72-
A coordinator is a Python object registered in the ``[sdk] coordinators`` configuration. This is considered
73-
a part of an Airflow worker. When the worker picks up a stub task, it looks up the coordinator mapped to
74-
that task's specified ``queue``, and uses the coordinator to execute the task. The coordinator is
75-
responsible for managing the target language's runtime, forwarding messages from, and relaying results back
76-
to Airflow. All coordinators extend
77-
:class:`task-sdk:airflow.sdk.execution_time.coordinator.BaseCoordinator`.
84+
For TypeScript, see :ref:`typescript-sdk/native-dag`.
7885

79-
**Language runtime**
80-
The coordinator calls out one short-lived runtime per task instance. In most cases, this would be a
81-
subprocess of an executable implemented in a non-Python language. The runtime receives messages from the
82-
worker to identify the workload, executes the task, and communicates through the coordinator as a proxy
83-
back to the worker process.
86+
.. _language-sdks/mixed-language-tasks:
87+
88+
Mixed-language tasks
89+
--------------------
90+
91+
A Python Dag declares a task that another language implements as a *stub task*. The scheduler sees a normal
92+
task: it takes part in dependencies, retries, pools and every other task-level feature like any other
93+
``@task`` function. When a worker picks it up, the worker does not run the Python function body; it hands the
94+
task to the coordinator its queue maps to, which runs the implementation in the target language.
8495

8596
.. _language-sdks/stub-tasks:
8697

8798
Stub tasks
88-
----------
99+
~~~~~~~~~~
89100

90101
A stub task is declared with the :func:`@task.stub <airflow.sdk.task.stub>` decorator. Since it is still a
91102
Python task declaration, every parameter available on a normal Dag or task applies. Task dependencies are also
@@ -129,16 +140,22 @@ documentation on how to do this correctly.
129140

130141
.. note::
131142

132-
For a Dag containing stub tasks, the **Code** view in the Airflow UI shows only the Python Dag
133-
file — including the stub declarations — as the Dag's source. The non-Python implementation source
134-
is not displayed anywhere in the UI; consult your project repository or the build artifact shipped
135-
in the bundle to inspect it. This is an intentional architecture decision, not a bug.
143+
For a Dag containing stub tasks, the **Code** view in the Airflow UI shows only the Python Dag file,
144+
including the stub declarations, as the Dag's source. The non-Python implementation source is not
145+
displayed anywhere in the UI; consult your project repository or the build artifact shipped in the bundle
146+
to inspect it. This is an intentional architecture decision, not a bug.
136147

137148
.. _language-sdks/coordinator-config:
138149

139150
Coordinator configuration
140151
-------------------------
141152

153+
A coordinator is a Python object registered in the ``[sdk] coordinators`` configuration, and runs as part of an
154+
Airflow worker. When the worker picks up a task, it looks up the coordinator mapped to the task's ``queue``,
155+
and uses it to run the task. The coordinator starts one short-lived runtime per task instance, usually a
156+
subprocess of an executable written in the target language, and relays messages between that runtime and the
157+
worker. All coordinators extend :class:`task-sdk:airflow.sdk.execution_time.coordinator.BaseCoordinator`.
158+
142159
Coordinators are registered in ``airflow.cfg`` (or via environment variables) under ``[sdk]``.
143160

144161
``coordinators``

0 commit comments

Comments
 (0)