Skip to content

Commit b830155

Browse files
jason810496claude
andcommitted
Explain dag_bundle_name by the coordinator that parses a Dag file
dag_bundle_name lets Dag processing pick which of several coordinators of one class starts the runtime that parses an artifact. With a single coordinator it can be omitted. Reword the Java, TypeScript and Go docs and READMEs to say so, without tying it to the Python stub Dag. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 9945516 commit b830155

7 files changed

Lines changed: 40 additions & 34 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst‎

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -207,12 +207,11 @@ then register the coordinator and route the queue to it in ``airflow.cfg`` (or t
207207
}
208208
queue_to_coordinator = {"golang": "go"}
209209
210-
``dag_bundle_name`` is optional. Omit it and the coordinator finds executable bundles in the task's own Dag
211-
bundle, the one that holds the Python stub Dag. Set it, as here, to keep them in a separate Dag bundle, for
212-
example when you configure more than one ``ExecutableCoordinator`` and each has its own bundles.
213-
``queue_to_coordinator``
214-
routes stub tasks with ``queue="golang"`` to this Go coordinator. See :ref:`go-sdk/coordinator-config` for
215-
the full list of accepted ``kwargs``.
210+
``dag_bundle_name`` is only needed when you configure more than one ``ExecutableCoordinator``. Each one must
211+
then name its own Dag bundle, so Dag processing knows which one runs an executable bundle to parse it. With a
212+
single ``ExecutableCoordinator``, as here, it is optional: omit it and the coordinator finds executable bundles
213+
in the task's own Dag bundle. ``queue_to_coordinator`` routes stub tasks with ``queue="golang"`` to this Go
214+
coordinator. See :ref:`go-sdk/coordinator-config` for the full list of accepted ``kwargs``.
216215

217216
There is no separate Go worker to run: the Airflow worker forks the bundle binary once per task instance.
218217

@@ -502,7 +501,7 @@ Deploying
502501
~~~~~~~~~
503502

504503
Copy or mount the packed bundle into the Dag bundle the coordinator scans: the one named by
505-
``dag_bundle_name``, or the stub Dag's own bundle. The
504+
``dag_bundle_name``, or the task's own Dag bundle. The
506505
:class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator` scans that Dag bundle recursively,
507506
matches the incoming ``dag_id`` against each bundle's manifest, verifies the bundle's integrity hash, and
508507
launches the matching bundle. Bundles are identified by the trailer magic, not by filename (no extension on
@@ -526,7 +525,8 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
526525
* - ``dag_bundle_name``
527526
- *(auto: task's own bundle)*
528527
- Name of a configured Dag bundle to load executable bundles from. It must name a bundle in
529-
``[dag_processor] dag_bundle_config_list``.
528+
``[dag_processor] dag_bundle_config_list``. Only needed when more than one
529+
``ExecutableCoordinator`` is configured.
530530
* - ``task_startup_timeout``
531531
- ``10.0``
532532
- Seconds to wait for the bundle subprocess to connect after launch. Increase this if your
@@ -537,10 +537,11 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
537537
**Locating bundles.** Executable bundles always live in a Dag bundle:
538538

539539
* Leave ``dag_bundle_name`` unset (the default) to load bundles from the **task's own** Dag bundle,
540-
pinned to the version the run was created with.
541-
* Set ``dag_bundle_name`` to keep bundles in a separate Dag bundle, for example one per
542-
``ExecutableCoordinator`` when you configure more than one. The task uses the version that Dag
543-
bundle is on when it starts, pinned for the whole task.
540+
pinned to the version the run was created with. This is all you need with a single
541+
``ExecutableCoordinator``.
542+
* With more than one ``ExecutableCoordinator``, set ``dag_bundle_name`` on each to a different Dag
543+
bundle, so Dag processing knows which one runs an executable bundle to parse it. The task uses the
544+
version that Dag bundle is on when it starts, pinned for the whole task.
544545

545546
.. _go-sdk/limitations:
546547

‎airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -274,10 +274,10 @@ Configure Airflow so the parent JAR directory is a Dag bundle, the coordinator s
274274
}
275275
queue_to_coordinator = {"java": "java"}
276276
277-
``dag_bundle_name`` is only needed when you configure more than one ``JavaCoordinator``. Each one must then
278-
name its own Dag bundle, otherwise the Dag importer cannot tell which coordinator should parse a JAR. With a
279-
single ``JavaCoordinator``, as here, it is optional: omit it and the coordinator finds the JARs in the task's
280-
own Dag bundle, the one that holds ``sales_pipeline.py``.
277+
``dag_bundle_name`` is only needed when you configure more than one ``JavaCoordinator``, for example ``jdk-11``
278+
and ``jdk-17``. Each one must then name its own Dag bundle, so Dag processing knows which one starts the JVM
279+
that parses a JAR. With a single ``JavaCoordinator``, as here, it is optional: omit it and the coordinator finds
280+
the JARs in the task's own Dag bundle.
281281

282282
``java`` is a user-chosen coordinator name, not a reserved value. The value assigned to the queue in
283283
``queue_to_coordinator`` must match a key in ``coordinators``.
@@ -1090,7 +1090,7 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
10901090
* Leave ``dag_bundle_name`` unset (the default) to load JARs from the **task's own** Dag bundle, pinned
10911091
to the version the run was created with. This is all you need with a single ``JavaCoordinator``.
10921092
* With more than one ``JavaCoordinator``, set ``dag_bundle_name`` on each to a different Dag bundle, so
1093-
the Dag importer can tell which coordinator parses which JAR. The task uses the version that bundle is
1093+
Dag processing knows which one starts the JVM that parses a JAR. The task uses the version that bundle is
10941094
on when it starts, pinned for the whole task.
10951095

10961096
.. note::

‎airflow-core/docs/authoring-and-scheduling/language-sdks/typescript.rst‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -238,10 +238,10 @@ variables):
238238
}
239239
queue_to_coordinator = {"typescript": "ts"}
240240
241-
``dag_bundle_name`` is only needed when you configure more than one ``NodeCoordinator``. Each one must then
242-
name its own Dag bundle, otherwise the Dag importer cannot tell which coordinator should parse a ``*.min.mjs``
243-
bundle. With a single ``NodeCoordinator``, as here, it is optional: omit it and the coordinator finds the
244-
bundles in the task's own Dag bundle, the one that holds the Python stub Dag. ``queue_to_coordinator`` routes stub tasks with
241+
``dag_bundle_name`` is only needed when you configure more than one ``NodeCoordinator``, for example one per
242+
Node.js version. Each one must then name its own Dag bundle, so Dag processing knows which one starts Node.js
243+
to parse a ``*.min.mjs`` bundle. With a single ``NodeCoordinator``, as here, it is optional: omit it and the
244+
coordinator finds the bundles in the task's own Dag bundle. ``queue_to_coordinator`` routes stub tasks with
245245
``queue="typescript"`` to this coordinator. See :ref:`typescript-sdk/coordinator-config` for the full list
246246
of accepted ``kwargs``.
247247

@@ -526,7 +526,7 @@ Deploying
526526
~~~~~~~~~
527527

528528
Copy or mount the bundle into the Dag bundle the coordinator scans: the one named by ``dag_bundle_name``,
529-
or the stub Dag's own bundle. :class:`~airflow.sdk.coordinators.node.NodeCoordinator` searches that Dag
529+
or the task's own Dag bundle. :class:`~airflow.sdk.coordinators.node.NodeCoordinator` searches that Dag
530530
bundle recursively and launches the first integrity-verified ``*.min.mjs`` bundle whose metadata declares the
531531
task instance's Dag. The artifact's name does not matter beyond that suffix, so one Dag bundle can hold several
532532
bundles and a Dag is routed to whichever declares it. If multiple bundles declare the same Dag, the first in
@@ -567,7 +567,7 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
567567
* Leave ``dag_bundle_name`` unset (the default) to load the bundle from the **task's own** Dag bundle,
568568
pinned to the version the run was created with. This is all you need with a single ``NodeCoordinator``.
569569
* With more than one ``NodeCoordinator``, set ``dag_bundle_name`` on each to a different Dag bundle, so
570-
the Dag importer can tell which coordinator parses which bundle. The task uses the version that Dag
570+
Dag processing knows which one starts Node.js to parse a bundle. The task uses the version that Dag
571571
bundle is on when it starts, pinned for the whole task.
572572

573573
Limitations

‎go-sdk/README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -292,9 +292,10 @@ the full range of task states, and alternate XCom backends without implementing
292292
queue_to_coordinator = {"golang": "go"}
293293
```
294294

295-
`dag_bundle_name` is optional. Omit it and the coordinator finds executable bundles in the task's own
296-
Dag bundle, the one that holds the Python stub Dag. Set it, as here, to keep them in a separate Dag bundle. `queue_to_coordinator` routes stub tasks with
297-
`queue="golang"` to this Go coordinator.
295+
`dag_bundle_name` is only needed when more than one `ExecutableCoordinator` is configured. Each one must
296+
then name its own Dag bundle, so Dag processing knows which one runs an executable bundle to parse it. With
297+
a single `ExecutableCoordinator` it is optional: omit it and the coordinator finds executable bundles in the
298+
task's own Dag bundle. `queue_to_coordinator` routes stub tasks with `queue="golang"` to this Go coordinator.
298299

299300
> [!IMPORTANT]
300301
> The coordinator is part of the Airflow worker, so the `[sdk]` config (and the executable bundle

‎java-sdk/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,8 +120,10 @@ Now `cd example` into the example project, and
120120
export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}'
121121
```
122122

123-
With a single `JavaCoordinator`, `dag_bundle_name` is optional. It is set here because the JARs live in
124-
`build/bundle`, outside the Dags folder; omit it if you ship the JARs in the same Dag bundle as the stub Dag.
123+
`dag_bundle_name` is only needed when more than one `JavaCoordinator` is configured, for example `jdk-11` and
124+
`jdk-17`. Each one must then name its own Dag bundle, so Dag processing knows which one starts the JVM that
125+
parses a JAR. With a single `JavaCoordinator` it is optional: omit it and the coordinator finds the JARs in the
126+
task's own Dag bundle.
125127

126128
* Ensure the Connection and Variable needed by the example DAG are available:
127129

‎ts-sdk/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -98,11 +98,11 @@ coordinators = {
9898
queue_to_coordinator = {"typescript": "ts"}
9999
```
100100

101-
The stub Dag's own Dag bundle is searched recursively for `*.min.mjs` bundles built with `airflow-ts-pack`
101+
The task's own Dag bundle is searched recursively for `*.min.mjs` bundles built with `airflow-ts-pack`
102102
(see [Packing bundles](#packing-bundles)), which embeds the Airflow metadata in the bundle itself. Set
103-
`dag_bundle_name` only when you configure more than one `NodeCoordinator`: each must then name its own Dag
104-
bundle from `[dag_processor] dag_bundle_config_list`, otherwise the Dag importer cannot tell which coordinator
105-
should parse a bundle.
103+
`dag_bundle_name` only when you configure more than one `NodeCoordinator`, for example one per Node.js version.
104+
Each one must then name its own Dag bundle from `[dag_processor] dag_bundle_config_list`, so Dag processing
105+
knows which one starts Node.js to parse a bundle.
106106

107107
TypeScript entrypoint:
108108

‎ts-sdk/example/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,10 @@ export AIRFLOW__SDK__COORDINATORS='{
8383
export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"typescript": "ts"}'
8484
```
8585

86-
With a single `NodeCoordinator`, `dag_bundle_name` is optional. It is set here because the bundle lives in
87-
`dist/`, outside the Dags folder; omit it if you copy the bundle into the same Dag bundle as the stub Dags.
86+
`dag_bundle_name` is only needed when more than one `NodeCoordinator` is configured, for example one per
87+
Node.js version. Each one must then name its own Dag bundle, so Dag processing knows which one starts Node.js
88+
to parse a bundle. With a single `NodeCoordinator` it is optional: omit it and the coordinator finds the bundle
89+
in the task's own Dag bundle.
8890

8991
Copy both files in `dags/` into your Airflow Dags folder.
9092

0 commit comments

Comments
 (0)