Skip to content

Commit d21aea9

Browse files
jason810496claude
authored andcommitted
Say dag_bundle_name is only needed for two coordinators of one class
Reword the docs so dag_bundle_name reads as optional with one coordinator per class, and needed when one class is configured more than once. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 39fcab0 commit d21aea9

8 files changed

Lines changed: 46 additions & 29 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -207,8 +207,10 @@ then register the coordinator and route the queue to it in ``airflow.cfg`` (or t
207207
}
208208
queue_to_coordinator = {"golang": "go"}
209209
210-
``dag_bundle_name`` names the Dag bundle the coordinator scans for executable bundles. Omit it to ship the
211-
executable bundles in the same Dag bundle as the Python stub Dag. ``queue_to_coordinator``
210+
``dag_bundle_name`` is optional. Omit it and the coordinator finds executable bundles in the task's own Dag
211+
bundle, the one that holds the Python stub Dag. Set it, as here, to keep them in a separate Dag bundle, for
212+
example when you configure more than one ``ExecutableCoordinator`` and each has its own bundles.
213+
``queue_to_coordinator``
212214
routes stub tasks with ``queue="golang"`` to this Go coordinator. See :ref:`go-sdk/coordinator-config` for
213215
the full list of accepted ``kwargs``.
214216

@@ -534,11 +536,11 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
534536

535537
**Locating bundles.** Executable bundles always live in a Dag bundle:
536538

537-
* Set ``dag_bundle_name`` to load bundles from a configured Dag bundle, so they are delivered
538-
and versioned through the same bundle machinery as your Dags. The task uses the version that
539+
* Leave ``dag_bundle_name`` unset (the default) to load bundles from the **task's own** Dag bundle,
540+
pinned to the version the run was created with.
541+
* Set ``dag_bundle_name`` to keep bundles in a separate Dag bundle, for example one per
542+
``ExecutableCoordinator`` when you configure more than one. The task uses the version that Dag
539543
bundle is on when it starts, pinned for the whole task.
540-
* Leave it unset (the default) to load bundles from the **task's own** Dag bundle, pinned
541-
to the version the run was created with.
542544

543545
.. _go-sdk/limitations:
544546

‎airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -274,8 +274,10 @@ Configure Airflow so the parent JAR directory is a Dag bundle, the coordinator s
274274
}
275275
queue_to_coordinator = {"java": "java"}
276276
277-
``dag_bundle_name`` names the Dag bundle that holds the JARs. You can instead omit it and ship the JARs in
278-
the same Dag bundle as ``sales_pipeline.py``.
277+
``dag_bundle_name`` is only needed when you configure more than one ``JavaCoordinator``. Each one must then
278+
name its own Dag bundle, otherwise the Dag importer cannot tell which coordinator should parse a JAR. With a
279+
single ``JavaCoordinator``, as here, it is optional: omit it and the coordinator finds the JARs in the task's
280+
own Dag bundle, the one that holds ``sales_pipeline.py``.
279281

280282
``java`` is a user-chosen coordinator name, not a reserved value. The value assigned to the queue in
281283
``queue_to_coordinator`` must match a key in ``coordinators``.
@@ -1063,7 +1065,8 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
10631065
* - ``dag_bundle_name``
10641066
- *(auto: task's own bundle)*
10651067
- Name of a configured Dag bundle to load JARs from. It must name a bundle in
1066-
``[dag_processor] dag_bundle_config_list``.
1068+
``[dag_processor] dag_bundle_config_list``. Only needed when more than one
1069+
``JavaCoordinator`` is configured.
10671070
* - ``java_executable``
10681071
- ``"java"``
10691072
- Path to the ``java`` binary. Defaults to ``java`` on ``$PATH``.
@@ -1084,11 +1087,11 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
10841087

10851088
**Locating JARs.** JARs always live in a Dag bundle:
10861089

1087-
* Set ``dag_bundle_name`` to load JARs from a configured Dag bundle, so they are delivered and
1088-
versioned through the same bundle machinery as your Dags. The task uses the version that bundle
1089-
is on when it starts, pinned for the whole task.
1090-
* Leave it unset (the default) to load JARs from the **task's own** Dag bundle, pinned to
1091-
the version the run was created with.
1090+
* Leave ``dag_bundle_name`` unset (the default) to load JARs from the **task's own** Dag bundle, pinned
1091+
to the version the run was created with. This is all you need with a single ``JavaCoordinator``.
1092+
* With more than one ``JavaCoordinator``, set ``dag_bundle_name`` on each to a different Dag bundle, so
1093+
the Dag importer can tell which coordinator parses which JAR. The task uses the version that bundle is
1094+
on when it starts, pinned for the whole task.
10921095

10931096
.. note::
10941097

‎airflow-core/docs/authoring-and-scheduling/language-sdks/typescript.rst‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -238,8 +238,10 @@ variables):
238238
}
239239
queue_to_coordinator = {"typescript": "ts"}
240240
241-
``dag_bundle_name`` names the Dag bundle the coordinator scans for ``*.min.mjs`` bundles. Omit it to ship the
242-
bundles in the same Dag bundle as the Python stub Dag. ``queue_to_coordinator`` routes stub tasks with
241+
``dag_bundle_name`` is only needed when you configure more than one ``NodeCoordinator``. Each one must then
242+
name its own Dag bundle, otherwise the Dag importer cannot tell which coordinator should parse a ``*.min.mjs``
243+
bundle. With a single ``NodeCoordinator``, as here, it is optional: omit it and the coordinator finds the
244+
bundles in the task's own Dag bundle, the one that holds the Python stub Dag. ``queue_to_coordinator`` routes stub tasks with
243245
``queue="typescript"`` to this coordinator. See :ref:`typescript-sdk/coordinator-config` for the full list
244246
of accepted ``kwargs``.
245247

@@ -548,7 +550,8 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
548550
* - ``dag_bundle_name``
549551
- *(auto: task's own bundle)*
550552
- Name of a configured Dag bundle to load the ``*.min.mjs`` bundle from. It must name a bundle in
551-
``[dag_processor] dag_bundle_config_list``.
553+
``[dag_processor] dag_bundle_config_list``. Only needed when more than one ``NodeCoordinator`` is
554+
configured.
552555
* - ``node_executable``
553556
- ``"node"``
554557
- Path to the ``node`` binary. Defaults to ``node`` on ``$PATH``.
@@ -561,11 +564,11 @@ All ``kwargs`` in the ``coordinators`` config entry are passed to the
561564

562565
**Locating the bundle.** The ``*.min.mjs`` bundle always lives in a Dag bundle:
563566

564-
* Set ``dag_bundle_name`` to load the bundle from a configured Dag bundle, so it is delivered
565-
and versioned through the same bundle machinery as your Dags. The task uses the version that
567+
* Leave ``dag_bundle_name`` unset (the default) to load the bundle from the **task's own** Dag bundle,
568+
pinned to the version the run was created with. This is all you need with a single ``NodeCoordinator``.
569+
* With more than one ``NodeCoordinator``, set ``dag_bundle_name`` on each to a different Dag bundle, so
570+
the Dag importer can tell which coordinator parses which bundle. The task uses the version that Dag
566571
bundle is on when it starts, pinned for the whole task.
567-
* Leave it unset (the default) to load the bundle from the **task's own** Dag bundle, pinned to the
568-
version the run was created with.
569572

570573
Limitations
571574
-----------

‎airflow-core/src/airflow/config_templates/config.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2192,9 +2192,11 @@ sdk:
21922192
versions).
21932193
21942194
A coordinator finds its artifacts in the task's own Dag bundle, or in the
2195-
Dag bundle named by its ``dag_bundle_name`` kwarg. Two coordinators whose
2196-
Dag importers parse the same file extension need different
2197-
``dag_bundle_name`` values.
2195+
Dag bundle named by its ``dag_bundle_name`` kwarg. ``dag_bundle_name`` is
2196+
only needed when the same coordinator class is configured more than once
2197+
(for example, two ``JavaCoordinator`` entries). Each one must then name
2198+
its own Dag bundle, otherwise the Dag importer cannot tell which
2199+
coordinator should parse an artifact. With one entry per class, omit it.
21982200
21992201
An entry may also carry an optional ``extra`` object for additional
22002202
information associated with the coordinator that the coordinator itself

‎go-sdk/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -292,8 +292,8 @@ the full range of task states, and alternate XCom backends without implementing
292292
queue_to_coordinator = {"golang": "go"}
293293
```
294294

295-
`dag_bundle_name` names the Dag bundle the coordinator scans for executable bundles. Omit it to ship
296-
them in the same Dag bundle as the Python stub Dag. `queue_to_coordinator` routes stub tasks with
295+
`dag_bundle_name` is optional. Omit it and the coordinator finds executable bundles in the task's own
296+
Dag bundle, the one that holds the Python stub Dag. Set it, as here, to keep them in a separate Dag bundle. `queue_to_coordinator` routes stub tasks with
297297
`queue="golang"` to this Go coordinator.
298298

299299
> [!IMPORTANT]

‎java-sdk/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,9 @@ Now `cd example` into the example project, and
120120
export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}'
121121
```
122122

123+
With a single `JavaCoordinator`, `dag_bundle_name` is optional. It is set here because the JARs live in
124+
`build/bundle`, outside the Dags folder; omit it if you ship the JARs in the same Dag bundle as the stub Dag.
125+
123126
* Ensure the Connection and Variable needed by the example DAG are available:
124127

125128
```bash

‎ts-sdk/README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -99,9 +99,10 @@ queue_to_coordinator = {"typescript": "ts"}
9999
```
100100

101101
The stub Dag's own Dag bundle is searched recursively for `*.min.mjs` bundles built with `airflow-ts-pack`
102-
(see [Packing bundles](#packing-bundles)), which embeds the Airflow metadata in the bundle itself. To keep
103-
the `*.min.mjs` bundles in a separate Dag bundle, set `dag_bundle_name` to a bundle configured in
104-
`[dag_processor] dag_bundle_config_list`.
102+
(see [Packing bundles](#packing-bundles)), which embeds the Airflow metadata in the bundle itself. Set
103+
`dag_bundle_name` only when you configure more than one `NodeCoordinator`: each must then name its own Dag
104+
bundle from `[dag_processor] dag_bundle_config_list`, otherwise the Dag importer cannot tell which coordinator
105+
should parse a bundle.
105106

106107
TypeScript entrypoint:
107108

‎ts-sdk/example/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,9 @@ export AIRFLOW__SDK__COORDINATORS='{
8383
export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"typescript": "ts"}'
8484
```
8585

86+
With a single `NodeCoordinator`, `dag_bundle_name` is optional. It is set here because the bundle lives in
87+
`dist/`, outside the Dags folder; omit it if you copy the bundle into the same Dag bundle as the stub Dags.
88+
8689
Copy both files in `dags/` into your Airflow Dags folder.
8790

8891
The example also reads one Variable and one Connection:

0 commit comments

Comments
 (0)