Skip to content

Commit 36958b3

Browse files
committed
Document which Dags the TaskGroup cycle check covers
A setup and teardown pair in a TaskGroup around work outside it is flagged on purpose, and users with that layout need to know before Airflow 3.5 rejects it. Lang SDK Dags are not checked yet, and turning the warning into an error only catches Dags loaded through a DagBag.
1 parent 4a70751 commit 36958b3

2 files changed

Lines changed: 24 additions & 4 deletions

File tree

‎airflow-core/docs/core-concepts/dags.rst‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -663,15 +663,33 @@ group are ordered directly as well:
663663
664664
a >> bridge >> b # group -> bridge -> group
665665
666+
This includes a setup and teardown pair inside a TaskGroup with the work they wrap outside it. This is
667+
intended: a TaskGroup that holds a setup and its teardown must also hold the tasks between them.
668+
669+
.. code-block:: python
670+
671+
with TaskGroup("cluster"):
672+
create = EmptyOperator(task_id="create")
673+
delete = EmptyOperator(task_id="delete")
674+
675+
work = EmptyOperator(task_id="work")
676+
677+
create >> work >> delete.as_teardown(setups=create) # cluster -> work -> cluster
678+
666679
These Dags still parse and run, but features that act on a TaskGroup as a whole need an unambiguous order
667680
between groups. Parsing them emits a ``TaskGroupCycleDeprecationWarning`` and a Dag warning in the UI that
668681
name the TaskGroups and tasks involved.
669682

670683
To remove the cycle, move tasks between TaskGroups, or out of them, so that each group depends on the others
671-
in one direction only. In the second example, move ``bridge`` into ``group``, or move ``b`` out of it.
684+
in one direction only. In the second example, move ``bridge`` into ``group``, or move ``b`` out of it. In the
685+
setup and teardown example, move ``work`` into ``cluster``.
686+
687+
Only Dags authored with the Python Task SDK are checked. Dags from the Go, Java and TypeScript SDKs are not
688+
checked yet.
672689

673-
To catch these Dags in CI, turn the warning into an error. For example, when a pytest test loads your Dags
674-
into a ``DagBag`` and asserts there are no import errors:
690+
To catch these Dags in CI, turn the warning into an error. The check runs when a Dag is added to a
691+
``DagBag``, so this works for a pytest test that loads your Dags into a ``DagBag`` and asserts there are no
692+
import errors, but not for one that only imports a Dag file:
675693

676694
.. code-block:: bash
677695

‎airflow-core/newsfragments/73746.significant.rst‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ and ``b`` in the same TaskGroup and ``bridge`` outside it, makes the group depen
1515
- Move tasks between TaskGroups, or out of them, so that each group depends on the others in one direction
1616
only. In the example above, move ``bridge`` into the group, or move ``b`` out of it.
1717
- To catch these Dags in CI, run your Dag tests with
18-
``-W error::airflow.sdk.exceptions.TaskGroupCycleDeprecationWarning``.
18+
``-W error::airflow.sdk.exceptions.TaskGroupCycleDeprecationWarning``. The check runs when a Dag is
19+
added to a ``DagBag``, so only tests that load your Dags into a ``DagBag`` and assert there are no import
20+
errors catch them; a test that only imports a Dag file does not.
1921

2022
See "Cyclic TaskGroup dependencies" in the Dags documentation.

0 commit comments

Comments
 (0)