Repository navigation
Deprecate cyclic TaskGroup dependencies - #73746
Open
dheerajturaga wants to merge 12 commits into
Open
dheerajturaga wants to merge 12 commits into
dheerajturaga wants to merge 12 commits into
Conversation
dheerajturaga
requested review from
XD-DENG,
amoghrajesh,
ashb,
bbovenzi,
bugraoz93,
choo121600,
ephraimbuddy,
guan404ming,
henry3260,
jason810496,
jedcunningham,
pierrejeambrun,
potiuk,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
September 26, 2026 04:42
dheerajturaga
force-pushed
the
deprecate-cyclic-taskgroup-dependencies
branch
2 times, most recently
from
September 29, 2026 19:35
6204b64 to
e08d1f1
Compare
Member
Author
|
@bbovenzi would like your thoughts on the UI, would like to get this in 3.4 as agreed in Lazy consensus https://lists.apache.org/thread/5fpwlxg4y6jhyrw7n2m3jz3o7t6xvhlk cc: @ashb |
Contributor
|
Makes sense to me UI-wise. I am working on improving our warning banner UX but having another category doesn't get in the way of that |
dheerajturaga
force-pushed
the
deprecate-cyclic-taskgroup-dependencies
branch
from
October 6, 2026 22:30
e08d1f1 to
a8f63d7
Compare
dheerajturaga
force-pushed
the
deprecate-cyclic-taskgroup-dependencies
branch
from
October 7, 2026 15:19
a8f63d7 to
895d91e
Compare
1 task done
dheerajturaga
force-pushed
the
deprecate-cyclic-taskgroup-dependencies
branch
from
October 7, 2026 17:18
895d91e to
36fd4cd
Compare
kaxil
reviewed
Oct 7, 2026
1 task done
dheerajturaga
force-pushed
the
deprecate-cyclic-taskgroup-dependencies
branch
from
October 8, 2026 01:47
29b252b to
36958b3
Compare
kaxil
reviewed
Oct 8, 2026
dheerajturaga
force-pushed
the
deprecate-cyclic-taskgroup-dependencies
branch
from
October 8, 2026 22:01
36958b3 to
7ecffd6
Compare
kaxil
reviewed
Oct 9, 2026
kaxil
approved these changes
Oct 10, 2026
kaxil
left a comment
Member
There was a problem hiding this comment.
LGTM, thanks. Feel free to ignore these:
test_dag_topological_sort_task_group_cyclebuilds thea >> bridge >> bshape and callsdag.check_cycle()bare, so it now emitsTaskGroupCycleDeprecationWarningwithout asserting it, and the docstring's "although check_cycle passes" is out of date. Wrapping the call inpytest.warns(TaskGroupCycleDeprecationWarning)would make that explicit.- The function-body
from airflow.sdk import TriggerRuleintest_dag_add_task_checks_trigger_ruleis redundant now thatTriggerRuleis in the module-level imports.
Dags whose TaskGroups depend on each other in a cycle, when each group is treated as a single unit, parse and run today, but planned TaskGroup features need an unambiguous order between groups. Following the dev list discussion, users are warned about these Dags now so they have a release to restructure them before Airflow 3.5 rejects them at parse time.
Following dev list feedback on the lazy consensus, a Dag in which a path leaves a TaskGroup and comes back into it is cyclic when the group is treated as a single unit, even if the tasks inside the group are also ordered directly. The warning only looked at edges into a group's roots, so these Dags would have parsed silently in 3.4 and then failed in 3.5 without a deprecation period.
The warning category is defined only in the Task SDK, and bag_dag already works on SDK Dag objects in the Dag processor, so the import is deliberate rather than a new scheduler or API server dependency on the SDK.
The TaskGroup cycle check walked each group through its iterator, and a mapped group's iterator rejects a task with trigger_rule="always" even when the group is expanded over a literal list. A Dag with no cycle at all that parsed before the deprecation became an import error.
The warning was recorded before the Dag was added to the bag, so a duplicate Dag id that the bag rejected could give the bagged Dag a cycle warning it does not have, or remove the one it does. Folder-wide bags such as `airflow dags reserialize` write these warnings to the database.
A setup and teardown pair in a TaskGroup around work outside it is flagged on purpose, and users with that layout need to know before Airflow 3.5 rejects it. Lang SDK Dags are not checked yet, and turning the warning into an error only catches Dags loaded through a DagBag.
A Dag warning is stored in a Text column, which MySQL limits to 64 KB. Naming every member of every cycle could exceed that for a large Dag, and the failed insert shares a transaction with the serialized Dag writes, so none of the file's parse results would be saved.
Importing airflow.sdk.exceptions does not load the DagBag module, so there is no import cycle to defer.
The same strongly connected component search now lives in the shared TaskGroupMixin, which TaskGroup inherits, so a second copy would only drift.
The noqa hid a runtime airflow.sdk import from the check that keeps core from gaining Task SDK dependencies. DagBag already depends on airflow.sdk.exceptions for AirflowDagCycleException, so sharing that import adds no new dependency and leaves the file's recorded count unchanged.
The bridged TaskGroup in that test is now a deprecated cycle, so check_cycle warns there instead of passing silently, and its docstring no longer held. TriggerRule is already imported at module level for the mapped group test.
dheerajturaga
force-pushed
the
deprecate-cyclic-taskgroup-dependencies
branch
from
October 11, 2026 03:02
9330730 to
264b03d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dags whose TaskGroups depend on each other in a cycle, when each group is treated as a single unit, parse and run today even though planned TaskGroup features need an unambiguous order between groups. Following the dev list discussion, this deprecates them: they keep working, users are warned now, and Airflow 3.5 is planned to reject them at parse time. This replaces the immediate parse-time rejection proposed in #73087.
Treating a group as a single unit means a dependency into or out of any of its tasks counts as a dependency of the whole group. This covers sibling groups that depend on each other in both directions, and a path that leaves a TaskGroup and comes back into it, even when the tasks inside the group are also ordered directly:
To remove the cycle, move
bridgeinto the group, orbout of it.Parsing a Dag with such a cycle now:
TaskGroupCycleDeprecationWarningfromDAG.check_cycle(), so CI can catch it, for example withpytest -W error::airflow.sdk.exceptions.TaskGroupCycleDeprecationWarning;task group cycleDag warning, shown in the UI's Dag warnings and returned byGET /api/v2/dagWarnings.The message names only the tasks and TaskGroups on the cycle:
The Grid/Graph HTTP 500 for these Dags is fixed separately in #73724. The parse-time rejection in 3.5 is tracked in #73678.
related: #73678
Discussion: https://lists.apache.org/thread/sossl7b2w2ftyk4028qrhps2tcdxj2px
Lazy consensus: https://lists.apache.org/thread/b6h120jw7tv1l1zf1ovwrok2zk4hyvwc
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines