Repository navigation
Add user-facing docs for the new Task Loops feature - #74274
Merged
Merged
Conversation
ashb
added this pull request to stack #74276
October 5, 2026 16:35
ashb
removed this pull request from stack #74276
October 5, 2026 16:36
Member
Author
|
Turns out you can only have a stack targeting main. So I'll wait to create the rest of the stack until #74222 lands. |
TJaniF
reviewed
Oct 6, 2026
TJaniF
left a comment
Contributor
There was a problem hiding this comment.
This looks great! Just had a few nits/suggestions and one question. :)
ashb
added this pull request to stack #74339
October 6, 2026 13:52
kaxil
reviewed
Oct 6, 2026
ashb
force-pushed
the
task-loops-stack-1
branch
2 times, most recently
from
October 7, 2026 12:26
6ec5dd6 to
29a26cc
Compare
ashb
commented
Oct 7, 2026
ashb
removed this pull request from stack #74339
October 7, 2026 15:20
ashb
force-pushed
the
task-loops-stack-1
branch
from
October 7, 2026 15:22
09d1041 to
5ca5192
Compare
ashb
added this pull request to stack #74410
October 7, 2026 15:22
ashb
marked this pull request as ready for review
October 7, 2026 15:27
ashb
force-pushed
the
task-loops-stack-1
branch
4 times, most recently
from
October 8, 2026 16:06
c89e715 to
2e45493
Compare
kaxil
approved these changes
Oct 8, 2026
ashb
force-pushed
the
task-loops-stack-1
branch
7 times, most recently
from
October 9, 2026 22:18
edc5fbb to
23eaa02
Compare
Task Loops, introduced in AIP-111, are a whole new way for dag authors to thing
Loops chaNge how a Dag author thinks about repeating work, and the change to
implement it reaches storage, the scheduler, the Execution API and the UI. Since
all these changes are relatively large, it is helpful before we introduct those
that reviewers have a mental model of what the behaviour should be, and the
features we want to allow are. Rather than explaining this just in PR messages
or review threads, lets add this as user facing docs, as most of that will be
needed there too.
The new Loops doc covers how a loop decides whether to run another pass, how
a task reads the previous pass's result, what clearing part of a loop does to
the rest, and how loops interact with mapped tasks. At this point/in this PR,
nothing exists as code mind you.
The short "loop or mapped tasks" page is net-new, and is ther mostly as a
jumping off point for "here are the dynamic features you can build your dag
with".
This oVerview is also why "dynamic mapped tasks" is getting a name change.
Mapping is no longer the only way for a Dag's shape to depend on data at
runtime, and arguably, mapped tasks aren't really that "dynamic" anymore.
"Mapped tasks" is what the UI and the API already call the result ("Clear All
Mapped Tasks", MappedOperator), and it reads next to "Loops" as the title of the
page the overview links to. "Task mapping" never appears in the UI.
However, please note that the rename is "soft" on purpose. The mappIng page
keeps its file name and its `dynamic-task-mapping` label, and the old name stays
in its opening sentence so existing searches still land. Provider docs link that
page by path through intersphinx so moving the file would break their builds
until each one is updated. The `dynamic-task-mapping` capability key that the
language SDKs report is an identifier rather than prose and is left alone.
The snippets are inline here because the example Dags that the guide will
include ship with the authoring API, and a file that calls `.loop()` cannot be
imported before `.loop()` exists. That change replaces them with included copies
that a test executes, so the documented code cannot drift from the behaviour.
The placeholder newsfragment is also here because `.loop()` is a new
user-visible Dag authoring feature that ships through task-sdk with
airflow-core. It will grow and be adjusted by future PRs in this stack.
ashb
force-pushed
the
task-loops-stack-1
branch
from
October 10, 2026 07:05
23eaa02 to
dd9cac4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task Loops, introduced in AIP-111, are a whole new way for dag authors
to thing Loops change how a Dag author thinks about repeating work, and
the change to implement it reaches storage, the scheduler, the Execution
API and the UI. Since all these changes are relatively large, it is
helpful before we introduct those that reviewers have a mental model
of what the behaviour should be, and the features we want to allow are.
Rather than explaining this just in PR messages or review threads, lets
add this as user facing docs, as most of that will be needed there too.
The new Loops doc covers how a loop decides whether to run another pass,
how a task reads the previous pass's result, what clearing part of a
loop does to the rest, and how loops interact with mapped tasks. At this
point/in this PR, nothing exists as code mind you.
The short "loop or mapped tasks" page is net-new, and is ther mostly
as a jumping off point for "here are the dynamic features you can build
your dag with".
This overview is also why "dynamic mapped tasks" is getting a name
change. Mapping is no longer the only way for a Dag's shape to depend on
data at runtime, and arguably, mapped tasks aren't really that "dynamic"
anymore. "Mapped tasks" is what the UI and the API already call the
result ("Clear All Mapped Tasks", MappedOperator), and it reads next to
"Loops" as the title of the page the overview links to. "Task mapping"
never appears in the UI.
However, please note that the rename is "soft" on purpose. The mapping
page keeps its file name and its
dynamic-task-mappinglabel, andthe old name stays in its opening sentence so existing searches still
land. Provider docs link that page by path through intersphinx so
moving the file would break their builds until each one is updated. The
dynamic-task-mappingcapability key that the language SDKs report isan identifier rather than prose and is left alone.
The snippets are inline here because the example Dags that the guide
will include ship with the authoring API, and a file that calls
.loop()cannot be imported before.loop()exists. That changereplaces them with included copies that a test executes, so the
documented code cannot drift from the behaviour.
The placeholder newsfragment is also here because
.loop()is a newuser-visible Dag authoring feature that ships through task-sdk with
airflow-core. It will grow and be adjusted by future PRs in this stack.