Skip to content

Add user-facing docs for the new Task Loops feature - #74274

Merged
ashb merged 1 commit into
mainfrom
task-loops-stack-1
Oct 10, 2026
Merged

ashb merged 1 commit into
mainfrom
task-loops-stack-1

Conversation

@ashb

@ashb ashb commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Task Loops, introduced in AIP-111, are a whole new way for dag authors
to thing Loops change how a Dag author thinks about repeating work, and
the change to implement it reaches storage, the scheduler, the Execution
API and the UI. Since all these changes are relatively large, it is
helpful before we introduct those that reviewers have a mental model
of what the behaviour should be, and the features we want to allow are.
Rather than explaining this just in PR messages or review threads, lets
add this as user facing docs, as most of that will be needed there too.

The new Loops doc covers how a loop decides whether to run another pass,
how a task reads the previous pass's result, what clearing part of a
loop does to the rest, and how loops interact with mapped tasks. At this
point/in this PR, nothing exists as code mind you.

The short "loop or mapped tasks" page is net-new, and is ther mostly
as a jumping off point for "here are the dynamic features you can build
your dag with".

This overview is also why "dynamic mapped tasks" is getting a name
change. Mapping is no longer the only way for a Dag's shape to depend on
data at runtime, and arguably, mapped tasks aren't really that "dynamic"
anymore. "Mapped tasks" is what the UI and the API already call the
result ("Clear All Mapped Tasks", MappedOperator), and it reads next to
"Loops" as the title of the page the overview links to. "Task mapping"
never appears in the UI.

However, please note that the rename is "soft" on purpose. The mapping
page keeps its file name and its dynamic-task-mapping label, and
the old name stays in its opening sentence so existing searches still
land. Provider docs link that page by path through intersphinx so
moving the file would break their builds until each one is updated. The
dynamic-task-mapping capability key that the language SDKs report is
an identifier rather than prose and is left alone.

The snippets are inline here because the example Dags that the guide
will include ship with the authoring API, and a file that calls
.loop() cannot be imported before .loop() exists. That change
replaces them with included copies that a test executes, so the
documented code cannot drift from the behaviour.

The placeholder newsfragment is also here because .loop() is a new
user-visible Dag authoring feature that ships through task-sdk with
airflow-core. It will grow and be adjusted by future PRs in this stack.

@ashb
ashb added this pull request to stack #74276 October 5, 2026 16:35
@ashb
ashb removed this pull request from stack #74276 October 5, 2026 16:36
@ashb
ashb changed the base branch from main to store-historic-ti-ownership-data October 5, 2026 16:36
@ashb ashb changed the title task loops stack 1 Add user-facing docs for the new Task Loops feature Oct 5, 2026
@ashb

ashb commented Oct 5, 2026 •

Copy link
Copy Markdown
Member Author

Turns out you can only have a stack targeting main. So I'll wait to create the rest of the stack until #74222 lands.

@ashb ashb added the skip newsfragment check Skip the newsfragment PR number check label Oct 5, 2026

@TJaniF TJaniF left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Just had a few nits/suggestions and one question. :)

Comment thread airflow-core/docs/authoring-and-scheduling/loops-and-mapped-tasks.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops-and-mapped-tasks.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops-and-mapped-tasks.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Base automatically changed from store-historic-ti-ownership-data to main October 6, 2026 13:40
@ashb
ashb added this pull request to stack #74339 October 6, 2026 13:52
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
@ashb
ashb force-pushed the task-loops-stack-1 branch 2 times, most recently from 6ec5dd6 to 29a26cc Compare October 7, 2026 12:26
@ashb
ashb removed this pull request from stack #74339 October 7, 2026 15:20
@ashb
ashb force-pushed the task-loops-stack-1 branch from 09d1041 to 5ca5192 Compare October 7, 2026 15:22
@ashb
ashb added this pull request to stack #74410 October 7, 2026 15:22
@ashb
ashb marked this pull request as ready for review October 7, 2026 15:27
@ashb
ashb requested a review from amoghrajesh as a code owner October 7, 2026 15:27
@ashb
ashb force-pushed the task-loops-stack-1 branch 4 times, most recently from c89e715 to 2e45493 Compare October 8, 2026 16:06
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
@ashb
ashb force-pushed the task-loops-stack-1 branch 7 times, most recently from edc5fbb to 23eaa02 Compare October 9, 2026 22:18
Task Loops, introduced in AIP-111, are a whole new way for dag authors to thing
Loops chaNge how a Dag author thinks about repeating work, and the change to
implement it reaches storage, the scheduler, the Execution API and the UI. Since
all these changes are relatively large, it is helpful before we introduct those
that reviewers have a mental model of what the behaviour should be, and the
features we want to allow are. Rather than explaining this just in PR messages
or review threads, lets add this as user facing docs, as most of that will be
needed there too.

The new Loops doc covers how a loop decides whether to run another pass, how
a task reads the previous pass's result, what clearing part of a loop does to
the rest, and how loops interact with mapped tasks. At this point/in this PR,
nothing exists as code mind you.

The short "loop or mapped tasks" page is net-new, and is ther mostly as a
jumping off point for "here are the dynamic features you can build your dag
with".

This oVerview is also why "dynamic mapped tasks" is getting a name change.
Mapping is no longer the only way for a Dag's shape to depend on data at
runtime, and arguably, mapped tasks aren't really that "dynamic" anymore.
"Mapped tasks" is what the UI and the API already call the result ("Clear All
Mapped Tasks", MappedOperator), and it reads next to "Loops" as the title of the
page the overview links to. "Task mapping" never appears in the UI.

However, please note that the rename is "soft" on purpose. The mappIng page
keeps its file name and its `dynamic-task-mapping` label, and the old name stays
in its opening sentence so existing searches still land. Provider docs link that
page by path through intersphinx so moving the file would break their builds
until each one is updated. The `dynamic-task-mapping` capability key that the
language SDKs report is an identifier rather than prose and is left alone.

The snippets are inline here because the example Dags that the guide will
include ship with the authoring API, and a file that calls `.loop()` cannot be
imported before `.loop()` exists. That change replaces them with included copies
that a test executes, so the documented code cannot drift from the behaviour.

The placeholder newsfragment is also here because `.loop()` is a new
user-visible Dag authoring feature that ships through task-sdk with
airflow-core. It will grow and be adjusted by future PRs in this stack.
@ashb
ashb force-pushed the task-loops-stack-1 branch from 23eaa02 to dd9cac4 Compare October 10, 2026 07:05
@ashb
ashb merged commit 0c38536 into main Oct 10, 2026
71 checks passed
@ashb
ashb deleted the task-loops-stack-1 branch October 10, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:documentation skip newsfragment check Skip the newsfragment PR number check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants