Skip to content

Resolve live producers by region and read their XCom by attempt - #74344

Merged
ashb merged 1 commit into
task-loops-stack-2from
task-loops-stack-3
Oct 10, 2026
Merged

ashb merged 1 commit into
task-loops-stack-2from
task-loops-stack-3

Conversation

@ashb

@ashb ashb commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Once a loop body or a mapped region can hold several live task instances with
the same task_id and map_index, a consumer can no longer find its producer from
(dag_id, run_id, task_id, map_index) alone, so the lookup has to know where the
caller sits: in the same iteration, in the previous one (what a loop body means
by "the last result"), outside the loop, or in an explicitly named region. When
more than one live candidate still fits we have no sensible option to raise
an error.

Only live (non-archived or superceded) try are candidates, and their data is
read by UUID. An archived try keeps its XCom under its own UUID, so reading by
the resolved try is exact and cannot revive data from work a clear replaced.

Callers that predate regions must see what they saw before, so the default read
scope stays the sentinel region and regional rows appear only when a caller asks
for them. Lookups of earlier runs refuse a non-sentinel region because a region
belongs to a single Dag run, so the producer has to be resolved again for each
run.

The scheduler detected changes to upstream state and tracked map-length
revisions by TaskInstanceKey, which cannot tell two regions apart. One region's
expansion would have marked another's as changed, so both now key on the attempt
and on (task, region).


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@ashb
ashb added this pull request to stack #74339 October 6, 2026 15:06
Comment thread airflow-core/src/airflow/serialization/definitions/xcom_arg.py
Comment thread airflow-core/src/airflow/models/dynamic_region.py
Comment thread airflow-core/src/airflow/models/dagrun.py
Comment thread airflow-core/tests/unit/models/test_dynamic_region.py Outdated
Comment thread airflow-core/tests/unit/models/test_dynamic_region.py Outdated
Comment thread airflow-core/src/airflow/models/xcom.py
@ashb
ashb force-pushed the task-loops-stack-3 branch from c65dc01 to 2cdf742 Compare October 7, 2026 13:57
@ashb
ashb removed this pull request from stack #74339 October 7, 2026 15:20
@ashb
ashb force-pushed the task-loops-stack-3 branch from 2cdf742 to 1624bc8 Compare October 7, 2026 15:22
@ashb
ashb added this pull request to stack #74410 October 7, 2026 15:22
@ashb
ashb force-pushed the task-loops-stack-3 branch from 1624bc8 to 337cba0 Compare October 7, 2026 20:57
@ashb
ashb force-pushed the task-loops-stack-3 branch from 337cba0 to 356e35f Compare October 8, 2026 13:48
@ashb
ashb force-pushed the task-loops-stack-3 branch from 356e35f to 8d326be Compare October 8, 2026 13:53
@ashb
ashb marked this pull request as ready for review October 8, 2026 13:55
@ashb
ashb force-pushed the task-loops-stack-3 branch from 8d326be to 1b3d613 Compare October 8, 2026 16:06
Comment thread airflow-core/tests/unit/models/test_dynamic_region.py
Comment thread airflow-core/tests/unit/models/test_dynamic_region.py Outdated
Comment thread airflow-core/src/airflow/models/dagrun.py Outdated
Comment thread airflow-core/tests/unit/models/test_xcom_arg.py Outdated
Comment thread airflow-core/src/airflow/models/xcom.py
@ashb
ashb force-pushed the task-loops-stack-3 branch 2 times, most recently from 6bf6974 to a67defe Compare October 9, 2026 10:32
@ashb
ashb force-pushed the task-loops-stack-3 branch from a67defe to e650913 Compare October 9, 2026 14:03
@ashb
ashb force-pushed the task-loops-stack-3 branch from e650913 to 27ce78f Compare October 9, 2026 16:28
@ashb
ashb force-pushed the task-loops-stack-3 branch from 27ce78f to 7ab99d4 Compare October 9, 2026 20:34
@ashb
ashb force-pushed the task-loops-stack-3 branch 2 times, most recently from 832e405 to b419bda Compare October 9, 2026 22:18
Once a loop body or a mapped region can hold several live task instances with
the same task_id and map_index, a consumer can no longer find its producer from
(dag_id, run_id, task_id, map_index) alone, so the lookup has to know where the
caller sits: in the same iteration, in the previous one (what a loop body means
by "the last result"), outside the loop, or in an explicitly named region. When
more than one live candidate still fits we have no sensible option to raise
an error.

Only live (non-archived or superceded) try are candidates, and their data is
read by UUID. An archived try keeps its XCom under its own UUID, so reading by
the resolved try is exact and cannot revive data from work a clear replaced.

Callers that predate regions must see what they saw before, so the default read
scope stays the sentinel region and regional rows appear only when a caller asks
for them. Lookups of earlier runs refuse a non-sentinel region because a region
belongs to a single Dag run, so the producer has to be resolved again for each
run.

The scheduler detected changes to upstream state and tracked map-length
revisions by TaskInstanceKey, which cannot tell two regions apart. One region's
expansion would have marked another's as changed, so both now key on the attempt
and on (task, region).
@ashb
ashb force-pushed the task-loops-stack-3 branch from b419bda to 2dda9b4 Compare October 10, 2026 07:05
@ashb
ashb merged commit 2f7d966 into main Oct 10, 2026
132 of 140 checks passed
@ashb
ashb deleted the task-loops-stack-3 branch October 10, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants