Repository navigation
Conversation
dabla
force-pushed
the
feature/xcom-iterable-flatten
branch
from
September 27, 2026 18:54
bf1b5a4 to
a3e1e15
Compare
Contributor
Author
|
Interaction with #70223 to handle when both land: |
1 task done
dabla
force-pushed
the
feature/xcom-iterable-flatten
branch
15 times, most recently
from
October 2, 2026 07:52
e1ca970 to
bfa2dc4
Compare
dabla
force-pushed
the
feature/xcom-iterable-flatten
branch
4 times, most recently
from
October 8, 2026 18:15
772416a to
980397c
Compare
The success and skip callbacks fired from IndexedTaskRunner.__exit__, as soon as the operator returned and before the indexed task's checkpoint and result were written. A checkpoint write that failed had already announced a success, and the retry ran the indexed task again and announced it a second time; the callback also found no end_date on the task instance. A plain task fires its callback after the push and the state report, with end_date set. The exit now only notes a failure. IterableOperator._run_task calls the runner's report_success() or report_skip() once the matching checkpoint is written, where the indexed task's code ran (its worker thread for a sync operator, the loop for an async one); both set end_date and the state before the callback. A result push that fails after the checkpoint is replayed from it, so the callback fires once. The docs page and the class docstring say so. test_the_success_callback_waits_for_the_checkpoint and the two runner report tests fail before.
…valuating the policy again With several failed indexed tasks whose retry policy decisions were all the default, or whose evaluation raised, no exception outweighed the others, the group was handed to the runner and no decision was kept for it. The callbacks then evaluated the policy once more, on the BaseExceptionGroup rather than on any indexed task's exception: K+1 evaluations instead of K, and for a policy that answers differently per call a retry announced here that the runner's own evaluation of the group could refuse. The once-per-item test did not reach this path because its policy answers retry(), so an exception was always chosen. _failure_for_the_runner now keeps the default decision for the group when no decision outweighs it, so _task_will_retry follows it and falls back to retry eligibility without evaluating the policy again. test_an_undecided_group_is_not_evaluated_again_for_the_callbacks fails before with four evaluations.
…the iteration on_kill() reaches the sub-operators that have started, and the stop flag keeps the executor from pulling more. An indexed task instance is neither until its runner starts: on a retry attempt it first reads its checkpoint, and a kill that lands during that read found it unregistered, so it started afterwards, ran to completion unkilled, its remote work included, and the drain waited for it before the task could conclude as terminated. IterableOperator._run_task now asks the iteration state whether a stop was requested once the checkpoint replay is done and before the runner is built, and returns IndexedTaskInstanceNotStarted as the outcome: no code run, no checkpoint, no callback, and the next attempt runs it. IndexedTaskOutcomes counts these apart from the indexed tasks that ran and the kill message names them. A sync indexed task is still handed to the pool afterwards, whose threads are as many as the calls in flight, so only that pickup remains. The docs page and the class docstring list it with the other outcomes that fire no callback. test_an_item_pulled_before_the_kill_but_not_started_does_not_run fails before with two of three indexed tasks run; the outcomes unit test covers the count.
The static checks fail on D401 for IndexedTaskInstance.context_for: its summary line named what the method returns instead of saying what it does.
The message claimed "the rest never pulled" also when every item had been pulled, and said the input was being resolved for a kill that landed before the run started, where nothing was. IndexedTaskOutcomes._killed_message now names the items that ran, those pulled before the kill and never started and those never pulled, each only when the count is not zero, and a kill before the input was resolved says so. test_a_kill_after_every_item_was_pulled_claims_no_remainder fails before.
…puts compare by it
…nt's execution_timeout
… would leave taken
… task instance's identity
The Source docstring still described a scalar as read as a one-item sequence, which from_argument stopped doing when literal scalars were refused at parse time.
The IterableOperator docstring carries both warnings, but the task SDK docs do not render the API reference, so the page is the only place users read them. The async calls get a sync-to-async lookup table, the comparison table gains rows for SDK calls and pools, and the warnings after the example now come before the notes.
An item's own AirflowTaskTimeout next to another item's failure reaches the runner as a BaseExceptionGroup, which it retries like any failure, but the failed items' callbacks reported a failure because the group is not an Exception. A group now counts as retryable when each of its members is.
Cancelling an asend parked on its response released the thread lock while the read thread was still in recv, or had not started, so the next send could read the cancelled request's response as its own. The request now runs in one worker thread that takes the lock, writes, reads and releases it; a request abandoned before it was written is dropped. This replaces the acquire hand-off, and takes one thread hop per request instead of two.
A .map() callable and the template rendering ran while the executor pulled the next item, so what they raised ended the whole iteration: a skip skipped the task, and a failure stopped every later element on each retry. Each item now reads its own input inside its run, so what reading it raises is that item's outcome, as under .expand().
xcom_arg now imports the loop module, which imports BaseOperator, so a module-level import of XComArg in expandinput closes an import cycle through mappedoperator and fails airflow.sdk on import. The module keeps it under TYPE_CHECKING and imports it where it is used, as main does.
An iteration shares the parent's identity, but its region was left to the defaults, so inside a loop it reported region 0 and its own map index instead of the parent's region coordinates.
…ence Inside a loop, a lazy XCom sequence can stand for the previous iteration's values, and its synchronous reads ask for them, but the async item read and count left the flag out, so an async consumer got the current iteration's values instead.
The request frame is a bytearray, so the exchange helpers take bytes or bytearray; and the parent task instance is typed by a protocol without the region fields, so they are read with getattr, as the server context next to them is.
…quence An iterated task whose iterations each return a page of items (a list of rows per API call) pushes one XCom per iteration and returns the XComIterable over them. flatten() gives the downstream task those pages as one Sequence of items: nested lists, tuples and sets are expanded recursively, strings, bytes and scalars stay single items. FlattenedXComIterable speaks in flattened positions. Its length is the flattened_length that IterableOperator.axcom_push tallies from each value as it is pushed, so the view knows it without reading a page; an XComIterable pushed without the tally is counted by reading its pages once. Reads hold the last page fetched and nothing more: a sequential read fetches every page exactly once, a read within the held page costs nothing, and a jump backwards restarts from the first page, so memory stays bounded by one page. Split out of the task iteration PR, which does not need it, so it can be reviewed and documented on its own.
dabla
force-pushed
the
feature/xcom-iterable-flatten
branch
from
October 12, 2026 05:35
c82f05a to
64abc2c
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #62922 (Task Iteration): only the last commit, c490ee3, belongs to this PR. It will be rebased once #62922 merges.
related: #62922
What
An iterated task whose iterations each return a page of items, for example one list of rows per API call, pushes one XCom per iteration and returns the
XComIterableover them..flatten()on that output gives a downstream task those pages as oneSequenceof items:Nested lists, tuples and sets are expanded recursively; strings, bytes and scalars stay single items.
How
FlattenedXComIterablespeaks in flattened positions. Its length is theflattened_lengththatIterableOperator.axcom_pushtallies from each value as it is pushed, while the value is in memory, so the view knows its length without reading a page. AnXComIterablepushed without the tally is counted by reading its pages once.__getitem__/__iter__(blocking reads) andaget/__aiter__(reads throughasend, safe on the task's event loop).Why a separate PR
This was part of #62922, but task iteration does not need it: nothing in the SDK outside
bases/xcom.pyreferred to it, and it had no documentation there. Splitting it out keeps #62922 about running a task over its input, and gives flatten its own review and its own paragraph in the task-sdk docs, included here.Tests
test_xcom.pycovers expansion of lists, tuples, sets, generators and mixed pages, string/bytes/scalar pass-through, negative indices on both paths, the tallied length round-tripping through serde, one page read per page on a sequential read, the no-tally counting pass, and the one-page memory bound.test_serde.pyround-trips the flattened view.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code following the guidelines
🤖 Generated with Claude Code