Repository navigation
Move Dag cycle detection from the Task SDK to shared - #74034
Conversation
jason810496
left a comment
There was a problem hiding this comment.
The current static check failure is not related and will be resolved once #73963 (review) finish.
541001b to
e5ae872
Compare
Cycle detection only runs for Dags that go through the Python SDK's DAG object. Dags authored with non-Python language SDKs arrive at core already serialized and never pass through that object, so nothing rejects a cycle in them today. Core cannot reuse the SDK implementation as-is: it is a method on the authoring class and reaches for task objects, while core holds raw serialized data it should not have to hydrate just to run a check. Moving the traversal into the shared distribution both distributions already depend on, and describing the graph through callbacks instead of node objects, lets core add that check against the serialized form in a follow-up without duplicating the algorithm or inverting the dependency by importing the SDK's exception.
The reversed-edge test passed or failed with the acyclic cases, so it is replaced by a cyclic case whose node id is empty. It fails if the traversal treats an empty id as no child. Co-Authored-By: Claude <noreply@anthropic.com>
e5ae872 to
2060349
Compare
|
From what I can tell this does not make anything in core call the shared code yet. Is this planned to be done in a later PR? (The PR title probably should be adjusted slightly if that’s the case.) |
There was a problem hiding this comment.
LGTM.
From what I can tell this does not make anything in core call the shared code yet. Is this planned to be done in a later PR? (The PR title probably should be adjusted slightly if that’s the case.)
I updated the description which is misleading at this point. Core calling this is done in #74041 I believe and then wired in the dag processor in #74035 (by calling validate_serialized_dag)
Stack (bottom to top): #74034, #74040, #74041, #74042, #74035, #74043, #74036, #74037, #73841, #73845, #73846, #73847
main.Why
Move the Cycle detection part to share so that all the further SDKs native Dag can be validated by the same single source.
How
Core cannot reuse
DAG.check_cycleas it is: it is a method on the authoring class and reads task objects, while core holds serialized data it should not hydrate just to run this check.shared/dagnode, whichtask-sdkandairflow-corealready depend on.AirflowDagCycleExceptionlives in the Task SDK, and importing it here would invert the dependency, so each caller raises its own error.Was generative AI tooling used to co-author this PR?