Repository navigation
Let a Dag importer find the coordinator that parses its files - #74042
Merged
Merged
Conversation
This was referenced Oct 1, 2026
jason810496
added this pull request to stack #74044
October 1, 2026 15:18
kaxil
reviewed
Oct 1, 2026
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
2 times, most recently
from
October 2, 2026 04:07
107f1eb to
e3fc440
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
from
October 2, 2026 05:44
e3fc440 to
3018722
Compare
1 task done
pierrejeambrun
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
from
October 2, 2026 09:00
3018722 to
d21aea9
Compare
1 task done
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
from
October 2, 2026 11:41
d21aea9 to
42c00d4
Compare
jason810496
marked this pull request as ready for review
October 2, 2026 12:28
jason810496
requested review from
amoghrajesh,
ashb and
uranusjr
as code owners
October 2, 2026 12:28
jason810496
commented
Oct 2, 2026
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Thanks for the reivew.
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
from
October 2, 2026 14:07
42c00d4 to
63a5cd3
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
from
October 2, 2026 15:09
63a5cd3 to
b830155
Compare
jason810496
removed this pull request from stack #74044
October 3, 2026 17:58
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
from
October 3, 2026 17:58
b830155 to
b9fcd45
Compare
jason810496
added this pull request to stack #74170
October 3, 2026 17:59
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
from
October 4, 2026 01:06
b9fcd45 to
3dfc4d4
Compare
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
from
October 4, 2026 08:51
3dfc4d4 to
1e177e6
Compare
Base automatically changed from
jason/lang-sdk-e2e/03b-serialized-dag-validation
to
main
October 5, 2026 03:38
A CoordinatorDagImporter names its coordinator class in coordinator_classpath and finds that coordinator in [sdk] coordinators. With several coordinators of the class, the new optional [sdk] dag_bundle_to_coordinator picks the one for a Dag bundle. A Dag bundle's importer registry registers the importers listed in COORDINATOR_DAG_IMPORTERS that have a coordinator. SubprocessCoordinator.parse_dag replaces the calling process with the runtime that parses one Dag file, and the runtime gets the resolved [api] base_url, [operators] default_deferrable and [triggerer] queues_enabled in its environment.
jason810496
force-pushed
the
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
branch
2 times, most recently
from
October 5, 2026 05:20
aef2336 to
6fd1704
Compare
uranusjr
reviewed
Oct 5, 2026
uranusjr
reviewed
Oct 5, 2026
uranusjr
approved these changes
Oct 5, 2026
uranusjr
left a comment
Member
There was a problem hiding this comment.
Two minor points needing resolution. Generally looks good to me.
Python's signal module never exposes SIGXFZ, so the entry was a no-op. Co-Authored-By: Claude <noreply@anthropic.com>
A configured coordinator that cannot be loaded is skipped, while a classpath from code that cannot be imported raises ImportError. Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom to top): #74042, #74035, #74043, #74036, #74037, #73845, #73846, #73847
Why
To parse native Dags, the Dag importer that claims a file has to find the coordinator that parses it. It looks in
[sdk] coordinators, so a runtime is still declared in one place.What changes
CoordinatorDagImporteris the base class for a runtime's Dag importer. A subclass setscoordinator_classpath, andget_parsing_coordinator()returns the coordinator of that class that parses its files in its Dag bundle.[sdk] dag_bundle_to_coordinatormust map the bundle to one of them, orget_parsing_coordinator()raisesInvalidCoordinatorError.DagImporterRegistry.from_config(bundle_name)registers each importer inCOORDINATOR_DAG_IMPORTERSthat has a coordinator. The tuple is empty until Parse native Java Dags and show their source #74036 and Parse native TypeScript Dags with the Node coordinator #74037 add the Java and Node importers.SubprocessCoordinator.parse_dag(...)replaces the calling process with the runtime that parses one Dag file.[api] base_url,[operators] default_deferrableand[triggerer] queues_enabledin its environment, like the log levels.With two
JavaCoordinators,jdk-11andjdk-17, tasks still follow their queue and the new option picks which one parses each Dag bundle:Was generative AI tooling used to co-author this PR?