Repository navigation
Add ADR for Lang SDK Dag and mixed-language Task processing flow - #71929
Merged
jason810496 merged 9 commits intoSep 29, 2026
Merged
jason810496 merged 9 commits into
jason810496 merged 9 commits into
Conversation
jason810496
force-pushed
the
feature/lang-sdk/adr-0007-mixed-language-dag-processing
branch
from
August 21, 2026 11:04
6324a01 to
ca8ebdd
Compare
jason810496
requested review from
Lee-W,
amoghrajesh,
ashb,
kaxil,
phanikumv and
uranusjr
August 21, 2026 11:10
phanikumv
reviewed
Aug 21, 2026
uranusjr
reviewed
Aug 23, 2026
jason810496
commented
Aug 24, 2026
Member
Author
There was a problem hiding this comment.
Here're the feedbacks suggested by TP from the offline sync:
- The Lang SDK user interface should be different for the mixed Lang Dag case (e.g.
ExternalDag,MixedLangDagorTaskBuilderinstead ofDagBuilderetc.) - Don't treat the mixed Lang Dag artifact same as the native Dag atifact.
- Instead of introducing the
is_mixed_languageDag-level argument, we need to adjust theDagFileParsingRequestto make the runtime subprocess propagate the mixed Lang Dag only list.
jason810496
marked this pull request as draft
August 24, 2026 08:43
jason810496
marked this pull request as ready for review
August 25, 2026 08:28
jason810496
force-pushed
the
feature/lang-sdk/adr-0007-mixed-language-dag-processing
branch
from
August 25, 2026 08:31
65bff0b to
72aaf57
Compare
jason810496
commented
Aug 25, 2026
jason810496
left a comment
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Hi @uranusjr,
I just refined the ADR based on the our discussion yesterday.
- Replace the Dag-level
is_mixed_language_dagby introducing differences at the user authoring interface level (@Dagvs@MixedLangDagin the case of Java SDK) - Add
mixed_language_dags_onlyfield on theDagFileParseRequest - Replace the
JavaDagImporterfor the mixed lang with the existing bundle discovery method plus theJavaCoordinator.run_dag_parsing
Please let me know WDYT when you have a moment, thanks.
1 task done
jason810496
marked this pull request as draft
September 8, 2026 07:51
jason810496
force-pushed
the
feature/lang-sdk/adr-0007-mixed-language-dag-processing
branch
from
September 13, 2026 14:34
72aaf57 to
0220ce7
Compare
jason810496
changed the title
jason810496
marked this pull request as ready for review
September 14, 2026 06:10
Member
|
0008 is already taken now. We’ll need to update the PR to use the next free number. Currently 0010 would be the most natural sine 0009 is also already taken. |
This was referenced Sep 28, 2026
A Lang-SDK artifact backing `@task.stub` tasks used to author a Dag under the same dag_id the Python file already owns, so two conflicting definitions reached persistence and something downstream had to pick between them. Removing the Dag from the authoring interface answers that once, rather than leaving every consumer of a serialized Dag to ask whether the Dag in front of it is real. Terms follow the Language SDK spec so the decision reads the same in Go, Java, and TypeScript, and the parse-side contract is grounded in the shipped coordinator and supervisor-schema code rather than in a proposed shape.
…nics An ADR is read first to understand what was decided and why, and only later to build the thing. Exact message shapes, call sites, and the code that has to change were interleaved with that reasoning, so neither audience could skim its half. The decisions now stand on their own and the mechanics wait in an appendix.
The single document answered three questions with different audiences and different review cycles: the wire protocol a runtime implements, how a Lang-SDK Dag source reaches the Dag processor, and how a Python stub is validated against the handler behind it. Readers had to take all three to act on one, and a change to any one of them forced the other two back through review.
The diagrams carry the decision faster than the prose around them did, so they belong where a reader lands rather than behind an appendix. The argument for each choice is still worth keeping, but it is what someone reads second. Coordinator lookup gains a named answer: for_bundle, beside the existing for_queue, rather than an open question about how a registry finds the coordinators serving a bundle.
…oint An earlier draft gave task-handler parsing its own message pair on the theory that the runtime answered the coordinator. It does not: the coordinator forwards bytes and decodes nothing, so the peer is whichever process started the parse. Two reply unions would have meant two decoder configurations and two relay paths for identical traffic. Validation likewise had no home. It needs the parsed Dags and the argument bindings that only exist after serialization, and the only place holding both is the Dag-file parse itself — not an importer, which should not have to know what a coordinator or a queue is. Making a coordinator's importer optional restated in a return type what the artifact source already decides, leaving two answers free to disagree.
A reader meeting native Dag processing and mixed-language handlers for the first time needs to know what each one decides before the message shapes and subprocess classes that carry them mean anything. The parse protocol is the mechanics both decisions share, so it reads better after them than in front of them.
jason810496
force-pushed
the
feature/lang-sdk/adr-0007-mixed-language-dag-processing
branch
from
September 29, 2026 01:33
af9fe9c to
f684ee0
Compare
Member
Author
I just updated the numbers, thanks. |
uranusjr
reviewed
Sep 29, 2026
uranusjr
reviewed
Sep 29, 2026
jason810496
commented
Sep 29, 2026
jason810496
left a comment
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Thanks TP for the review and the discussion.
henry3260
approved these changes
Sep 29, 2026
uranusjr
approved these changes
Sep 29, 2026
This was referenced Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
related: AIP-108, AIP-85 Design only, no code changes.
Design how the DagImporter interface and the Coordinator interface works together for the native Dag processing.
Design how the mixed language task validation works in the Dag processing path.
Was generative AI tooling used to co-author this PR?