Repository navigation
Probe a Lang-SDK artifact for its task handlers - #73974
Draft
jason810496 wants to merge 11 commits into
Draft
jason810496 wants to merge 11 commits into
jason810496 wants to merge 11 commits into
Conversation
jason810496
added this pull request to stack #73978
September 30, 2026 17:59
This was referenced Sep 30, 2026
This was referenced Sep 30, 2026
jason810496
force-pushed
the
jason/core-taskhandler-refactor/05-runtime-parse-transport
branch
from
October 1, 2026 06:26
ff05737 to
99e8fa5
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/05-runtime-parse-transport
branch
2 times, most recently
from
October 1, 2026 10:55
11b1aa6 to
0b77a65
Compare
This was referenced Oct 1, 2026
jason810496
force-pushed
the
jason/core-taskhandler-refactor/05-runtime-parse-transport
branch
from
October 1, 2026 14:44
0b77a65 to
f6918d8
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/05-runtime-parse-transport
branch
from
October 2, 2026 00:54
f6918d8 to
7a8bab8
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/05-runtime-parse-transport
branch
from
October 2, 2026 05:48
7a8bab8 to
69e2b80
Compare
1 task done
jason810496
force-pushed
the
jason/core-taskhandler-refactor/05-runtime-parse-transport
branch
from
October 2, 2026 11:36
69e2b80 to
b45c6ec
Compare
This was referenced Oct 3, 2026
A supervised child reopens stdin, stdout and stderr on its sockets and replaces sys.stdin and friends. When that child supervises a child of its own, as a Dag-parsing child starting a task-handler probe will, the grandchild replaces handles that already wrap fds 0-2, and dropping them closed the fds it had just dup2'd, so the grandchild lost its streams. The handles no longer own their fds; the interpreter's original ones are kept alive by sys.__stdin__ and friends and were never closed here.
The Dag processor will ask a coordinator's runtime which task handlers an artifact registers (ADR-0011, ADR-0012). SubprocessCoordinator.parse_task_handler replaces the calling process with that runtime, which _build_parse_task_handler_command builds, taking parse_dag's steps and helpers (#74042) in the same order. On Linux the runtime also gets PR_SET_PDEATHSIG, so it dies with the process that reads its sockets, which the manager kills by pid when a Dag-parsing child times out. BaseCoordinator gains nothing: only a subprocess coordinator can answer.
The parent death signal is only sent for a parent that exits after it is set. A parent that exited between the schema-version report and the prctl call left the runtime with no signal to come and nobody to read its sockets. The child now checks that its parent is unchanged once the signal is set.
The Dag processor will probe the artifact that a stub task actually runs, so a check sees what the worker will run. _find_task_handler_artifact finds it for a Dag the way execute_task does. The coordinators of each SDK implement it in later changes.
A runtime built before task handler parsing cannot answer the request, so starting it only produces a failure. TASK_HANDLER_PARSING_SCHEMA_VERSION is the in-progress supervisor schema version, as native's Java Dag-parsing gate uses, because contributors never add a version date. parse_task_handler refuses an older runtime, and the Dag processor will skip such artifacts before probing.
The Dag-file parse will ask a coordinator's runtime which task handlers an artifact registers (ADR-0011, ADR-0012). LangSDKTaskHandlerProcessorProcess sends TaskHandlerParseRequest and collects one TaskHandlerParsingResult. A failed start, a missing result, an invalid frame or message, or a timeout is an import error on it. Nothing calls it yet. It is written like LangSDKDagFileProcessorProcess (#74035) and reuses its schema-version message and import timeout, so #74058 can move both onto one transport. It runs in a Dag-parsing child, which has no API client, so the runtime's requests are relayed up that child's supervisor channel. The runtime's channel takes ToManager messages only, and only this class handles TaskHandlerParsingResult.
The frame reader reads a frame's payload in the same callback as its length header. On the blocking comm socket, a runtime that stopped after the header blocked the loop in that read, so the import timeout never fired and the runtime was never killed. Reads now return at once and the reader completes the frame on a later callback; replies still go out on the blocking socket.
A process the runtime started can keep its output open after the runtime exits, so the probe does not finish and that process keeps running. As in LangSDKDagFileProcessorProcess (#74035), the grace kill after the result, the timeout kill and close() now kill the runtime's process group once it has exited, unless its pid was reused.
Only the fork itself was guarded. A failure while registering the listeners or sending the start request left the child waiting for that request and both listeners open, and the standalone run does not guard start() either.
The Dag-file parse will probe its artifacts one after another, and the manager kills the parse child at dag_file_processor_timeout. A probe bounded only by its import timeout could outlast the child, and the parse would send no result at all. run() now also stops at a deadline the caller passes: a probe still running then is killed, and its result is an import error unless the runtime had already answered.
… ADR-0013 ADR-0012 still named BaseParsingProcess, SDKTaskHandlerProcessorProcess and its entry point, put both parse verbs on BaseCoordinator, ran both Lang-SDK processes in a Dag-parsing child, and described the coordinator as forwarding bytes. It now names the classes the code defines, puts the verbs and hooks on SubprocessCoordinator, and says the runtime connects back to the process that asked. ADR-0013's probe step uses the new names.
jason810496
force-pushed
the
jason/core-taskhandler-refactor/05-runtime-parse-transport
branch
from
October 5, 2026 18:12
b45c6ec to
4a12877
Compare
jason810496
removed this pull request from stack #73978
October 6, 2026 02:27
jason810496
added this pull request to stack #74318
October 6, 2026 02:35
1 task done
jason810496
force-pushed
the
jason/core-taskhandler-refactor/05-runtime-parse-transport
branch
from
October 6, 2026 02:44
4a12877 to
6129fc1
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom to top), on native #74037 (stack #74170): #73973, #73974, #73975, #74317, #73976, #74067, #74135, #74140, #74141, #73971, #74030, #74031, #74032, #74136, #74137, #74138, #74139
To check stub tasks, the Dag processor has to find the artifact a worker would run, start its runtime inside the Dag-parsing child, and ask it with #73973's messages. This PR adds that probe and the two coordinator hooks it needs. Nothing calls it yet: #73975, #74317 and #73976 implement the hooks for Go, Java and TypeScript, and #74135 calls the probe.
SubprocessCoordinatoronly, andBaseCoordinatoris unchanged. A coordinator without them gives an import error that says so.2026-10-30or later is probed. That is the in-progress version, so no new date is invented.LangSDKTaskHandlerProcessorProcessis a standalone process class beside Parse native Lang-SDK Dags in the Dag processor with their runtime #74035'sLangSDKDagFileProcessorProcess, with no API client. A failed start, a missing or invalid result, or a timeout is an import error keyed by the artifact's path in its bundle.ToManager, where aDagFileParsingResultis an unhandled request, so a Dag registration is never read as a task handler.PR_SET_PDEATHSIGon Linux, and what it leaves in its process group is killed on exit or timeout.Known limits
import airflow. The check then logs a warning and leaves the stub tasks unchecked. Linux is not affected.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines