You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 020bdcc
Browse filesBrowse the repository at this point in the historyBrowse files
Describe the all-handlers probe in ADR-0011 and ADR-0012
Both ADRs still had the request name the file's Dags and the runtime
omit the rest. SDK authors and later layers read them as the spec for
what a runtime answers, so they must say the answer depends only on the
artifact.
The parse owns validation, not an importer. `PythonDagImporter` returns `airflow.sdk.DAG` objects and knows nothing about coordinators or queues, so `@task.stub` keeps working for
179
-
any importer that can produce a Dag carrying stub tasks. `_parse_file` is also the only place where the whole file's Dags are visible at once, which is what lets one request cover
180
-
every `dag_id` that resolved to the same artifact ([ADR-0012](0012-lang-sdk-parse-protocol.md)).
178
+
any importer that can produce a Dag carrying stub tasks. `_parse_file` is also the only place where the whole file's Dags are visible at once, which is what lets one request per
179
+
artifact serve every Dag in the file whose stubs resolved to it ([ADR-0012](0012-lang-sdk-parse-protocol.md)).
181
180
182
181
Resolution goes through the coordinator registry, not the filesystem, so the Python Dag and the Lang-SDK artifact **do not need to be in the same DagBundle**. Nothing here needs an
183
182
`airflow.sdk.DAG` round-trip either — validation compares against the Dag the Python parser already built. Appendix B states exactly what is compared.
@@ -187,7 +186,7 @@ Resolution goes through the coordinator registry, not the filesystem, so the Pyt
187
186
| Caller | Coordinator call | What comes back | Action |
|`_parse_file` → `PythonDagImporter`| — (the Python file is parsed in process) | its own parsed Dags | PERSIST |
190
-
|`_parse_file`, per (coordinator, artifact) |`parse_task_handler`, scoped to that group's dag_ids|`TaskHandlerParsingResult`| VALIDATE only — not a Dag, so nothing to persist |
189
+
|`_parse_file`, per (coordinator, artifact) |`parse_task_handler`, for every handler it registers|`TaskHandlerParsingResult`| VALIDATE only — not a Dag, so nothing to persist |
191
190
|`_parse_file` → `JavaDagImporter`|`parse_dag`|`DagFileParsingResult`, native Dags only | PERSIST |
192
191
193
192
There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — and nothing reading a `DagImporter`'s results — ever sees one.
@@ -235,7 +234,8 @@ with one `bundle.serve()`. One artifact can hold both, so neither the file nor t
235
234
236
235
### Appendix B — What is compared
237
236
238
-
Handler declarations from every process spawned in Step 4 are unioned per `dag_id` before comparison, since one Dag's stubs can target several queues.
237
+
Handler declarations from every process spawned in Step 4 are unioned per `dag_id` before comparison, since one Dag's stubs can target several queues. Each reply covers
238
+
every Dag its artifact registers handlers for; only the parsed file's Dags are compared.
239
239
240
240
-`task_id` sets must match exactly. A missing or extra handler is an error.
241
241
-`arg_bindings` against `params` as each declaration's `binding` says: by position for `positional`, by name for `named`, `named_or_whole` and `named_open`, compared
Copy file name to clipboardExpand all lines: airflow-core/adr/lang-sdk/0012-lang-sdk-parse-protocol.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,8 +26,8 @@ Proposed
26
26
## Context
27
27
28
28
The Dag processor asks a Lang-SDK runtime two different questions. "Which Dags does this artifact define?" is answered over the messages [ADR-0004](0004-dag-parsing.md) already
29
-
defines. "Which task handlers does this artifact registerfor a `dag_id` Python already owns?" has no answer in those messages, because a `TaskHandler` registration carries no Dag
defines. "Which task handlers does this artifact register, each for a `dag_id` Python already owns?" has no answer in those messages, because a `TaskHandler` registration carries
30
+
no Dag ([ADR-0011](0011-mixed-language-dag-processing.md)).
31
31
32
32
This ADR defines the request that carries the second question, the subprocess classes that carry both, and the two parse-side entry points on the coordinator.
33
33
@@ -82,7 +82,6 @@ relay is only type-safe because both hops speak the same pair, which is the reas
82
82
```
83
83
class TaskHandlerParseRequest:
84
84
file: str # the artifact resolved for this coordinator
85
-
dag_ids: list[str] # every Dag in the parsed file with stub tasks that resolved here
86
85
bundle_path: Path
87
86
bundle_name: str
88
87
type: Literal["TaskHandlerParseRequest"]
@@ -106,9 +105,10 @@ class TaskHandlerParam:
106
105
exact_name: bool = False # match as spelled, not case-insensitively with underscores ignored
107
106
```
108
107
109
-
One request carries every `dag_id` that resolved to the same artifact under the same coordinator, so a file whose stubs all target one runtime costs one process. A `dag_id` the
110
-
artifact registers nothing for is **omitted** from `task_handlers` rather than returned empty: the key set is not required to match `dag_ids`, because it is the union across
111
-
coordinators that has to cover the stubs ([ADR-0011](0011-mixed-language-dag-processing.md)).
108
+
The request names no Dags. The runtime answers with every task handler the artifact registers, keyed by `dag_id`, and with `{}` when it registers none. The answer must depend only
109
+
on the artifact, never on the request, so the Dag processor can cache it per artifact ([ADR-0013](0013-persisted-task-handler-bindings.md)). One request per (coordinator, artifact)
110
+
pair serves every Dag in the parsed file whose stubs resolved there, so a file whose stubs all target one runtime costs one process. The key set is not required to match the file's
111
+
Dags: it can hold other files' Dags, and it is the union across coordinators that has to cover the stubs ([ADR-0011](0011-mixed-language-dag-processing.md)).
112
112
113
113
`value_schema` reuses the `ArgValueSchema` definition `arg_bindings` already carries ([ADR-0007](0007-taskflow-across-language-boundary.md)), so both sides of a comparison are the
114
114
same type. Two properties matter to validation: the field is nullable on both sides, and each declaration names its binding mode. Appendix B says what that forces.
0 commit comments