Repository navigation
Commit a8a6257
authored
TS SDK: embed one source region per native Dag file (#73723)
* TS SDK: embed one source region per native Dag file
A bundle whose native Dags live in more than one file only carried its
entrypoint's source in the airflowSource region. The Code tab would show
the entrypoint for every Dag, which is the file for the ones declared
there and unrelated for the ones declared in imported modules. Multi-Dag
bundles were effectively single-source in the UI.
Every Dag file that declares at least one native Dag now ships its own
source region, mapped to its Dags by dag_id. The `Dag` constructor
records which file it ran in via a cross-realm global slot that
`airflow-ts-pack`'s onLoad plugin writes at the top of each author-owned
source file. `Function.prototype.toString` and stack-trace parsing stay
out — ES module hoisting is what makes the slot the current file for
every top-level `new Dag(...)` call, and the constructor just reads it.
* Rename sourceFiles, drop the entrypoint fallback
Per-file source-size cap, back to the same 1 MB limit each source region
already had. The bundle grows with the number of Dags, not with any
aggregate cap on their sum.
`EMBEDDED_SOURCE_OPEN_PREFIX`/`_SUFFIX` collapse to a single
`EMBEDDED_SOURCE_MARKER`; the trailing newline is a formatting detail
of the concatenation, not a named constant.
`entrypointName` and its `--source` CLI option are gone. The metadata
`entrypoint` field they filled is only useful as a fallback for
"no per-Dag source", and each native Dag now carries its own source path.
Bundles that serve nothing still fail at pack time; mixed-lang bundles
still work with an empty `dag_source_paths`.
* Reject block-comment terminators in source paths, and utf-8 the header
A source path holding `*/` would close the marker's own block comment
early, splicing the rest of the region into executable position; the
packer now rejects such paths (and stray newlines) rather than trying
to escape them.
`renderHeader` was writing the layout line with `Buffer.from(..., "ascii")`,
which silently truncated a non-ASCII path like `données.ts` and broke
the metadata-to-region round trip. Encode the header as utf-8.
Drops the whitespace/syntax/keepNames minify split back to plain
`minify: true`: nothing in this PR depends on identifier preservation,
so the earlier split needed a rationale it did not have.
* Save and restore the module-source slot around each module body
The tag used to only write. When A imported B, B's write overwrote A's,
and any `new Dag(...)` in A after the import was attributed to B. The
tag now captures the previous value on entry and restores it after the
module's top-level code returns, so nested imports and top-level `await`
within a single module hand the slot back on the way out.
Two scoping limits remain and are documented in module-source.ts: a Dag
constructed in a scheduled callback (setTimeout, microtask) fires after
the module's epilog has restored, so it lands under whichever module
touched the slot most recently; and two modules genuinely evaluated in
parallel through Promise.all([import, import]) can interleave. Both are
non-standard authoring patterns.
* Read the multi-source TypeScript bundle layout in the Task SDK
The airflow-ts-pack encoder now writes a `sources` array — one embedded region
per native Dag file, keyed by `dag_source_paths`, and empty for a bundle whose
Dags are all authored in another language — in place of the single `source`
section. The Task SDK bundle reader still demanded that singular section, so
every bundle the current packer produces was rejected with "bundle layout is
missing the source section" and no TypeScript task could launch. Teach the
reader only the new layout; no released bundle uses the old one, so this is a
clean break rather than carrying both formats.
* Embed the entrypoint source as a per-Dag fallback
With one source region per native Dag, a Dag the packer cannot attribute to a
file — one constructed dynamically in a callback, or a mixed-language Dag owned
by Python — had no embedded source at all, so reading its source failed. Restore
the earlier guarantee that the entry file is always embedded: record its path as
entrypoint_path in the metadata and pack its contents as a source region, so a
reader falls back to it whenever a Dag has no attributed file of its own.
* Return no source for a Dag the bundle does not define in TypeScript
read_bundle_source fell back to the bundle entrypoint for any Dag it could not
map to a source file. That is wrong for a mixed-language Dag: the Dag is defined
in Python, which owns its Code tab, and the TypeScript bundle only supplies task
handlers — showing the TypeScript entrypoint for it would be misleading. Return
None instead, so the caller can show the owning language's source. The entrypoint
stays embedded and recorded as entrypoint_path, a bundle-level reference a caller
can fall back to for a native TypeScript Dag that has no source of its own.
* Let the caller choose when a bundle serves a Dag's TypeScript source
A mixed-language Dag is defined in Python, which owns its Code tab, so the
TypeScript bundle should offer no source for it. The caller signals that by
omitting dag_id, and the reader returns None instead of an unrelated file. When a
dag_id is given the caller is asking for a Dag it treats as TypeScript-defined, so
a Dag missing from dag_source_paths — one built dynamically that the packer could
not attribute — falls back to the embedded entrypoint source rather than nothing.1 parent 5799660 commit a8a6257
13 files changed
Lines changed: 989 additions & 208 deletions
File tree
- airflow-core/docs/authoring-and-scheduling/language-sdks
- task-sdk
- src/airflow/sdk/coordinators/node
- tests/task_sdk/coordinators/node
- ts-sdk
- src
- cli
- coordinator
- sdk
- tests
- cli
- fixtures
- coordinator
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
435 | 435 | | |
436 | 436 | | |
437 | 437 | | |
438 | | - | |
439 | | - | |
440 | | - | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
441 | 444 | | |
442 | 445 | | |
443 | 446 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
22 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
47 | | - | |
48 | | - | |
| 48 | + | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
67 | 68 | | |
68 | 69 | | |
69 | 70 | | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
70 | 81 | | |
71 | 82 | | |
72 | 83 | | |
73 | 84 | | |
74 | 85 | | |
75 | | - | |
| 86 | + | |
| 87 | + | |
76 | 88 | | |
77 | 89 | | |
78 | 90 | | |
| |||
82 | 94 | | |
83 | 95 | | |
84 | 96 | | |
85 | | - | |
| 97 | + | |
86 | 98 | | |
87 | 99 | | |
88 | 100 | | |
| |||
96 | 108 | | |
97 | 109 | | |
98 | 110 | | |
99 | | - | |
| 111 | + | |
100 | 112 | | |
101 | 113 | | |
102 | 114 | | |
| |||
119 | 131 | | |
120 | 132 | | |
121 | 133 | | |
122 | | - | |
123 | | - | |
124 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
125 | 157 | | |
126 | 158 | | |
127 | 159 | | |
128 | 160 | | |
129 | | - | |
| 161 | + | |
130 | 162 | | |
131 | 163 | | |
132 | | - | |
| 164 | + | |
133 | 165 | | |
134 | 166 | | |
135 | 167 | | |
| |||
198 | 230 | | |
199 | 231 | | |
200 | 232 | | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
201 | 244 | | |
202 | 245 | | |
203 | 246 | | |
| |||
206 | 249 | | |
207 | 250 | | |
208 | 251 | | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
217 | 278 | | |
218 | 279 | | |
219 | 280 | | |
| |||
225 | 286 | | |
226 | 287 | | |
227 | 288 | | |
228 | | - | |
| 289 | + | |
229 | 290 | | |
230 | 291 | | |
231 | 292 | | |
| |||
301 | 362 | | |
302 | 363 | | |
303 | 364 | | |
304 | | - | |
305 | | - | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
306 | 368 | | |
307 | 369 | | |
308 | 370 | | |
| |||
328 | 390 | | |
329 | 391 | | |
330 | 392 | | |
331 | | - | |
| 393 | + | |
332 | 394 | | |
333 | 395 | | |
334 | 396 | | |
| |||
344 | 406 | | |
345 | 407 | | |
346 | 408 | | |
347 | | - | |
| 409 | + | |
348 | 410 | | |
349 | | - | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
350 | 415 | | |
351 | | - | |
| 416 | + | |
| 417 | + | |
352 | 418 | | |
353 | 419 | | |
354 | 420 | | |
| |||
385 | 451 | | |
386 | 452 | | |
387 | 453 | | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
388 | 479 | | |
389 | 480 | | |
390 | 481 | | |
| |||
411 | 502 | | |
412 | 503 | | |
413 | 504 | | |
414 | | - | |
415 | | - | |
416 | | - | |
417 | | - | |
| 505 | + | |
| 506 | + | |
418 | 507 | | |
419 | 508 | | |
420 | | - | |
| 509 | + | |
421 | 510 | | |
422 | 511 | | |
423 | | - | |
| 512 | + | |
424 | 513 | | |
425 | | - | |
426 | | - | |
427 | | - | |
428 | | - | |
429 | | - | |
430 | | - | |
431 | | - | |
432 | | - | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
440 | | - | |
441 | | - | |
442 | | - | |
443 | | - | |
444 | | - | |
445 | | - | |
446 | | - | |
447 | | - | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
0 commit comments