Skip to content

Commit a3555c7

Browse files
authored
TS SDK: default a task id to the handler's name (#73437)
A native Dag handler takes one object of named arguments, and a call names each input, so nothing labels an argument arg0. A call given more than one argument is an error rather than a silently dropped value. The task id defaults to the handler's function name, so airflow-ts-pack passes esbuild's keepNames to keep that name through minification.
1 parent 6b90003 commit a3555c7

9 files changed

Lines changed: 423 additions & 296 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/language-sdks/typescript.rst‎

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -247,8 +247,8 @@ A ``Dag`` is declared on this side rather than in Python: its schedule, its task
247247
the edges between them are all written in TypeScript. The surface is still growing, so a Dag declared
248248
this way is not served to Airflow yet.
249249

250-
``dag.task(taskId, handler)`` returns a *factory*. Calling it places the task in the Dag and supplies the
251-
handler's arguments, so the call graph is the task graph:
250+
``dag.task(taskId, handler)`` returns a *factory*. A handler takes one object of named arguments, and
251+
calling the factory names each input, so the call graph is the task graph:
252252

253253
.. code-block:: typescript
254254
@@ -257,26 +257,38 @@ handler's arguments, so the call graph is the task graph:
257257
const dag = new Dag("ts_etl");
258258
259259
const extract = dag.task("extract", async (): Promise<number> => 42);
260-
const transform = dag.task("transform", async (rows: number, region: string) => rows * 2);
261-
const load = dag.task("load", async (total: number) => {});
260+
const transform = dag.task(
261+
"transform",
262+
async ({ rows, region }: { rows: number; region: string }) => rows * 2,
263+
);
264+
const load = dag.task("load", async ({ total }: { total: number }) => {});
262265
263-
load(transform(extract(), "us"));
266+
const extracted = extract();
267+
const total = transform({ rows: extracted, region: "us" });
268+
load({ total });
264269
265-
Arguments are passed in the order the handler declares them. A handler that declares a single object of
266-
named arguments can also be called with that object, which names each input instead of ordering it:
267-
268-
.. code-block:: typescript
269-
270-
const store = dag.task("store", async ({ total }: { total: number }) => {});
271-
272-
store({ total: extract() });
270+
Naming the inputs is how a task is called. A handler that takes no arguments is called with none, and
271+
a single argument is named like any other, ``load({ total })``. The compiler checks the call: it reports
272+
an argument left out, a misspelled one, and a literal of the wrong type.
273273

274274
Each argument takes either an upstream reference or a literal JSON value. A reference has to be the
275275
argument itself: one buried inside an array or an object is a literal, and draws no edge.
276276

277277
Every task has to be called exactly once. An uncalled task fails when the Dag is read, so none can be
278278
left out of the graph by accident.
279279

280+
The task id may be omitted, in which case it is the handler's function name:
281+
282+
.. code-block:: typescript
283+
284+
const extract = dag.task(async function extract(): Promise<number> {
285+
return 42;
286+
});
287+
288+
``airflow-ts-pack`` keeps function names intact, so bundling cannot rename a task. A handler with no
289+
name of its own, such as an arrow function passed inline, has nothing to take an id from and needs
290+
one: either positionally or as ``taskId`` in its spec. Give it in one place only, not both.
291+
280292
``new Dag`` and ``dag.task`` both take a trailing spec of Airflow options:
281293
``{ schedule: "@daily", tags: ["etl"] }`` for the Dag, ``{ retries: 2, retryDelay: 30 }`` for a task.
282294

@@ -398,9 +410,8 @@ layout header. The layout records the byte ranges and SHA-256 digests of the man
398410
so there is one file to deploy, with no separate manifest or ``node_modules``.
399411

400412
The code is minified because an integrity digest is only worth taking over an artifact nobody is expected to
401-
read or edit in place. The ``/*! */`` license banners of bundled dependencies are kept. Nothing is identified by
402-
a function name, so minified names are safe: a Dag and a task are named by the string ids their registration
403-
states, and a handler is dispatched by reference.
413+
read or edit in place. Function names are kept through minification, since a task id defaults to its
414+
handler's name. The ``/*! */`` license banners of bundled dependencies are kept.
404415

405416
Because the shipped code is not the code anyone wrote, the packer also embeds the entry module verbatim in a
406417
``/*# airflowSource ... #*/`` block comment, verified by its own digest, so Airflow has something readable to

‎docs/spelling_wordlist.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1102,6 +1102,7 @@ midnights
11021102
milli
11031103
millis
11041104
milton
1105+
minification
11051106
minikube
11061107
misconfiguration
11071108
misconfigured

‎ts-sdk/adr/0002-native-dag-interface.md‎

Lines changed: 25 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -27,16 +27,13 @@ Proposed. Revised after the review on #72047.
2727

2828
1. **`dag.task(handler)` returns a factory, and the task id is optional.** With no id the task takes
2929
the handler's function name (`dag.task(extract)` → task `"extract"`); `dag.task(taskId, handler)`
30-
sets it explicitly, which an anonymous handler must do. Calling the factory both places the task in
31-
the Dag and supplies its arguments, in the order the handler declares them
32-
(`load(transform(extract(), "us"))`). A handler that declares a single object of named arguments can
33-
also be called with that object — the shape Python TaskFlow uses for
34-
`load(transformed=transform(...))`.
35-
2. **The call graph is the task graph.** `tsc` checks every wired key against the handler's own
36-
parameter type, and a `TaskRef` exists only once its producing call has returned, so a cycle
37-
through arguments is unrepresentable rather than rejected by a validator. A reference passed by
38-
position is checked against the argument's own type, which is what tells the two call shapes apart
39-
when a handler declares a single argument.
30+
sets it explicitly, which an anonymous handler must do. A handler takes one object of named
31+
arguments, and calling the factory both places the task in the Dag and names each of its inputs
32+
(`load({ total: transform({ rows: extract(), region: "us" }) })`) — the shape Python TaskFlow uses
33+
for `load(transformed=transform(...))`.
34+
2. **The call graph is the task graph.** `tsc` checks every named input against the handler's own
35+
argument type, and a `TaskRef` exists only once its producing call has returned, so a cycle
36+
through arguments is unrepresentable rather than rejected by a validator.
4037
3. **Every task is called exactly once.** An uncalled task fails when the Dag is read, so none can be
4138
silently left out of the graph.
4239
4. **`before` and `after` draw order-only edges** — the TypeScript pair for `>>` and `<<`, both
@@ -107,13 +104,12 @@ const extract = dag.task(async function extract(): Promise<number> {
107104
// task id "extract"
108105
```
109106

110-
The id comes from the handler's *source* name, resolved when the bundle is packed and written into
111-
the registration — not from `handler.name` at runtime, which minification renames (see
112-
Implementation Notes). A handler with no source name — a bare anonymous arrow passed inline,
113-
`dag.task(async () => 42)` — has nothing to resolve and is a compile error until given an explicit
114-
id. This default is for native Dags, where both ends of every name are TypeScript; a mixed-language
115-
handler names the Python-owned task explicitly and does not default from the handler's function
116-
name ([ADR-0001](0001-mixed-lang-dag-interface.md), decision 3).
107+
The id is the handler's `name`, which `airflow-ts-pack` keeps through minification (see
108+
Implementation Notes). A handler with no name, such as a bare anonymous arrow passed inline,
109+
`dag.task(async () => 42)`, has nothing to take an id from and fails when the Dag is declared until
110+
given an explicit id. This default is for native Dags, where both ends of every name are
111+
TypeScript; a mixed-language handler names the Python-owned task explicitly and does not default
112+
from the handler's function name ([ADR-0001](0001-mixed-lang-dag-interface.md), decision 3).
117113

118114
The `TaskSpec` also carries the task id, so it can be set alongside the other task options:
119115

@@ -149,20 +145,19 @@ convention.
149145
by design. Native declaration is what fills them, generated from the serialized-Dag JSON schema the
150146
way `src/generated/supervisor.ts` is. This ADR does not choose those fields; it fixes where an
151147
author writes them.
152-
- `TaskOptions` carries the spec and the handler's positional argument names, which the packer fills in
153-
from the parameter list so the Dag names each argument as its handler does. With wiring moved to the
154-
factory call, `inputs` is no longer an option.
148+
- `TaskOptions` carries the task's spec and nothing else: the names on the wire are the keys of the
149+
call itself. With wiring moved to the factory call, `inputs` is no longer an option.
155150
- `TaskHandlerArgs` is removed from the public API, `DagRegistry` becomes `Bundle`, and
156151
`serveDags(registry)` becomes `bundle.serve()`, which breaks
157152
0.1.0-beta1 authors; see [ADR-0001](0001-mixed-lang-dag-interface.md) for the shipped call sites
158153
that change.
159154

160155
## Alternatives
161156

162-
- **Named-only wiring**, rejected in the review on #73435: naming every input reads well at twenty
163-
tasks but forces an object around a single argument, and positional calls are what TypeScript
164-
authors write. Both are offered, and the handler's own parameter list decides which one a task can
165-
use.
157+
- **Positional handlers**, `async (rows: number, region: string) => ...`, offered first and then
158+
dropped: a positional parameter list has no names on the wire unless the SDK reads them out of the
159+
handler's source, and a single object of named arguments is what a TypeScript library takes
160+
anyway.
166161
- **Injected `ctx`/`client` arguments**, mimicking the Python signature. Rejected, per the above and
167162
because feeling native to TypeScript matters more than matching Python's parameter list.
168163

@@ -181,17 +176,14 @@ convention.
181176
- **The spec argument already has its slot.** `dag.task(taskId, handler, options)` reads `{ spec = {} }`
182177
and runs `validateEmptySpec` on it (`ts-sdk/src/sdk/dag.ts`), so task fields land on a path that
183178
exists rather than a new one.
184-
- **A positional argument binds by order, and its name is a label.** The serialized Dag names each
185-
argument, so the packer reads the names from the handler's parameter list; `arg0`, `arg1` and so on
186-
stand in for a name it cannot see, without changing which value reaches which argument.
187179
- **A `TaskRef` is inert** — a handle for wiring, not a promise. Nothing in a Dag file executes a task
188180
body.
189-
- **A defaulted task id is resolved at pack time, not read at runtime.** esbuild renames function
190-
identifiers, so `handler.name` in a packed bundle is the minified name, not the author's. The pack
191-
step (`ts-sdk/src/cli/pack.ts`) therefore reads an omitted id from the handler's declared name in
192-
source and writes it into the registration, rather than depending on `handler.name` or enabling
193-
esbuild's `keepNames` across the whole bundle. A handler with no source name leaves nothing to
194-
read, which is why an anonymous handler must state its id.
181+
- **A defaulted task id is read off the handler itself.** The pack step (`ts-sdk/src/cli/pack.ts`)
182+
minifies and passes esbuild's `keepNames`, so `handler.name` is the author's in a packed bundle as
183+
much as in one run from source. Rewriting the call at pack time was tried first and dropped: it
184+
needed a TypeScript parser in the packer to tell a real `.task(` from one inside a string or a
185+
comment, and it could not see a handler declared in another module. A handler with no name leaves
186+
nothing to read, which is why an anonymous handler must state its id.
195187
- **`withArgNames` and the name folding behind it** ([ADR-0001](0001-mixed-lang-dag-interface.md))
196188
exist for the mixed-language case and are never needed here: both ends of every name are
197189
TypeScript, so `tsc` checks the wiring end to end and there is no foreign name to reconcile.

‎ts-sdk/src/cli/pack.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,10 @@ export async function runPack(argv: readonly string[]): Promise<void> {
217217
target: "node22",
218218
// A digest is only worth taking over an artifact nobody reads or edits in place.
219219
minify: true,
220+
// A task id comes from the handler's name, and bundling renames a symbol
221+
// that two modules both declare, so the name the SDK reads is pinned to
222+
// the one the author wrote.
223+
keepNames: true,
220224
// The manifest is read by running the staged bundle, so the metadata describes what ships.
221225
outfile: stagingPath,
222226
});

‎ts-sdk/src/index.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@ export type { ArgNameMap } from "./sdk/arg-names.js";
2828
export type { Registerable } from "./sdk/bundle.js";
2929
export type {
3030
DagSpec,
31-
PositionalInputs,
3231
TaskFactory,
3332
TaskInput,
3433
TaskInputs,

0 commit comments

Comments
 (0)