Skip to content

TS SDK: name a task's inputs, and take its id from the handler - #73437

Merged
jason810496 merged 1 commit into
apache:mainfrom
jason810496:feature/ts-sdk/default-task-id-at-pack-time
Sep 29, 2026
Merged

jason810496 merged 1 commit into
apache:mainfrom
jason810496:feature/ts-sdk/default-task-id-at-pack-time

Conversation

@jason810496

@jason810496 jason810496 commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Why

Every argument reaches the serialized Dag by name, and a positional parameter list has none unless the SDK reads them out of the handler's source. A handler that takes one object of named arguments is what a TypeScript library takes anyway, and it settles both ends: the call site carries the names, and the handler's own name is still the task id.

const transform = dag.task(
  "transform",
  async ({ rows, region }: { rows: number; region: string }) => rows * 2,
);

transform({ rows: extracted, region: "us" });

How

  • A call records its own keys, so nothing reads a function's parameter list and no argument is ever labelled arg0. TaskOptions.argBindings is gone with it.
  • A handler that declares more than one parameter, or a call given more than one argument, is an error, so a positional call from plain JavaScript fails instead of losing a value.
  • dag.task(handler) takes its id from handler.name, and TaskSpec.taskId sets one for an anonymous handler. Giving the id twice is an error rather than a silent winner.
  • airflow-ts-pack passes esbuild's keepNames, so the id survives minification.

ts-sdk/adr/0002 records the positional handler as tried and dropped. An earlier revision of this PR read the names at pack time with an esbuild onLoad hook and TypeScript's parser; it needed a parser in the packer and could not see a handler declared in another module.


Was generative AI tooling used to co-author this PR?

@jason810496 jason810496 self-assigned this Sep 21, 2026
@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch 2 times, most recently from d7a6936 to dc92410 Compare September 22, 2026 10:50
@jason810496 jason810496 changed the title TS SDK: resolve an omitted task id at pack time TS SDK: resolve a task id and its argument names at pack time Sep 22, 2026
@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch 2 times, most recently from e3f5dc3 to 0c3b08b Compare September 22, 2026 13:10
@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch from 0c3b08b to adb4c2e Compare September 23, 2026 02:45
@jason810496
jason810496 requested review from guan404ming and pierrejeambrun and removed request for pierrejeambrun September 23, 2026 02:51
@jason810496
jason810496 marked this pull request as ready for review September 23, 2026 02:52

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice thanks.

A few minor comments

Comment thread ts-sdk/tests/sdk/dag.test.ts
Comment thread ts-sdk/package.json Outdated
Comment thread ts-sdk/src/cli/pack.ts Outdated
@pierrejeambrun

pierrejeambrun commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

@ashb TS SDK: Retrieve task id from fn name via a plugin to manipulate the source before packaging.

Since this adds some complexity, I'm second guessing if keepNames (basically not minifying names) was a best option

  • Would remove most of this code
  • Would allow for cleaner debug process, with stack traces pointing to real function names instead of minified ones
  • This shouldn't matter in term of bundle 'size' (my agent is hinting at 1-2%), the workers are already pulling tons of dependencies (python stuff), a few extra names shouldn't be that bad. And it shouldn't have a meaningful impact on execution speed as well.

Also common node recommendation for server side is typically not to minify at all. (keepNames can meet best of both world).

(Sorry for the incorrect ping Andrew)

My bad

@jason810496 jason810496 left a comment •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also common node recommendation for server side is typically not to minify at all.

By this way and from your perspective, should we support the tsx way to start the TS SDK runtime?
Since Ash and I had a conversation back in Airflow Submit that Ash would like to have a way to start the TS runtime without bundling. I'm not sure the convention of "bundling or not" for TS server-side project myself.

However, all the current SDKs are heavily rely on the metadata at the artifacts for discovery purpose.
Since this topic is related with this PR so I would like to bring this up.

@ashb

ashb commented Sep 23, 2026

Copy link
Copy Markdown
Member

(Sorry for the incorrect ping Andrew)

@pierrejeambrun

Copy link
Copy Markdown
Member

By this way and from your perspective, should we support the tsx way to start the TS SDK runtime?
Since Ash and I had a conversation back in Airflow Submit that Ash would like to have a way to start the TS runtime without bundling.

We can add this alongside current capability if this helps our users.

I'm not sure the convention of "bundling or not" for TS server-side project myself.

For portability reason i'd say that we still want bundling to happen. (pnpm install + moving whole source file around is probably not great)

@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch from adb4c2e to 1fa5029 Compare September 24, 2026 04:07
@jason810496 jason810496 changed the title TS SDK: resolve a task id and its argument names at pack time TS SDK: default a task id and argument labels to the handler's names Sep 24, 2026
@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch from 1fa5029 to 8fb2d5f Compare September 24, 2026 05:25

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heads up on one consequence:

Without the plugin, the task id and the argument names are read off the handler when the Dag module "runs" runtime.

The "Runs" here is not the task runtime. It is the Dag module executes during packing, where airflow-ts-pack runs the staged bundle with --airflow-metadata (node bundle.min.mjs --airflow-metadata) to collect the dag ids and task ids for the embedded manifest.

Not sure would it still be more clean if we implement the pack time plugin as last round to retrieve the function and arguments names as TaskSpec.

The design choice between Packing transformation vs Runtime inferring

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, two comments we need to address before moving forward.

Comment thread ts-sdk/src/sdk/dag.ts
Comment thread ts-sdk/src/sdk/dag.ts Outdated
@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch from 8fb2d5f to f86a495 Compare September 24, 2026 14:19
@jason810496 jason810496 changed the title TS SDK: default a task id and argument labels to the handler's names TS SDK: name a task's inputs, and take its id from the handler Sep 24, 2026
@jason810496
jason810496 marked this pull request as draft September 27, 2026 13:43
@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch from f86a495 to ea3d111 Compare September 27, 2026 13:50
@jason810496
jason810496 marked this pull request as ready for review September 27, 2026 15:00

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review and the discussion.
I just addressed all the comments.

Comment thread ts-sdk/src/sdk/dag.ts
Comment thread ts-sdk/src/sdk/dag.ts Outdated
Comment thread ts-sdk/adr/0002-native-dag-interface.md Outdated
@jason810496
jason810496 marked this pull request as draft September 29, 2026 12:04
@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch from ea3d111 to 851c98d Compare September 29, 2026 12:13
@jason810496
jason810496 marked this pull request as ready for review September 29, 2026 12:18
Comment thread ts-sdk/adr/0002-native-dag-interface.md Outdated
@jason810496
jason810496 marked this pull request as draft September 29, 2026 12:56
A native Dag handler takes one object of named arguments, and a call names
each input, so nothing labels an argument arg0. A call given more than one
argument is an error rather than a silently dropped value.

The task id defaults to the handler's function name, so airflow-ts-pack
passes esbuild's keepNames to keep that name through minification.
@jason810496
jason810496 force-pushed the feature/ts-sdk/default-task-id-at-pack-time branch from 851c98d to bd3594f Compare September 29, 2026 12:58
@jason810496
jason810496 marked this pull request as ready for review September 29, 2026 15:47

@jason810496 jason810496 left a comment •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Pierre, CI passed while marking the PR as draft.
I will merge the PR now.

Comment thread ts-sdk/adr/0002-native-dag-interface.md Outdated
@jason810496
jason810496 merged commit a3555c7 into apache:main Sep 29, 2026
121 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants