Skip to content

TS SDK: generate Dag and task options from the serialization schema - #73436

Merged
jason810496 merged 1 commit into
apache:mainfrom
jason810496:feature/ts-sdk/dag-schema-codegen
Sep 24, 2026
Merged

jason810496 merged 1 commit into
apache:mainfrom
jason810496:feature/ts-sdk/dag-schema-codegen

Conversation

@jason810496

@jason810496 jason810496 commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Why

A native Dag has to express what Python would otherwise own: the schedule, and each task's options. Both already have a source of truth — Airflow's own Dag serialization schema — so the types are generated from it rather than hand-written, as src/generated/supervisor.ts is.

How

  • ts-sdk/schema/dag-schema.json is a vendored copy of airflow-core/src/airflow/serialization/schema.json, kept honest by a prek hook, so a published npm package builds without the monorepo.
  • scripts/generate-dag-schema.mjs emits GeneratedDagFields / GeneratedTaskFields plus the table the serializer reads. Task fields are derived; Dag fields come from a curated allowlist, because the dag definition mixes authoring options with bundle bookkeeping.
  • Every eligible key of either definition has to be accounted for — a field, an exclusion, or a shape the spec cannot express. So a key the schema gains fails generation instead of going missing, and a key that changes shape fails instead of silently deleting its option and turning every call site that sets it into "Unknown option".
  • TaskOptions.argBindings replaces argNames, matching the arg_bindings it fills in the serialized Dag and leaving withArgNames the only argNames in the SDK.

Was generative AI tooling used to co-author this PR?

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a couple of thing worth addressing.

Comment thread ts-sdk/src/sdk/dag.ts
Comment thread ts-sdk/schema/dag-schema.json
A Dag authored in TypeScript has to describe itself with the same options
Airflow's scheduler reads, and that vocabulary lives in airflow-core's Dag
serialization schema. Hand-listing those options in the SDK would leave two
copies to keep aligned by hand, and the copy the SDK ships would quietly rot
every time core adds, renames, or drops a key.

Deriving them instead makes the drift visible: a new scalar key shows up in the
regenerated diff or fails generation until someone says why the SDK omits it,
and a curated entry that outlives the key it names fails the same way. The
serialization version is the one thing the schema cannot supply, so a core bump
now fails a test in airflow-core rather than producing Dags stamped with a
version core no longer writes.

This follows the policy the Go SDK's TaskSpec generator set and ADR-0008
records, so the three language SDKs expose the same Dag surface for the same
documented reasons.
@jason810496
jason810496 force-pushed the feature/ts-sdk/dag-schema-codegen branch from 69dde3f to 006e05e Compare September 24, 2026 04:07
@jason810496
jason810496 merged commit ea1f472 into apache:main Sep 24, 2026
99 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants