Skip to content

Commit 20cece7

Browse files
committed
TS SDK: add triggerDagRun to the native Dag surface
1 parent 92e3b28 commit 20cece7

14 files changed

Lines changed: 1409 additions & 24 deletions

File tree

‎airflow-core/adr/lang-sdk/0008-control-flow-constructs.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,8 @@ Proposed.
3030
2. **A branch selects a task, not a string.** A case is the task reference the SDK already handed back, and the value on the wire is that task's task_id, so the compiler checks the candidate exists and no label has to be kept in step with it.
3131
3. **No default case.** `BranchPythonOperator` has none to serialize, and a one-sided `If` whose condition is false follows nothing at all.
3232
A decider returning a ref that is not one of the declared cases is a run-time error the SDK raises, a narrower check than `skip_all_except`, which only rejects a task_id missing from the whole Dag.
33-
4. **Triggering a Dag run is an ordinary DSL task**, it is pure DSL purpose instead of a new runtime.
33+
4. **Triggering a Dag run is an ordinary task that the SDK's own runtime runs.** It has no handler.
34+
The runtime sends the `TriggerDagRun` request itself and follows `TriggerDagRunOperator` for every option it offers, including `wait_for_completion` and `deferrable`.
3435
5. **Grouping keeps Python's semantics**: a scope offering the same task and nesting methods as the Dag, prefixing each task_id with the group id (`prefix_group_id`),
3536
and can be ordered against a task or another group, as `group1 >> group2` does in Python.
3637

@@ -89,7 +90,12 @@ That operator defaults to skipping every task in its downstream closure and igno
8990
a single-ref return cannot express that, and no Lang SDK offers it for now.
9091
An author needing several paths together puts them behind one task, or gates each with its own condition.
9192
This limitation is accepted rather than open.
92-
- **No Lang SDK needs a deferral mechanism for now** to offer `deferrable` or `wait_for_completion`, because the trigger task runs in Python.
93+
- **Each Lang SDK runtime implements the trigger task**, so a native Dag needs no Python worker.
94+
`deferrable` needs no trigger written in that language: the runtime defers to the standard provider's `DagStateTrigger`, which runs in the Python triggerer, and the task keeps its queue, so it resumes in the same runtime with `next_method` set.
95+
- **The runtime reads the config the trigger task needs from its environment**, since it cannot read `airflow.cfg`.
96+
It reads `AIRFLOW__API__BASE_URL`, `AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE` and `AIRFLOW__TRIGGERER__QUEUES_ENABLED`, and an unset one takes Python's fallback.
97+
The coordinator exports them from Airflow's config (#73842).
98+
- **The trigger's arguments are not Jinja-rendered**, because no Python code runs the task. A value is sent as the author wrote it.
9399
- **Nothing extra reaches the Dag JSON**, which carries no branch-candidate field at all. A ref is a task_id by the time the decision is sent, so each SDK stores its cases and nothing else.
94100
- **A group edge needs one base type per SDK** that both a task and a group satisfy, since either can sit at the end of an edge.
95101
Python already has it: `TaskGroup(TaskGroupMixin, DAGNode)` (`task-sdk/src/airflow/sdk/definitions/taskgroup.py:96`) and every operator inherit `DependencyMixin` (`.../definitions/_internal/mixins.py:35`), where `set_upstream` and `set_downstream` live.

‎airflow-core/docs/authoring-and-scheduling/language-sdks/typescript.rst‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -436,6 +436,50 @@ Exactly one case is selected. Python's branch callable may return a list of task
436436
SDK offers that yet: put the paths that run together behind one task, or gate each with its own
437437
condition.
438438

439+
Triggering another Dag
440+
~~~~~~~~~~~~~~~~~~~~~~
441+
442+
``dag.triggerDagRun`` starts another Dag's run, as a task of this one:
443+
444+
.. code-block:: typescript
445+
446+
const trigger = dag.triggerDagRun({
447+
taskId: "trigger_downstream",
448+
dagId: "downstream_etl",
449+
waitForCompletion: true,
450+
});
451+
452+
trigger.after(loaded);
453+
454+
There is nothing to call: the task takes no TypeScript arguments, so it hands back the reference
455+
directly. A second options object carries the task's own spec, as ``dag.task`` takes one.
456+
457+
The task runs in the TypeScript runtime, like the Dag's other tasks, and inherits the Dag's queue. It
458+
behaves as ``TriggerDagRunOperator`` does. ``waitForCompletion`` polls the run every ``pokeInterval``
459+
seconds until it reaches one of ``allowedStates`` or ``failedStates``. With ``deferrable`` as well, the
460+
task defers to ``DagStateTrigger`` instead of holding a worker slot, and resumes in this runtime when
461+
the run finishes. ``DagStateTrigger`` runs in the Python triggerer, so the triggerer needs the standard
462+
provider installed.
463+
464+
The task pushes the ``trigger_run_id`` XCom, and the task's "Triggered DAG" link opens the run it
465+
started.
466+
467+
The task follows these config options, with Python's fallback when one is unset:
468+
469+
- ``[api] base_url`` is the base of the "Triggered DAG" link. It falls back to ``/``.
470+
- ``[operators] default_deferrable`` is the default of ``deferrable``. It falls back to ``false``.
471+
- ``[triggerer] queues_enabled`` decides whether the deferred trigger gets the task's queue. It falls
472+
back to ``false``, so the trigger gets none.
473+
474+
Some of what ``TriggerDagRunOperator`` does is not offered:
475+
476+
- ``logical_date`` and ``run_after``. The triggered run's logical date is the time the task triggers
477+
it, as in Python when neither is set.
478+
- Jinja. Values are sent as written, so ``{{ ds }}`` in ``conf`` reaches the triggered run as that
479+
literal string.
480+
- OpenLineage parent injection (``openlineage_inject_parent_info``). The runtime does not add the
481+
parent task's OpenLineage details to ``conf``.
482+
439483
``new Dag`` and ``dag.task`` both take a trailing spec of Airflow options:
440484
``{ schedule: "@daily", tags: ["etl"] }`` for the Dag, ``{ retries: 2, retryDelay: 30 }`` for a task.
441485

‎ts-sdk/src/coordinator/client.ts‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ import type {
3131
SetXCom,
3232
GetConnection,
3333
SkipDownstreamTasks,
34+
TriggerDagRun,
35+
GetDagRunState,
36+
GetDag,
3437
ConnectionResult as WireConnectionResult,
3538
} from "./protocol.js";
3639

@@ -83,6 +86,15 @@ export interface CoordinatorClient extends TaskClient {
8386
* output fails the task, while one that pushed null binds null.
8487
*/
8588
getXComEntry(opts: GetXComOpts): Promise<XComEntry>;
89+
/**
90+
* Trigger a Dag run. `"already_exists"` is the one refusal a trigger task
91+
* handles itself; any other error throws.
92+
*/
93+
triggerDagRun(msg: Omit<TriggerDagRun, "type">): Promise<"triggered" | "already_exists">;
94+
/** The state of a Dag run, as `GetDagRunState` reports it. */
95+
getDagRunState(dagId: string, runId: string): Promise<string>;
96+
/** Whether a Dag is paused. */
97+
isDagPaused(dagId: string): Promise<boolean>;
8698
}
8799

88100
export function createCoordinatorClient(
@@ -213,6 +225,47 @@ export function createCoordinatorClient(
213225
await rpc("SkipDownstreamTasks", null, msg, () => undefined, "throw");
214226
},
215227

228+
// ---- Dag runs ----
229+
230+
async triggerDagRun(msg: Omit<TriggerDagRun, "type">) {
231+
const request: TriggerDagRun = { type: "TriggerDagRun", ...msg };
232+
logs?.debug("TriggerDagRun request");
233+
const frame = await comm.request(request);
234+
const err = parseFrameError(frame);
235+
// Sent as an ErrorResponse body, not a frame error; see
236+
// `ActivitySubprocess._handle_trigger_dag_run`.
237+
if (err?.code === "DAGRUN_ALREADY_EXISTS") return "already_exists";
238+
if (err) {
239+
logs?.warning("TriggerDagRun failed", { error: err.code });
240+
throw new Error(`TriggerDagRun failed: ${err.code}`);
241+
}
242+
return "triggered";
243+
},
244+
245+
async getDagRunState(dagId: string, runId: string): Promise<string> {
246+
const msg: GetDagRunState = { type: "GetDagRunState", dag_id: dagId, run_id: runId };
247+
const state = await rpc(
248+
"GetDagRunState",
249+
"DagRunStateResult",
250+
msg,
251+
(body) => body!.state as string,
252+
"throw",
253+
);
254+
return state!;
255+
},
256+
257+
async isDagPaused(dagId: string): Promise<boolean> {
258+
const msg: GetDag = { type: "GetDag", dag_id: dagId };
259+
const paused = await rpc(
260+
"GetDag",
261+
"DagResult",
262+
msg,
263+
(body) => body!.is_paused === true,
264+
"throw",
265+
);
266+
return paused!;
267+
},
268+
216269
// ---- Connections ----
217270

218271
async getConnection(connId: string): Promise<ConnectionResult | null> {

‎ts-sdk/src/coordinator/protocol.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ import type {
2828
DagFileParseRequest,
2929
ErrorResponse,
3030
DagFileParsingResult,
31+
DeferTask,
3132
RetryTask,
3233
SucceedTask,
3334
TaskState,
@@ -47,6 +48,7 @@ export type {
4748

4849
export type {
4950
DagFileParsingResult,
51+
DeferTask,
5052
RetryTask,
5153
SucceedTask,
5254
TaskState,
@@ -65,6 +67,11 @@ export type {
6567
SetXCom,
6668
GetConnection,
6769
SkipDownstreamTasks,
70+
TriggerDagRun,
71+
GetDagRunState,
72+
DagRunStateResult,
73+
GetDag,
74+
DagResult,
6875
} from "../generated/supervisor.js";
6976

7077
// -------- Frames from supervisor --------
@@ -82,6 +89,7 @@ export type MsgFromSupervisor =
8289

8390
export type RuntimeTaskState = WithRequiredType<TaskState>;
8491
export type RuntimeRetryTask = WithRequiredType<RetryTask>;
92+
export type RuntimeDeferTask = WithRequiredType<DeferTask>;
8593
export type RuntimeSucceedTask = WithRequiredType<SucceedTask> & {
8694
task_outlets: NonNullable<SucceedTask["task_outlets"]>;
8795
outlet_events: NonNullable<SucceedTask["outlet_events"]>;

‎ts-sdk/src/coordinator/runtime.ts‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,16 +49,18 @@ import {
4949
asMsgFromSupervisor,
5050
SUPERVISOR_API_VERSION,
5151
type RuntimeDagFileParsingResult,
52+
type RuntimeDeferTask,
5253
type RuntimeRetryTask,
5354
type RuntimeSucceedTask,
5455
type RuntimeTaskState,
5556
type StartupDetails,
5657
} from "./protocol.js";
5758
import { getArgNames } from "../sdk/arg-names.js";
58-
import { bundleDags, bundleDagTaskIds, type Bundle } from "../sdk/bundle.js";
59+
import { bundleDags, bundleDagTaskIds, getBundleTrigger, type Bundle } from "../sdk/bundle.js";
5960
import { finalizeDag } from "../sdk/dag.js";
6061
import { SERIALIZATION_VERSION } from "../generated/dag-schema-fields.js";
6162
import { computeRelativeFileloc, serializeDag } from "./serde.js";
63+
import { runTriggerDagRun } from "./trigger-runner.js";
6264
import { runInTaskScope, type TaskContext } from "../sdk/task.js";
6365
import type { JsonValue } from "../sdk/client-types.js";
6466

@@ -195,6 +197,8 @@ export async function startCoordinator(
195197
await sendSupervisorResponse(firstFrame.id, response, comm, runtimeLogs);
196198
if (response.type === "SucceedTask") {
197199
runtimeLogs.info("Task succeeded", { task_id: body.ti.task_id });
200+
} else if (response.type === "DeferTask") {
201+
runtimeLogs.info("Task deferred", { task_id: body.ti.task_id });
198202
}
199203
} else {
200204
const errMsg = `First frame must be DagFileParseRequest or StartupDetails, got ${body.type}`;
@@ -338,8 +342,22 @@ async function handleTask(
338342
logs: LogChannel,
339343
clientLogs: LogChannel,
340344
signal: AbortSignal,
341-
): Promise<RuntimeSucceedTask | RuntimeRetryTask | RuntimeTaskState> {
345+
): Promise<RuntimeSucceedTask | RuntimeRetryTask | RuntimeTaskState | RuntimeDeferTask> {
342346
const ti = details.ti;
347+
const trigger = getBundleTrigger(bundle, ti.dag_id, ti.task_id);
348+
if (trigger) {
349+
const ctx = buildContext(details, signal);
350+
const client = createCoordinatorClient(comm, ctx, clientLogs);
351+
const fail = (message: string) => {
352+
logs.error("Task failed", { task_id: ctx.taskId, error: message });
353+
return buildFailureResponse(details, message);
354+
};
355+
try {
356+
return await runTriggerDagRun(details, trigger, client, logs, ctx.signal, fail);
357+
} catch (err) {
358+
return fail((err as Error).message ?? String(err));
359+
}
360+
}
343361
const handler = bundle.getTaskHandler(ti.dag_id, ti.task_id);
344362

345363
if (!handler) {

‎ts-sdk/src/coordinator/serde.ts‎

Lines changed: 83 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ import {
5050
type Dag,
5151
type RecordedInputs,
5252
type TaskGroupRecord,
53+
type TaskRecord,
5354
} from "../sdk/dag.js";
5455

5556
/** A serialized Dag: JSON, by the time it reaches the supervisor as msgpack. */
@@ -83,6 +84,39 @@ const TASK_MODULE = "airflow.sdk.coordinators.node";
8384
*/
8485
const TASK_LANGUAGE = "typescript";
8586

87+
/**
88+
* The Dags this one reaches, as the UI's dependency graph and
89+
* `airflow dags show-dependencies` read them.
90+
*
91+
* Python derives these from the live operator with `detect_dependencies`. Only
92+
* a trigger task creates one here.
93+
*/
94+
function serializeDagDependencies(dag: Dag): SerializedValue {
95+
const dependencies: SerializedValue[] = [];
96+
for (const [taskId, record] of getDagTaskRecords(dag)) {
97+
if (!record.trigger) continue;
98+
dependencies.push({
99+
source: dag.dagId,
100+
target: record.trigger.dagId,
101+
label: taskId,
102+
dependency_type: "trigger",
103+
dependency_id: taskId,
104+
});
105+
}
106+
return dependencies;
107+
}
108+
109+
/**
110+
* What a trigger task adds to a TypeScript task, so the UI draws it as
111+
* `TriggerDagRunOperator` and shows its "Triggered DAG" link. The runtime
112+
* pushes the link's XCom under the key named here.
113+
*/
114+
const TRIGGER_DAG_RUN_FIELDS: Readonly<Record<string, SerializedValue>> = {
115+
_operator_name: "TriggerDagRunOperator",
116+
ui_color: "#ffefeb",
117+
_operator_extra_links: { "Triggered DAG": "_link_TriggerDagRunLink" },
118+
};
119+
86120
/** How one set of authoring fields is written into a serialized object. */
87121
interface FieldRules {
88122
readonly fields: Readonly<Record<string, SchemaField>>;
@@ -140,13 +174,13 @@ export function serializeDag(
140174
serializeTask(
141175
dag.dagId,
142176
taskId,
143-
withDagQueue(record.spec, dag.spec.queue),
177+
record,
144178
graph.downstreamTaskIds.get(taskId),
145179
inputs.get(taskId),
146-
record.canSkipDownstream === true,
180+
dag.spec.queue,
147181
),
148182
),
149-
dag_dependencies: [],
183+
dag_dependencies: serializeDagDependencies(dag),
150184
task_group: serializeTaskGroups(dag, graph),
151185
edge_info: {},
152186
params: [],
@@ -178,10 +212,10 @@ function withDagQueue(spec: object, dagQueue: string | undefined): object {
178212
function serializeTask(
179213
dagId: string,
180214
taskId: string,
181-
spec: object,
215+
record: TaskRecord,
182216
downstream: ReadonlySet<string> | undefined,
183217
inputs: RecordedInputs | undefined,
184-
canSkipDownstream: boolean,
218+
dagQueue: string | undefined,
185219
): SerializedValue {
186220
const data: Record<string, SerializedValue> = {
187221
task_id: taskId,
@@ -197,20 +231,63 @@ function serializeTask(
197231
// `get_arg_bindings` reads nothing without it.
198232
is_stub: true,
199233
};
234+
if (record.trigger) {
235+
// `conf` reaches Airflow at run time rather than in the Dag JSON, so a
236+
// value it cannot store is caught here, as this Dag's import error.
237+
if (record.trigger.conf !== undefined) {
238+
plainJsonArg(record.trigger.conf, `conf of task "${taskId}" of Dag "${dagId}"`);
239+
}
240+
Object.assign(data, structuredClone(TRIGGER_DAG_RUN_FIELDS));
241+
}
200242
const label = `task "${taskId}" of Dag "${dagId}"`;
201243
const bindings = serializeArgBindings(inputs, label);
202244
if (bindings) data["_arg_bindings"] = bindings;
203245
// What Python writes for a SkipMixin operator, and what makes
204246
// `NotPreviouslySkippedDep` consult this task's `skipmixin_key` XCom when one
205247
// of its skipped downstream tasks is cleared.
206-
if (canSkipDownstream) data["_can_skip_downstream"] = true;
248+
if (record.canSkipDownstream === true) data["_can_skip_downstream"] = true;
249+
const spec = withDagQueue(record.spec, dagQueue);
207250
applySchemaFields(data, spec, TASK_FIELD_RULES, label);
208251
if (downstream?.size) {
209252
data["downstream_task_ids"] = [...downstream].sort();
210253
}
211254
return { __type: "operator", __var: data };
212255
}
213256

257+
/**
258+
* A trigger task's `conf`, checked to be plain JSON.
259+
*
260+
* A value JSON cannot carry, such as a `Date`, a `Map` or a function, is
261+
* rejected, because msgpack would otherwise hand Airflow something it cannot
262+
* store as a Dag run's conf.
263+
*/
264+
function plainJsonArg(value: unknown, label: string): SerializedValue {
265+
if (value === null || value === undefined) return null;
266+
if (typeof value === "string" || typeof value === "boolean") return value;
267+
if (typeof value === "number") {
268+
if (!Number.isFinite(value)) throw new Error(`${label} is not a finite number`);
269+
return value;
270+
}
271+
if (Array.isArray(value))
272+
return value.map((item, index) => plainJsonArg(item, `${label}[${index}]`));
273+
if (isPlainObject(value)) {
274+
const copy: Record<string, SerializedValue> = {};
275+
for (const [key, item] of Object.entries(value))
276+
copy[key] = plainJsonArg(item, `${label}.${key}`);
277+
return copy;
278+
}
279+
throw new Error(
280+
`${label} is ${describeType(value)}, which a Dag run's conf cannot carry; ` +
281+
"pass a string, a number, a boolean, null, an array or a plain object",
282+
);
283+
}
284+
285+
function isPlainObject(value: unknown): value is Record<string, unknown> {
286+
if (typeof value !== "object" || value === null) return false;
287+
const prototype = Object.getPrototypeOf(value);
288+
return prototype === Object.prototype || prototype === null;
289+
}
290+
214291
/**
215292
* The task's arguments as the binding spec the API server hands back at run
216293
* time, one entry per argument in the order the call named them.

0 commit comments

Comments
 (0)