Show each native Dag's own source in the Code tab by pierrejeambrun · Pull Request #74249 · apache/airflow

pierrejeambrun · 2026-10-05T14:16:47Z

Stacked on #74037.

Why

A packed TypeScript bundle can declare several Dags, each mapped to its own source file via dag_source_paths. The Code tab showed the bundle's entry module for every one of them instead: NodeDagImporter.get_source_code always read the entrypoint, and the Dag processor read a bundle's source once per file and stored it under the file's own fileloc — every Dag sharing that file collided on the same key, so only one entry survived.

What changes

AbstractDagImporter.get_source_code takes the Dag's own dag_id now. NodeDagImporter resolves that Dag's own mapped file, falling back to the entry module when there is none (an unmapped, dynamically-constructed Dag). Every other importer's signature grew the same parameter for a uniform interface; their implementation is unchanged — only NodeDagImporter uses it so far.
Source code read for a Dag file is now keyed by dag_id end to end (DagBag.dag_source_codes, DagFileParsingResult.dag_source_codes, update_dag_parsing_results_in_db's lookup), since a fileloc is not unique per Dag the way a dag_id is — a plain Python file with several @dag-decorated functions has the same property, it just happened to be harmless there because every Dag in that case shares identical source anyway.

Out of scope

Go has no Dag importer yet, so there is nothing to wire up there.
JavaDagImporter already has the per-dag_id lookup machinery in _find_source_entry, unused by get_source_code. Wiring it in is left for a follow-up; this PR only updates its signature.

relates: #74241

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

A packed TypeScript bundle can declare several Dags, each mapped to its own source file. The Code tab showed the bundle's entry module for every one of them: the Dag processor read a bundle's source once per file and stored it under the file's own fileloc, so every Dag sharing that file collided on the same key and only one entry survived. get_source_code now takes the Dag's own dag_id, and NodeDagImporter resolves that Dag's mapped file instead of always falling back to the entry module. Every Dag importer's signature grew the same parameter for a uniform interface; only NodeDagImporter uses it so far. Source code read for a native Dag file is now keyed by dag_id end to end, since a fileloc is not unique per Dag the way a dag_id is.

pierrejeambrun · 2026-10-05T14:58:49Z

Multi dags bundle showing accurate source.

jason810496

Nice, thanks!
I just adjust the "closes" as "relates" in the PR description.

jason810496 · 2026-10-06T09:16:04Z

-        self.dag_source_codes.pop(fileloc, None)
+        # Keyed by dag_id, not fileloc: a bundle can embed several Dags' own source under one
+        # fileloc, so a stale entry is found by which dag_ids this fileloc previously bagged.
+        for dag_id, dag in list(self.dags.items()):


Here're some notes in my mind to solve the further write amplification if there's anyone complain in the future.

Suggested change

for dag_id, dag in list(self.dags.items()):

# TODO: store the file content more efficient (e.g. if multiple dag_ids will store the exact same DagCode content)

# There're two parts: 1) the Dagbag in-memory path 2) the DB persistent layer

# The bigger picture might be normalized the DagCode table, so we won't store the exact same DagCode content twice.

for dag_id, dag in list(self.dags.items()):

I will track a issue for this item.

Just merged the PR as my Go SDK DagImporter will need the core change of this PR as well.

A packed TypeScript bundle can declare several Dags, each mapped to its own source file. The Code tab showed the bundle's entry module for every one of them: the Dag processor read a bundle's source once per file and stored it under the file's own fileloc, so every Dag sharing that file collided on the same key and only one entry survived. get_source_code now takes the Dag's own dag_id, and NodeDagImporter resolves that Dag's mapped file instead of always falling back to the entry module. Every Dag importer's signature grew the same parameter for a uniform interface; only NodeDagImporter uses it so far. Source code read for a native Dag file is now keyed by dag_id end to end, since a fileloc is not unique per Dag the way a dag_id is.

* Parse native TypeScript Dags with the Node coordinator A NodeCoordinator configured without bundles_root now hands out a Dag importer, so the Dag bundles it serves have their *.min.mjs bundles parsed by running them with node. Only files that start with the airflow-ts-pack layout header are parsed. The Dag source shown for such a Dag is the bundle's entry module, as the bundle embeds it, or a short notice when there is none to show. * Document parsing native TypeScript Dags Describe the coordinator configuration that parses native Dags, what the Dag processor needs for it, what the Code view shows, the cost of handler-only bundles, and that cluster policies and some CLI commands do not apply to a native Dag. * Share the bundle section mutation helper between Node tests * Run a native TypeScript Dag's task from its own bundle A task used to run the first sorted *.min.mjs bundle that declares its dag_id, which can differ from the file the Dag was parsed from. When the task's Dag file is a bundle under the scan roots that still declares the Dag, run that file. Otherwise fall back to the search. * Say when the Dag processor needs Node.js for TypeScript bundles The Dag processor runs node on every packed bundle a Node coordinator serves, handler-only ones included, not only on bundles that declare TypeScript Dags. * Align the TypeScript build docs with the Code view The build section said each Dag's own file is embedded so Airflow can display it, but the Code view shows the entry module for every Dag in the bundle. Say what is embedded and what the Code view shows today. * Warn against defining a Dag in both TypeScript and Python A native TypeScript Dag and a Python stub with the same dag_id in one bundle overwrite each other's Dag on every parse. Tell users to remove the stub when they move a Dag to new Dag(...), as java.rst does. * Import NodeDagImporter at the top of the Node coordinator The importer only imports the coordinator for type checking, so there is no cycle. This matches how JavaCoordinator imports JavaDagImporter. * Let NodeDagImporter source read errors reach the caller Both callers of get_source_code already catch errors and store a placeholder, as they do for the Java and Python importers. Raising reports a broken bundle in one place with one wording. * Pin the Code view to the entry module in Node bundle tests The tests used a Dag's own file as the entrypoint, so a resolver that returned the first dag_source_paths entry would still pass. Use a separate main.ts entrypoint instead. * Hand out the Node Dag importer through get_dag_importer * Let the Node Dag importer find its coordinator The importer names the NodeCoordinator class that parses its files and is registered in a Dag bundle whenever a NodeCoordinator is configured. The coordinator no longer hands it out or picks the task's bundle itself, so a task scans the bundle for its Dag as before. * Run a native TypeScript Dag's own bundle through the native task branch NodeCoordinator builds the command for the bundle its Dag was parsed from with _build_dag_file_command. The runtime reports a Dag the bundle does not declare, so the coordinator does not check the Dag id. * Document how a NodeCoordinator is picked for parsing TypeScript Dags Once a NodeCoordinator is configured, every Dag bundle is parsed. With several, dag_bundle_to_coordinator picks one per Dag bundle. Artifact-only Dag bundles list * in .airflowignore, and a native task runs on the coordinator its queue routes to. * Say tasks list runs a native TypeScript Dag's bundle airflow tasks list parses a native TypeScript Dag, so it is not among the commands that refuse one. Also break the new TypeScript SDK text only at sentence or clause ends. * Show each native Dag's own source in the Code tab (#74249) A packed TypeScript bundle can declare several Dags, each mapped to its own source file. The Code tab showed the bundle's entry module for every one of them: the Dag processor read a bundle's source once per file and stored it under the file's own fileloc, so every Dag sharing that file collided on the same key and only one entry survived. get_source_code now takes the Dag's own dag_id, and NodeDagImporter resolves that Dag's mapped file instead of always falling back to the entry module. Every Dag importer's signature grew the same parameter for a uniform interface; only NodeDagImporter uses it so far. Source code read for a native Dag file is now keyed by dag_id end to end, since a fileloc is not unique per Dag the way a dag_id is. --------- Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:DAG-processing area:task-sdk labels Oct 5, 2026

pierrejeambrun marked this pull request as ready for review October 6, 2026 09:08

pierrejeambrun requested review from amoghrajesh, ashb, ephraimbuddy, jason810496, jedcunningham and uranusjr as code owners October 6, 2026 09:08

jason810496 approved these changes Oct 6, 2026

View reviewed changes

eladkal approved these changes Oct 6, 2026

View reviewed changes

eladkal added this to the Airflow 3.4.0 milestone Oct 6, 2026

jason810496 merged commit ca9bab3 into apache:jason/lang-sdk-e2e/05-node-dag-importer Oct 6, 2026
2 checks passed

jason810496 deleted the lang-sdk-code-tab-per-dag-source branch October 6, 2026 11:23

jason810496 mentioned this pull request Oct 6, 2026

Support optional DagID on the get_source_code signature for better UI code view visibility #74241

Open

1 task done

jason810496 mentioned this pull request Oct 6, 2026

Parse native Go Dags and show each Dag's own source #74341

Merged

1 task done

Navigation Menu

Repository navigation

Show each native Dag's own source in the Code tab - #74249

Show each native Dag's own source in the Code tab#74249
jason810496 merged 1 commit into
apache:jason/lang-sdk-e2e/05-node-dag-importerfrom
astronomer:lang-sdk-code-tab-per-dag-source

pierrejeambrun commented Oct 5, 2026 •

edited by jason810496

Loading

Uh oh!

pierrejeambrun commented Oct 5, 2026

Uh oh!

jason810496 left a comment

Uh oh!

jason810496 Oct 6, 2026

Uh oh!

jason810496 Oct 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-        for dag_id, dag in list(self.dags.items()):
+        # TODO: store the file content more efficient (e.g. if multiple dag_ids will store the exact same DagCode content)
+        # There're two parts: 1) the Dagbag in-memory path 2) the DB persistent layer
+        # The bigger picture might be normalized the DagCode table, so we won't store the exact same DagCode content twice.
+        for dag_id, dag in list(self.dags.items()):

Conversation

pierrejeambrun commented Oct 5, 2026 • edited by jason810496 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes

Out of scope

Was generative AI tooling used to co-author this PR?

Uh oh!

pierrejeambrun commented Oct 5, 2026

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

jason810496 Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

jason810496 Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pierrejeambrun commented Oct 5, 2026 •

edited by jason810496

Loading