Add ADR for Dag importer process model

[dilnazanlid]

This PR introduces ADR-0001 to document the process model and supervision interfaces for AIP-85: Dag Importer.

Key decisions:

1. The importer owns the process. The manager calls a non-blocking submit_import() and supervises the returned ImportHandle. By default the import forks a child (so existing importers keep working unchanged); importing inline is an opt-in for importers that run no author code; language runtimes can use a process they keep warm. This replaces the manager's current special-casing of coordinator importers.
2. The unit of work is the Dag definition, not the file - e.g. each zip member gets its own queue entry and child. Callbacks resolve against their requested bundle version, independent of discovery.
3. Fork and GC handling belong to the framework. A final fork_import() keeps forked children from copying the manager's heap; importer authors never touch the collector.
4. Importers live for the whole Dag processor run, with warm_up() / shut_down() hooks for long-lived state such as a warm runtime.
5. Definitions cross process boundaries as coordinates (a self-describing envelope), not file contents.
6. The handle's result is already serialised (the DagFileParsingResult shape), so no Task SDK objects reach the manager and non-Python runtimes can produce it.
7. Callbacks stay Python-only for now.

The ADR is aligned with the native Lang-SDK Dag parsing (lang-sdk ADR-0010/0012, #74035).

Related to: AIP-85

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5) following the guidelines

[ephraimbuddy]

Thanks for writing this down. The alternatives and explicit tradeoffs make the reasoning behind each decision easier to review.

I support giving importers control over their runtime. I’d like to clarify how that fits with an AIP I’m drafting on running Dag parsing through executors. It hasn’t reached the dev list yet, so please treat it as an early proposal.

Decision 1’s rejected alternative appears to rule out that direction, but I think it addresses a different design. In my proposal, executors would manage placement and capacity, while importers would retain control over the runtime inside each worker. My inline comments focus on making that boundary explicit and identifying the contracts the two proposals would need to agree on.

[pierrejeambrun]

Thanks for the PR

[jason810496]

Thanks a lot for drafting the design!

[jason810496]

From my perspative, the is_ready should be renamed to is_finished. Since we're using "not proc.is_ready" as "finished".

airflow/airflow-core/src/airflow/dag_processing/manager.py

Lines 1375 to 1386 in b6538c3

	def _collect_results(self):
	finished = []
	for file, proc in self._processors.items():
	if not proc.is_ready:
	# This processor hasn't finished yet, or we haven't read all the output from it yet
	continue
	finished.append(file)
	self.handle_parsing_result(file, proc)
	for file in finished:
	processor = self._processors.pop(file)
	processor.close()

airflow/airflow-core/src/airflow/dag_processing/processor.py

Lines 748 to 755 in b6538c3

	@property
	def is_ready(self) -> bool:
	if self._check_subprocess_exit() is None:
	# Process still alive, def can't be finished yet
	return False
	return not self._open_sockets

[dilnazanlid]

I considered both, but still is_ready seems more appropriate. It's the name the existing parse processes already use. DagFileProcessorProcess and LangSDKDagFileProcessorProcess both expose is_ready, and the manager's _collect_results loop polls exactly that property. ImportHandle is a structural protocol, so keeping the name means today's process classes already satisfy it without any renaming or adapter, and the manager loop doesn't change.

WDYT?

[jason810496]

• section title: "Fork mechanics and collector discipline belong to the DagImporter.start_import"
• "long-lived process" -> "ephemeral processor subprocesses"

Additionally, should we show the hierarchy of manager -> start_import -- (processor subprocess) --> fork_import instead of just fork_import?

(Please note that there's another comment for suggestion better naming for the "start_import" method)

[dilnazanlid]

Thanks, I added the call chain to the diagram, so it now shows submit_import() → fork_import() → spawn_child() → parse child.

I kept "long-lived" because there's no processor subprocess in between anymore. The importer runs inside the manager process, so the fork happens from the manager itself, which lives for the whole Dag processor run.

I also kept submit_import out of the title. The GC handling lives in fork_import, which is final, while submit_import is the method importers can override.