Skip to content

List only .zip archives as packaged Dags - #74033

Closed
jason810496 wants to merge 2 commits into
jason/core-taskhandler-refactor/11-candidate-listing-fast-pathfrom
jason/core-taskhandler-refactor/12-zip-dag-discovery
Closed

jason810496 wants to merge 2 commits into
jason/core-taskhandler-refactor/11-candidate-listing-fast-pathfrom
jason/core-taskhandler-refactor/12-zip-dag-discovery

Conversation

@jason810496

@jason810496 jason810496 commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Stack (bottom to top): #73970, #73971, #73972, #73973, #73974, #73975, #73976, #73977, #74030, #74031, #74032, #74033, #74067

Why

Dag discovery lists every zip archive as a packaged Dag, whatever its name. A JAR is a zip archive, so the Lang-SDK task-handler JARs and their dependency JARs in a Dag bundle are all listed as Dag files. Each gets a parse child every min_file_process_interval that finds no importer for .jar, and each bundle refresh opens it and checks every member. That work is wasted.

What changes

A zip archive is listed as a packaged Dag only when its name ends in .zip, so listing matches what an importer can import. This applies to list_py_file_paths, which the Dag processor uses to list a bundle's files. DagBag already lists by suffix through the Task SDK importers (#73853), where only .zip goes to the zip importer. .py files are listed as before.

dags/
  etl.py                  listed
  packaged.zip            listed
  reports.ZIP             listed
  handlers/etl.jar        no longer listed
  handlers/archive        no longer listed (a zip archive with no suffix)
  • dags.rst now says a packaged Dag's file name must end in .zip.
  • The might_contain_dag heuristic and the manager's per-archive member scan are unchanged.

How to test

uv run --project airflow-core --with-editable shared/secrets_masker pytest airflow-core/tests/unit/utils/test_file.py airflow-core/tests/unit/dag_processing/test_dagbag.py airflow-core/tests/unit/dag_processing/test_manager.py -q

Ran:

  • test_file.py and test_dagbag.py: 96 passed, 1 skipped. test_manager.py: 208 passed, 6 skipped (Postgres and MySQL only).
  • The new test fails without the change: a .jar archive and a zip archive with no suffix are listed next to the .zip and .ZIP ones.
  • prek run --files on the changed files passed, and so did mypy-airflow-core.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5.5) following the guidelines

@jedcunningham jedcunningham left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should get a significant newsfragment as it could potentially be breaking for some folks.

@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/12-zip-dag-discovery branch from 22de37d to f81c9cb Compare October 2, 2026 00:54
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/11-candidate-listing-fast-path branch from efd9eb0 to 3e03718 Compare October 2, 2026 00:54
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/12-zip-dag-discovery branch from f81c9cb to e963a67 Compare October 2, 2026 05:48
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/11-candidate-listing-fast-path branch from 3e03718 to 1c5b1ca Compare October 2, 2026 05:48
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/12-zip-dag-discovery branch from e963a67 to 745d796 Compare October 2, 2026 11:36
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/11-candidate-listing-fast-path branch from 1c5b1ca to 19e73b5 Compare October 2, 2026 11:36
A JAR is a zip archive, so every task-handler JAR and dependency JAR in a Dag bundle was listed as a Dag file. Each one got a parse child every min_file_process_interval that found no importer for its suffix, and each bundle refresh opened it and checked every member. Dag discovery now lists only what an importer can import.
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/11-candidate-listing-fast-path branch from 19e73b5 to d238a18 Compare October 2, 2026 14:57
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/12-zip-dag-discovery branch from 745d796 to bd1191b Compare October 2, 2026 14:57
A packaged Dag stored without the .zip suffix is no longer discovered, so call it out for users upgrading.
@jason810496

Copy link
Copy Markdown
Member Author

Closing this: #74020 made it unnecessary. Dag files are now listed through the Task SDK importers by suffix, where only a .zip archive goes to the zip importer, and the two functions this PR edited (list_py_file_paths and find_dag_file_paths) are gone from main. Thanks @jedcunningham for the review: the significant newsfragment you asked for is no longer needed, since no behaviour change ships from this PR.

@jason810496 jason810496 closed this Oct 6, 2026
@jason810496
jason810496 removed this pull request from stack #73978 October 6, 2026 02:27
@jason810496
jason810496 deleted the jason/core-taskhandler-refactor/12-zip-dag-discovery branch October 9, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants