Repository navigation
List only .zip archives as packaged Dags - #74033
Closed
jason810496 wants to merge 2 commits into
Closed
jason810496 wants to merge 2 commits into
jason810496 wants to merge 2 commits into
Conversation
jason810496
added this pull request to stack #73978
October 1, 2026 13:34
This was referenced Oct 1, 2026
This was referenced Oct 1, 2026
jason810496
force-pushed
the
jason/core-taskhandler-refactor/11-candidate-listing-fast-path
branch
from
October 1, 2026 14:44
4392e00 to
efd9eb0
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/12-zip-dag-discovery
branch
from
October 1, 2026 14:44
675d9c8 to
22de37d
Compare
jedcunningham
requested changes
Oct 1, 2026
jedcunningham
left a comment
Member
There was a problem hiding this comment.
This should get a significant newsfragment as it could potentially be breaking for some folks.
jason810496
force-pushed
the
jason/core-taskhandler-refactor/12-zip-dag-discovery
branch
from
October 2, 2026 00:54
22de37d to
f81c9cb
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/11-candidate-listing-fast-path
branch
from
October 2, 2026 00:54
efd9eb0 to
3e03718
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/12-zip-dag-discovery
branch
from
October 2, 2026 05:48
f81c9cb to
e963a67
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/11-candidate-listing-fast-path
branch
from
October 2, 2026 05:48
3e03718 to
1c5b1ca
Compare
1 task done
jason810496
force-pushed
the
jason/core-taskhandler-refactor/12-zip-dag-discovery
branch
from
October 2, 2026 11:36
e963a67 to
745d796
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/11-candidate-listing-fast-path
branch
from
October 2, 2026 11:36
1c5b1ca to
19e73b5
Compare
A JAR is a zip archive, so every task-handler JAR and dependency JAR in a Dag bundle was listed as a Dag file. Each one got a parse child every min_file_process_interval that found no importer for its suffix, and each bundle refresh opened it and checked every member. Dag discovery now lists only what an importer can import.
jason810496
force-pushed
the
jason/core-taskhandler-refactor/11-candidate-listing-fast-path
branch
from
October 2, 2026 14:57
19e73b5 to
d238a18
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/12-zip-dag-discovery
branch
from
October 2, 2026 14:57
745d796 to
bd1191b
Compare
A packaged Dag stored without the .zip suffix is no longer discovered, so call it out for users upgrading.
1 task done
This was referenced Oct 3, 2026
Member
Author
|
Closing this: #74020 made it unnecessary. Dag files are now listed through the Task SDK importers by suffix, where only a |
jason810496
removed this pull request from stack #73978
October 6, 2026 02:27
jason810496
deleted the
jason/core-taskhandler-refactor/12-zip-dag-discovery
branch
October 9, 2026 19:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom to top): #73970, #73971, #73972, #73973, #73974, #73975, #73976, #73977, #74030, #74031, #74032, #74033, #74067
Why
Dag discovery lists every zip archive as a packaged Dag, whatever its name. A JAR is a zip archive, so the Lang-SDK task-handler JARs and their dependency JARs in a Dag bundle are all listed as Dag files. Each gets a parse child every
min_file_process_intervalthat finds no importer for.jar, and each bundle refresh opens it and checks every member. That work is wasted.What changes
A zip archive is listed as a packaged Dag only when its name ends in
.zip, so listing matches what an importer can import. This applies tolist_py_file_paths, which the Dag processor uses to list a bundle's files.DagBagalready lists by suffix through the Task SDK importers (#73853), where only.zipgoes to the zip importer..pyfiles are listed as before.dags.rstnow says a packaged Dag's file name must end in.zip.might_contain_dagheuristic and the manager's per-archive member scan are unchanged.How to test
Ran:
test_file.pyandtest_dagbag.py: 96 passed, 1 skipped.test_manager.py: 208 passed, 6 skipped (Postgres and MySQL only)..jararchive and a zip archive with no suffix are listed next to the.zipand.ZIPones.prek run --fileson the changed files passed, and so didmypy-airflow-core.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines