Repository navigation
Pass Dag runs directly to Deadline references - #70714
imrichardwu wants to merge 13 commits into
Conversation
boring-cyborg
Bot
added
area:dag-processor
area:deadline-alerts
e9c48fd to
3b99b93
Compare
|
@ferruzzi can I get a review? |
3b99b93 to
db15212
Compare
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
|
Heads up on sequencing: this PR now overlaps two others in
This PR fixes the same The cheapest merge order looks like it will be to merge #70370 first, then rebase here. #70370 adds |
|
Adding a fourth PR to the picture since it breaks something here concretely. #71802 adds a core-side That will break Also worth noting, #71850's diff still has |
7893875 to
efb1bb9
Compare
|
Oh and also a rebase is needed |
|
Looks good pending CI, @ferruzzi |
|
Looks good to me. @ferruzzi can we merge? I think it just needs a rebase, but I think you can do that from your side if @imrichardwu doesn't get to it first. I want to merge this asap because it's blocking #71850 |
9013e64 to
352d438
Compare
|
@imrichardwu thanks for getting to it so quickly! Keeping an eye on CI and will try to get it merged when it's all green |
Preserve the Dag run evaluation context for custom references that accept additional keyword arguments, so they continue to evaluate correctly.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
The custom deadline reference examples returned the dagrun attribute directly, which read as if the attribute was the required return value rather than an input available to the author's business logic. Type the evaluation dagrun with the shared DagRunProtocol so authors get a concrete contract for what is available, and show the attribute feeding into business logic in the examples.
A custom deadline reference whose _evaluate_with declares **kwargs is expected to receive the full evaluation context, matching the standard Python "accept anything" convention rather than having the context filtered away. Align the test with that behaviour.
Clarify the supported Dag run interface for deadline references and document the migration away from required_kwargs before Airflow 4.0.
Updated the DeadlineDagRunProtocol class to eliminate its inheritance from DagRunProtocol, simplifying the interface for deadline references. The class now directly defines the necessary attributes for evaluating deadlines without relying on the ORM DagRun structure.
…evaluations This update introduces warning logs in the SerializedReferenceModels for cases where a DagRun's logical date or queued at time is missing. Additionally, the test suite has been enhanced to verify that these warnings are correctly logged when the respective attributes are None.
…adline evaluations
352d438 to
9d83e30
Compare
|
#70370 just merged which forces a rebase here. @imrichardwu thanks for getting it up to date, when CI is green this will be merged |
This branch has not been deployed
4 participants
Preserve the Dag run evaluation context for custom references that accept additional keyword arguments, so they continue to evaluate correctly.
Pass the in-memory Dag run to Deadline references so built-in references avoid redundant database reads and custom references can evaluate the current state.
Deprecate
required_kwargswhile preserving it for existing custom references. Use the shared evaluator when recalculating queued-at deadlines.closes: #70710
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.