Repository navigation
Validate a serialized Dag and fill its unset settings from the Airflow config - #74041
Merged
jason810496 merged 10 commits intoOct 5, 2026
Merged
Conversation
This was referenced Oct 1, 2026
jason810496
added this pull request to stack #74044
October 1, 2026 15:18
jason810496
requested review from
guan404ming,
henry3260,
pierrejeambrun and
uranusjr
October 1, 2026 16:52
jason810496
marked this pull request as ready for review
October 1, 2026 16:52
kaxil
reviewed
Oct 1, 2026
jason810496
requested review from
amoghrajesh,
ephraimbuddy,
jedcunningham and
potiuk
as code owners
October 2, 2026 00:53
jason810496
force-pushed
the
jason/lang-sdk-e2e/03b-serialized-dag-validation
branch
2 times, most recently
from
October 2, 2026 04:07
3570480 to
ad14011
Compare
Member
There was a problem hiding this comment.
Do we need prek run sync-go-sdk-schemas --hook-stage manual ? (I checked the code and not automatically syncing Java and Go schemas seems deliberate because it need a maintainer decision, but the CI doesn't warn about it and its really easy to miss the step).
I'll open a PR so at least the drift is caught so the author is asked to take a look instead of silently failing. Maybe something like #74091
LGTM beside kaxil comment and 1 suggestion as well.
Member
Author
Actually, I just noted down the ideal solution to prevent the schema drift in a proper way in #74047 (comment). TL;DR; having bi-weekly job to notify us there's a drift between the Python source and the vendored one. Yes, I will run the |
jason810496
force-pushed
the
jason/lang-sdk-e2e/03b-serialized-dag-validation
branch
from
October 2, 2026 14:07
0afbeb1 to
c930dd6
Compare
jason810496
commented
Oct 2, 2026
jason810496
left a comment
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Thanks for the review
Base automatically changed from
jason/lang-sdk-e2e/03a-dag-processor-base
to
main
October 2, 2026 15:09
jason810496
force-pushed
the
jason/lang-sdk-e2e/03b-serialized-dag-validation
branch
from
October 2, 2026 15:09
c930dd6 to
a07d7ad
Compare
kaxil
reviewed
Oct 2, 2026
DagSerialization.validate_serialized_dag checks the JSON schema, deserializes a copy and looks for a cycle in the task graph. It raises DeserializationError naming the Dag, and for a cycle a task on it. It is meant for a serialized Dag that no Python code built, such as one a Lang-SDK runtime returns, so the Python SDK's cycle check never ran on it. The cycle check runs the shared detect_cycle over the serialized downstream edges.
A Lang-SDK runtime cannot read the Airflow config, so it leaves out the Dag settings that a Python Dag reads from it when unset. The new DagSerialization.fill_config_defaults fills max_active_tasks, max_active_runs, max_consecutive_failed_dag_runs, catchup and disable_bundle_versioning in a serialized Dag from the config. A value the Dag sets is kept.
The schema put additionalProperties on the tasks array, which JSON
Schema ignores, so a task entry like {} passed validation. Each entry
must now be an operator with a string task_id, and
validate_serialized_dag rejects a repeated task id. A schema error
now names the failing field's path.
Also pin both boolean config defaults to different values in the
fill_config_defaults tests, and cover the unwrapped cause of a
deserialization error.
receive() now calls DagSerialization.fill_config_defaults and validate_serialized_dag, so the conformance job runs the same checks as the Dag processor and the config-backed field list lives in one place.
These tests passed the dag_maker proxy of the serialized Dag to LazyDeserializedDAG.from_dag, which serialized each task as its repr. The tasks schema now rejects that, so they serialize the SDK Dag.
validate_serialized_dag now applies the max_active_runs and catchup rules the SDK Dag runs when it is built, to the Dag it deserializes. The cycle check walks that Dag's tasks, so edges from the legacy _downstream_task_ids key and from client_defaults count too. It returns the deserialized Dag.
jason810496
removed this pull request from stack #74044
October 3, 2026 17:58
jason810496
force-pushed
the
jason/lang-sdk-e2e/03b-serialized-dag-validation
branch
from
October 3, 2026 17:58
a07d7ad to
962acf2
Compare
jason810496
added this pull request to stack #74170
October 3, 2026 17:59
jsonschema 4.26 writes $.dag.tasks[2]['__type'] where 4.23 writes $.dag.tasks[2].__type, so the import error and three tests changed with the installed version. Build the path from the error's absolute path.
jason810496
force-pushed
the
jason/lang-sdk-e2e/03b-serialized-dag-validation
branch
from
October 4, 2026 08:51
a729bf8 to
a7efc7c
Compare
jason810496
commented
Oct 4, 2026
jason810496
left a comment
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Thanks for the review!
uranusjr
approved these changes
Oct 5, 2026
jason810496
deleted the
jason/lang-sdk-e2e/03b-serialized-dag-validation
branch
October 5, 2026 03:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom to top): #74041, #74042, #74035, #74043, #74036, #74037, #73845, #73846, #73847
Why
A Lang-SDK runtime returns its Dags already serialized, so Airflow has to check them and fill what the runtime cannot know before storing them.
What changes
DagSerialization.validate_serialized_dag(data)checks a serialized Dag against the JSON schema, rejects duplicate task ids, checksmax_active_runs,catchupand cycles as the SDK does, and returns the loadedSerializedDAG.DagSerialization.fill_config_defaults(data)fills an unsetmax_active_tasks,max_active_runs,max_consecutive_failed_dag_runs,catchupanddisable_bundle_versioningfrom[core] max_active_tasks_per_dag,[core] max_active_runs_per_dag,[core] max_consecutive_failed_dag_runs_per_dag,[scheduler] catchup_by_defaultand[dag_processor] disable_bundle_versioning, as a Python Dag does. A value the Dag sets is kept.airflow-core/adr/lang-sdk/0004-dag-parsing.mdsays Airflow fills an unset field from its config.Was generative AI tooling used to co-author this PR?