Skip to content

Fix Python task startup with large integers in Dag run conf - #74797

Closed
ChahatKumar wants to merge 3 commits into
apache:mainfrom
ChahatKumar:pr-73783-surgical-fixes
Closed

ChahatKumar wants to merge 3 commits into
apache:mainfrom
ChahatKumar:pr-73783-surgical-fixes

Conversation

@ChahatKumar

@ChahatKumar ChahatKumar commented Oct 15, 2026 •

Copy link
Copy Markdown

This supersedes #73783 from a writable fork and includes the original implementation together with the failing-test correction, rebased onto current main.

Python tasks fail before their task body runs when Dag run conf contains an integer such as 18446744073709551616. The Execution API preserves the value, but encoding the supervisor startup message raises OverflowError because it exceeds MessagePack's native integer range.

This preserves those values across Python subprocess communication using a MessagePack extension with a signed binary payload. Both directions restore the original Python int, including nested values. External language-SDK subprocesses retain their existing behavior.

The additional test fix accounts for MySQL JSON's integer range: values below signed 64-bit minimum or above unsigned 64-bit maximum remain covered on PostgreSQL and SQLite, while MySQL continues to cover its native boundaries and a large numeric string.

Validation on the exact current-main branch in the Airflow Linux CI image:

  • Task SDK coordinator, communication, and supervisor suites: 496 passed, 1 skipped
  • SQLite large-integer Execution API regression cases: 5 passed
  • Earlier focused MySQL regression: 3 passed, 2 correctly backend-skipped
  • Ruff formatting and lint passed
  • git diff --check passed

The PR changes exactly eight files covering the original feature and its tests; no Dag Processor compatibility files are included because current main already contains #74731.

closes: #73712


Was generative AI tooling used to co-author this PR?
  • Yes — GitHub Copilot

@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:DAG-processing area:task-sdk labels Oct 15, 2026
@boring-cyborg

boring-cyborg Bot commented Oct 15, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

dhkim1920 and others added 3 commits October 15, 2026 16:48
Dag run configuration can contain integers beyond the native MessagePack range. These values currently fail before the Python task body runs, even though the Execution API preserves them.
Keep the shared wire identifier explicit so the encoder and decoder remain aligned during future protocol changes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ashb

ashb commented Oct 15, 2026

Copy link
Copy Markdown
Member

@ChahatKumar There was activity on this PR yesterday by the original poster, so no, opening a new PR is not appropriate.

@ashb ashb closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:DAG-processing area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CeleryExecutor task workload serialization fails with OverflowError for integers outside 64-bit range (msgspec msgpack)

3 participants