Skip to content

Fix Dag processor client compatibility with httpx2 - #74731

Merged
pierrejeambrun merged 2 commits into
apache:mainfrom
hussein-awala:fix-dag-processor-httpx2
Oct 14, 2026
Merged

pierrejeambrun merged 2 commits into
apache:mainfrom
hussein-awala:fix-dag-processor-httpx2

Conversation

@hussein-awala

@hussein-awala hussein-awala commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Use httpx2 throughout the Dag processor Execution API client and its tests, matching the Task SDK base client migrated in #73116. Imports and references use httpx2 explicitly, without an alias.

This fixes both mypy-airflow-core override errors (_update_auth and request) and the runtime mismatch: httpx exception handlers do not catch httpx2 errors, and httpx.Timeout objects are incompatible with the new client. Those failures affect credential renewal, registration retirement, and closed-Job handling.

Why #73116 did not catch this

The two changes landed independently:

The passing pre-merge checks therefore predated this incompatible combination. This is an integration gap between independently validated changes, not an error mypy is unable to detect. The import ban added by #73116 is scoped to task-sdk/pyproject.toml; it does not cover this airflow-core subclass. Git history establishes the branch contents and merge ordering; individual CI job logs were not available through the public checks view.

Validation

  • Reproduced exactly two mypy-airflow-core errors before the production fix; the hook now passes, including during commit. No mypy suppression or hook exclusion was added.
  • Updated the existing tests to use httpx2 responses and transport first: 29 failed, 61 passed with the unfixed production client. After the fix: 90 passed.
  • CI then exposed five failures in the separate HTTP integration test file, which still used httpx transports and exception assertions. Reproduced locally as 5 failed, 3 passed, then migrated those references to httpx2 as well.
  • Targeted validation after that correction: 98 passed on SQLite across both client test files; 8 passed on PostgreSQL and 8 passed on MySQL for the HTTP integration file. After rebasing onto current main, the 98 SQLite tests, regular checks (including mypy), and applicable manual checks passed again. The lowest-dependency CI rerun is still pending.
  • Regular static checks and applicable manual checks passed. Manual checks excluded only the three long-running/opt-in hooks documented in the contribution workflow.
  • Selected core non-database suites on Python 3.11: 5,345 passed, 11 skipped, 2 xfailed.
  • breeze build-docs apache-airflow: passed.
  • Selected SQLite database suites were attempted but the invocation hit its eight-minute execution limit before all suites completed. This is incomplete validation, not a passing result.

Related: #73116, #74370, #74244. The broader core migration in draft #74244 currently does not modify these Dag processor files; this is a targeted fix for the already-merged SDK migration.


Was generative AI tooling used to co-author this PR?
  • Yes — Pi coding agent 1.1.0 (OpenAI)

Generated-by: Pi coding agent 1.1.0 (OpenAI) following the guidelines

@hussein-awala
hussein-awala marked this pull request as ready for review October 14, 2026 13:29

@zach-overflow zach-overflow left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thank you! Also, thanks for adding the detailed analysis of the commit timelines in the PR description.

The Task SDK migration and scoped Dag processor client landed independently. Their combination leaves incompatible response types, timeouts, and exception handlers in the processor client, breaking credential renewal and Job error handling.
The HTTP integration tests still construct httpx transports and expect httpx errors. These do not interoperate with the migrated SDK client, causing five failures across the database and lowest-dependency CI jobs.
@hussein-awala
hussein-awala force-pushed the fix-dag-processor-httpx2 branch from 9723689 to deeb7dd Compare October 14, 2026 14:39

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-4-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ v3-4-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 6a91a17 v3-4-test

This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

@vatsrahul1001 vatsrahul1001 added this to the Airflow 3.5.0 milestone Oct 18, 2026
zozo123 added a commit to zozo123/airflow that referenced this pull request Oct 18, 2026
Picks up the httpx2 Dag processor client fix (apache#74731) that the core test jobs and the
airflow-core mypy check failed on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants