Repository navigation
Fix Dag processor client compatibility with httpx2 - #74731
Conversation
zach-overflow
left a comment
There was a problem hiding this comment.
LGTM and thank you! Also, thanks for adding the detailed analysis of the commit timelines in the PR description.
The Task SDK migration and scoped Dag processor client landed independently. Their combination leaves incompatible response types, timeouts, and exception handlers in the processor client, breaking credential renewal and Job error handling.
The HTTP integration tests still construct httpx transports and expect httpx errors. These do not interoperate with the migrated SDK client, causing five failures across the database and lowest-dependency CI jobs.
9723689 to
deeb7dd
Compare
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 6a91a17 v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
Picks up the httpx2 Dag processor client fix (apache#74731) that the core test jobs and the airflow-core mypy check failed on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Use
httpx2throughout the Dag processor Execution API client and its tests, matching the Task SDK base client migrated in #73116. Imports and references usehttpx2explicitly, without an alias.This fixes both
mypy-airflow-coreoverride errors (_update_authandrequest) and the runtime mismatch:httpxexception handlers do not catchhttpx2errors, andhttpx.Timeoutobjects are incompatible with the new client. Those failures affect credential renewal, registration retirement, and closed-Job handling.Why #73116 did not catch this
The two changes landed independently:
httpx2#73116's final head wasb312d3df05ecc229a8fcc2106b252cfa2ef459d4. Neitherairflow-core/src/airflow/dag_processing/api_client.pynor its test file exists in that tree. Its visible successful AMD CI run started on October 13, before the new processor client merged.a73ffaedbfon October 14 at 07:40 UTC, using the then-currenthttpxSDK client.httpx2#73116 merged as63f5845dd2at 09:27 UTC that day, changing the SDK base client tohttpx2without migrating the newly added subclass.The passing pre-merge checks therefore predated this incompatible combination. This is an integration gap between independently validated changes, not an error mypy is unable to detect. The import ban added by #73116 is scoped to
task-sdk/pyproject.toml; it does not cover thisairflow-coresubclass. Git history establishes the branch contents and merge ordering; individual CI job logs were not available through the public checks view.Validation
mypy-airflow-coreerrors before the production fix; the hook now passes, including during commit. No mypy suppression or hook exclusion was added.httpx2responses and transport first: 29 failed, 61 passed with the unfixed production client. After the fix: 90 passed.httpxtransports and exception assertions. Reproduced locally as 5 failed, 3 passed, then migrated those references tohttpx2as well.main, the 98 SQLite tests, regular checks (including mypy), and applicable manual checks passed again. The lowest-dependency CI rerun is still pending.breeze build-docs apache-airflow: passed.Related: #73116, #74370, #74244. The broader core migration in draft #74244 currently does not modify these Dag processor files; this is a targeted fix for the already-merged SDK migration.
Was generative AI tooling used to co-author this PR?
Generated-by: Pi coding agent 1.1.0 (OpenAI) following the guidelines