[v3-4-test] Migrate Task SDK to httpx2 (#73116)

[github-actions]

• Depend on httpx2 instead of httpx in the Task SDK

httpx has stalled: fixes that landed in httpx2 (a fork of httpx 0.28.1)
have not shipped upstream, and starlette has already migrated, so FastAPI
following is a matter of time. Moving the Execution API client now keeps
the SDK on a maintained HTTP stack.

Objects do not cross the httpx/httpx2 boundary, so the in-process
Execution API transports move with the client that consumes them; the
rest of airflow-core stays on httpx and migrates separately.

related: #70522

• Add newsfragment for the Task SDK httpx2 migration

• Match the in-process test client transport to the installed Task SDK

Compat tests pair the repo's devel-common helpers with a released Airflow
whose Task SDK is still on httpx. Pinning the helper to httpx2 hands those
clients a transport from the wrong package, and httpx rejects the response
stream it gets back.

• Rename newsfragment to the new PR number

• Pin the Task SDK ast-grep hook to a frozen revision

Every other third-party hook in the repo pins an immutable SHA; a mutable
tag lets the hook change under CI without a diff.

• Remove unneeded newsfragment

• Verify Task SDK HTTP logging and simplify test helpers

Both HTTP stacks can coexist while compatibility jobs use older SDK
distributions. The migration needs regression coverage for its logging
behavior without exposing development-only helpers as public APIs.

• Use httpx2 in the Dag run note client test

• Use httpx2 in the execution API shutdown tests

Tests added on main for in-process shutdown still referenced httpx, which this branch no longer imports in the module.

• Trust the system certificate store in the Task SDK API client

httpx2 verifies against the system trust store via truststore, so keeping certifi for the Execution API client would add a direct dependency only to maintain a second, different set of trusted roots in the same worker.

• Ban httpx imports in the Task SDK with ruff

Ruff already enforces banned imports, so a separate ast-grep hook and its extra pre-commit dependency are unnecessary. extend-banned-api adds to the root banned-api table instead of replacing it.

• Use httpx2 in the server termination acknowledgement test

The test arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports.

• Test the Task SDK SSL context against a real CA file

Mocking load_verify_locations while also writing a CA file to disk tested neither path convincingly. Keeping the certificate-generation imports inside the fixture limits them to the one place that needs them.

• Note when the in-process httpx resolver can be removed

• Apply suggestion from @ashb

Co-authored-by: Ash Berlin-Taylor ash_github@firemirror.com

• Use httpx2 in the supervisor launch error test helper

The helper arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports.

• Remove the duplicated in-process httpx resolver note

• Use httpx2 in the supervisor execution timeout test

The test arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports.

• Use httpx2 in the request handler tests

The tests arrived on main after the Task SDK moved to httpx2; the client rejects httpx transports.

---

(cherry picked from commit 63f5845)

Co-authored-by: Zach zach.gottesman@datadoghq.com
Co-authored-by: Ash Berlin-Taylor ash_github@firemirror.com