Base CI and PROD images on hardened Python images

[potiuk]

The CI image and the default PROD images now consume a Docker Hardened Image for Python on
Debian 13 "trixie" instead of compiling CPython from source on top of debian:bookworm-slim. The previous
way of building stays available for the PROD image as the deprecated legacy flavor.

Discussion: https://lists.apache.org/thread/y0mlxsx36kzc84tn31q1kyl4rdvgv1xx
Lazy consensus: https://lists.apache.org/thread/h0sxvd5g4drxfrph35dvlrk46zomsk0y

Image flavors

	CI image	PROD image (default)	PROD image (legacy)
Python	hardened base image	hardened base image	compiled from python.org sources, sigstore-verified
Debian	trixie	trixie	bookworm (trixie also possible)
BASE_IMAGE	ghcr.io/apache/airflow/base/python:<X.Y.Z>-debian13-dev	same	debian:bookworm-slim

• AIRFLOW_IMAGE_FLAVOR (hardened by default, or legacy) selects the flavor in the PROD Dockerfile. The
  legacy build stage compiles Python into /usr/python and hands it to the final image through
  /python-dist, so the hardened image gets no extra layer content. The flavor is recorded in the
  org.apache.airflow.image.flavor label.
• Breeze: breeze prod-image build --image-flavor hardened|legacy --debian-version trixie|bookworm. The
  legacy flavor builds on debian:<version>-slim. breeze ci-image build no longer takes
  --debian-version - the CI image is always hardened and trixie-based.
• prod-image-build.yml gets an image-flavor input and defaults debian-version to trixie.
• New scheduled-legacy-prod-image-canary.yml runs weekly (and on dispatch) and tests legacy images like
  regular ones. It runs the same selective checks as a scheduled CI build, publishes the constraints from
  the constraints branch as the usual constraints-<python> artifacts (so no CI image is needed), and builds
  legacy bookworm PROD images for every supported Python version through the reusable
  prod-image-build.yml (including prod-image verify). It then checks each image's flavor label and
  Debian codename, and runs the regular PROD image tests: additional-prod-image-tests.yml (docker-compose
  quick start, Task SDK integration, e2e suites, image extra checks) and the Kubernetes tests. Legacy
  images are not built in PRs or canary builds.

Debian trixie

• Runtime libraries renamed in trixie's 64-bit time_t transition are picked per codename
  (libssl3t64, libldap2 on trixie; libssl3, libldap-2.5-0 on bookworm).
• MariaDB publishes 10.11 only up to bookworm, so trixie gets the MariaDB 11.8 LTS repository, plus
  mariadb-client-compat, which provides the mysql named commands from 11.0 on. Bookworm keeps 10.11.
• The Microsoft (MSSQL), PostgreSQL, Docker and Adoptium repositories all publish trixie suites.
• The Debian 13 hardened images also configure Docker's own package repository (http://dhi.io/deb/debian/main,
  signed with /usr/share/keyrings/dhi-deb-main.gpg), whose +dhiN rebuilds take precedence over Debian's
  packages. It is accessible anonymously - apt-get update needs no credentials, unlike pulling the dhi.io
  images. The Debian 12 images have no such repository.
• The Debian 13 hardened images install Python as Debian packages (python-3.13, libpython-3.13, …)
  under /usr, while the Debian 12 ones used /opt/python. The image now takes the base Python's prefix
  from sys.base_prefix, so /usr/python links to /usr on trixie. The system-Python guard only matches
  Debian's own libpython3.X packages, so it no longer fails on the base image's libpython-3.13. That
  guard failure is what broke the first trixie CI image build.

Base image distribution

Pulling dhi.io requires a Docker Hub login. Rather than force credentials on every contributor and CI
job, the tags are mirrored to ghcr.io/apache/airflow/base/python and BASE_IMAGE defaults there, so
building needs no registry credentials at all. The images are Apache-2.0 licensed, so redistribution is
fine. breeze release-management mirror-base-images plus a weekly workflow keep the mirror current; the
mirror publishes the pinned (3.13.16-debian13-dev) and floating (3.13-debian13-dev) tags for both
debian13 and debian12. The debian13 tags are already mirrored, and their digests are identical to
dhi.io's.

One patch-level pin serves both Debian releases. The upgrade check reads Docker's catalog for both and
only moves the pin to a patch level published for each of them, so a bump never points one of them at a
tag that does not exist. Pins: 3.11.17 / 3.12.15 / 3.13.16 / 3.14.8.

Gaps the hardened base has versus debian:*-slim

The hardened images ship a deliberately minimal /etc and toolset. What had to be restored - only for the
hardened flavor - lives in restore_debian_base_files and link_python:

Missing	Symptom	Handling
base-passwd accounts	sasl2-bin: install: invalid group 'sasl'	install base-passwd, run update-passwd
/etc/shells	tmux: add-shell aborts	create it
libpam-runtime	adduser --gecos → chfn: PAM: Critical error	install it; it generates the /etc/pam.d/common-* files the shipped PAM configs @include
dash	libgcrypt20 postinst (#!/bin/dash) exits 127	install it first
/usr/local, gzip	ln: /usr/local/bin/pip3: No such file; tar: gzip: Cannot exec	mkdir -p in link_python; gzip in dev deps

lsb-release is dropped entirely - install_mysql/mssql/postgres.sh now read /etc/os-release through
common::debian_codename / common::debian_release. That package pulled in usrmerge, whose postinst
removed /lib64 and broke the build; it was the blocker in #60123.

Differences between the hardened base image and the previous images

Investigating this PR's CI failures turned up behaviour differences in Docker's hardened Python
(definitions: https://github.com/docker-hardened-images/catalog, package/python and image/python):

• Patched CPython. Docker applies 020-CVE-2026-3479-pkgutil-get-data.patch, so pkgutil.get_data()
  rejects .. resource paths (upstream CPython only documents this). moto < 5.2.2 loads its EC2 data via
  ../resources/…, which broke the lowest-dependency Providers[amazon] tests on every Python version.
  → moto floor raised to 5.2.2.
• 1 MiB thread stack. The hardened Python is built with -DTHREAD_STACK_SIZE=0x100000 (previous
  images: the 8 MiB glibc default). On 3.12/3.13 the fixed C-recursion depth then overflows, crashing an
  xdist worker on test_login.py::…[deeply_nested]. → hardened images install
  airflow-thread-stack-size.pth restoring 8 MiB thread stacks.
• No PGO. Built with --with-lto only (previous: --enable-optimizations --with-lto), so CPU-bound
  Python is slower.
• Modified /etc/debian_version. A base-files upgrade stopped at dpkg's conffile prompt. → hardened
  images ship /etc/dpkg/dpkg.cfg.d/airflow-keep-conffiles (force-confdef, force-confold).
• No which. → common.sh uses command -v.
• No .pyc for the standard library. → the hardened build compiles it, keeping the Do not remove .pyc and .pyo files after building Python #58944 fix.

These are documented in docker-stack-docs/build.rst ("Properties of the hardened base image", plus a new
"Debian trixie and the legacy image flavor" section).

PYTHON_LTO is removed: FIPS builds point BASE_IMAGE at a FIPS variant of the hardened image, and the
legacy flavor always compiles with LTO.

Drive-by fixes

• libxmlsec1-openssl added to the runtime deps. libxmlsec1 ships no crypto engine of its own, so
  import xmlsec (via python3-saml) failed with libxmlsec1-openssl.so.1: cannot open shared object file. Pre-existing - reproduced on debian:bookworm-slim with the same package set.
• scripts/ci/prek/upgrade_important_versions.py reads the supported Python versions from
  global_constants.py instead of a hard-coded list that had left the 3.14 pin stale since March.

Verification

• Rebased onto main (which dropped Python 3.10 since the last run); the previous run's only failure,
  test_local_executor.py::…test_actual_worker_death_after_start_releases_slot, was a timing assertion
  already loosened on main by Increase worker startup timeout in LocalExecutor process death test #74292.
• Package lists dry-run installed in debian:bookworm-slim and debian:trixie-slim; the MariaDB 11.8
  trixie repository resolves the dev and prod client package sets.
• Breeze and scripts unit tests pass, including new tests for the flavor/Debian base image selection,
  the per-distro mirror and the patch-level pin across both catalogs.

Follow-ups deliberately not in this PR

• Publishing the legacy images under the -legacy tag suffix with an entrypoint deprecation warning - part
  of the release tooling, to land before 3.4.0.
• Verifying Docker's signatures/attestations on the hardened images before the mirror copies them; the
  legacy flavor keeps the sigstore verification of the Python sources.
• The shell-less runtime variant for the PROD main stage - it would mean rewriting the entrypoint away
  from bash and dropping the ADDITIONAL_RUNTIME_APT_DEPS customization contract, and it is coupled to
  Helm Chart: Directly Start Airflow Containers #65384.

closes: #59625

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

🤖 Generated with Claude Code

[github-actions]

uv.lock on main just moved via #74429 ("Prepare providers release 2026-10-06"), commit 57ad5d3 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.