[bug]: dev images: dhi.io/deb publishing gaps and unpinned stock Debian fallback cause recurring apt dependency failures

[ucam-rh841]

Docker Hardened Image

dhi.io/python:3.14-debian13-dev

Bug Description

Downstream builds that run apt-get install on DHI -dev images fail intermittently with unmet exact-version dependencies between +dhi and stock Debian packages. The same digest-pinned image fails on one day and succeeds on another, with no change on the consumer side.

This has been reported before and fixed per incident (#481, #543, discussion #544). This issue is about the underlying pattern, and asks whether it can be prevented structurally.

Apt configuration shipped in the dev image

Three sources, all floating, none snapshotted:

Source	Suites
http://deb.debian.org/debian	trixie, trixie-updates
http://deb.debian.org/debian-security	trixie-security
http://dhi.io/deb/debian/main	trixie

• /etc/apt/preferences.d/ is empty, so all origins are at priority 500 and apt selects by version number alone.
• The Debian entries include commented-out snapshot.debian.org references (20251208T000000Z). The dhi.io entry has no equivalent.
• dhi.io/deb retains historical builds (e.g. perl and perl-base from +dhi0 through +dhi+deb13u1+dhi2).

Failure mechanism

1. The dev image ships +dhi builds of core packages, e.g. perl-base, gcc-14-base, libc6.
2. When a consumer installs a package with an exact-version dependency on one of those (e.g. perl requires perl-base (= ${binary:Version})), apt takes the newest candidate of that package across all origins.
3. If dhi.io/deb does not have a build of that package matching the installed +dhi sibling at that moment, the candidate is stock Debian, which requires the stock version (e.g. perl-base (= 5.40.1-6) while 5.40.1-6+dhi7 is installed).
4. apt does not fall back to an older matching build, and will not downgrade an essential package, so the install fails with "held broken packages".
5. Once dhi.io/deb publishes the matching build, the same install succeeds again.

Supporting observations (2026-09-25, linux/amd64)

• Mixed origins in a normal install. 5 packages resolve to stock Debian (krb5-locales, libglib2.0-data, publicsuffix, xauth, xdg-user-dirs).
• Partial rebuilds of a source package. libglib2.0-0t64 resolves to dhi.io while libglib2.0-data resolves to stock. libkrb5-3 resolves to dhi.io while krb5-locales resolves to stock. These succeed only because the stock binaries have no exact-version dependency on their +dhi siblings. Where one exists, the install breaks.

Impact

• Digest-pinning a DHI dev image does not make downstream builds reproducible.
• CI fails intermittently for reasons not visible from the consumer's repository.
• Each occurrence currently needs a manual republish, after a user has noticed and reported it.

Questions / requests

1. Publishing: Are image releases and dhi.io/deb publishes coordinated? Can all binaries built from a source package be published together, and before (or together with) any image that ships them?
2. Snapshots: Is there, or could there be, a snapshotted or versioned dhi.io/deb endpoint (similar to snapshot.debian.org) that consumers can pin to, ideally referenced from the image as the Debian sources are?
3. Partial rebuilds: Is the stock Debian fallback for binaries not rebuilt by DHI intentional? If so, could a CI check ensure no stock binary in the resolved set has an exact-version dependency on a +dhi binary?
4. Security ordering: libkrb5-3 1.21.3-5+dhi2 (dhi.io) sorts above 1.21.3-5+deb13u1 (Debian security). Can you confirm +dhiN builds include the corresponding +debNNuN security fixes, so the version ordering does not mask a Debian update?

Workaround in use

Per #481, an /etc/apt/preferences.d pin can steer individual packages, but it has to be written per package and after the fact. We would prefer a supported fix upstream.

Steps to Reproduce

The failure only occurs while dhi.io/deb is out of step with the image, so it cannot be triggered on demand. The steps below show the conditions that cause it, followed by the error seen during a publishing gap.

1. Start a shell in the dev image:

   docker run --rm -it --user root --entrypoint sh dhi.io/python:3.14-debian13-dev

2. Check the apt configuration. Note three floating sources and no preferences:

   cat /etc/apt/sources.list.d/*
   ls -la /etc/apt/preferences.d/

3. Update the package lists:

   apt-get update

4. Simulate a common install and list the packages resolved from stock Debian rather than dhi.io:

   apt-get install --simulate git libpango-1.0-0 | grep '^Inst' | grep -v 'DHI:'

   On 2026-09-25 this returned krb5-locales, libglib2.0-data, publicsuffix, xauth and xdg-user-dirs, i.e. a mix of origins within the same source packages (krb5, glib2.0).

5. Check the installed +dhi core packages against the versions available:

   apt-cache policy perl perl-base gcc-14-base

   Today the installed perl-base and gcc-14-base match the newest dhi.io builds, and a matching perl is available, so the install succeeds.

6. During a publishing gap (as in [bug]: debian-base trixie apt broken dependencies #543), the same install fails with errors of the form:

   The following packages have unmet dependencies:
    perl : Depends: perl-base (= 5.40.1-6) but 5.40.1-6+dhi7 is to be installed
    libpango-1.0-0 : Depends: ... gcc-14-base (= 14.2.0-19) but 14.2.0-19+dhi3 is to be installed
   E: Unable to correct problems, you have held broken packages.
   

Expected Behavior

Installing packages with apt-get install on a DHI -dev image should resolve consistently for a given image digest. Specifically:

• Every +dhi package shipped in the image should have its exact-version siblings (binaries built from the same source package) available at a matching version in dhi.io/deb whenever the image is available.
• apt should not resolve a stock Debian binary that has an exact-version dependency on a +dhi package installed in the image.
• A digest-pinned image should install the same packages successfully over time, or there should be a supported way (e.g. a snapshotted dhi.io/deb) to achieve this.

Actual Behavior

Installs intermittently fail with unmet dependencies and E: Unable to correct problems, you have held broken packages., when dhi.io/deb does not have a +dhi build of a package matching the +dhi sibling installed in the image. apt then selects the stock Debian build, whose exact-version dependency cannot be satisfied. For example:

• perl requires perl-base (= 5.40.1-6), but 5.40.1-6+dhi7 is installed
• libpango-1.0-0 pulls in a dependency requiring gcc-14-base (= 14.2.0-19), but 14.2.0-19+dhi3 is installed

The same install on the same image succeeds once dhi.io/deb publishes the matching build. This has occurred at least twice (#481, #543) and was fixed each time by a manual republish.

Even when installs succeed, packages resolve from a mix of dhi.io and stock Debian within the same source package (e.g. glib2.0, krb5), so the result depends on the state of both repositories at build time.

Environment

• Image: dhi.io/python:3.14-debian13-dev
• Digest: sha256:42cd56dede69350b250398097287cbf1020d0ead0ad8cb4e179bd3bac1a98634
• Architecture: linux/amd64
• Tested: 2026-09-25
• Also affects: likely all Debian 13 -dev images sharing the same apt configuration (see [bug]: debian-base trixie apt broken dependencies #543 for debian-base)
• Host: Docker on macOS (not relevant to the issue, as the failure is within the container's apt resolution)

Relevant Logs

Additional Context

No response

Pre-submission Checklist

• I have searched existing issues to ensure this bug hasn't been reported before
• I have provided all the requested information above
• I have tested this with the latest available version of the hardened image