Skip to content

fix(openhands): pin the postgres client image used by init containers and jobs - #1386

Draft
dylan-openhands wants to merge 2 commits into
mainfrom
dj/c3-0713-pgclient-main
Draft

dylan-openhands wants to merge 2 commits into
mainfrom
dj/c3-0713-pgclient-main

Conversation

@dylan-openhands

@dylan-openhands dylan-openhands commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

Every database init container and job that runs psql/pg_isready hardcoded its image: bitnamilegacy/postgresql:latest (openhands, runtime-api and plugin-directory wait-for-db/create-db, automation and integrations-hub wait-for-postgres) or postgres:14 (automation and integrations-hub create-db-user, runtime-api create-postgres-user job). Keycloak's wait-for-db was the only one with an override (keycloak.waitForDb.image).

Why this matters:

  • :latest is unpinned, so what customers run changes underneath a release.
  • The hardcoded refs bypass the Replicated proxy (images.r9.all-hands.dev/proxy/...) and the airgap local-registry rewrite. Online installs pull straight from Docker Hub and airgap installs can't pull them at all.
  • They are the largest third-party CVE bucket in an OHE install. Trivy (HIGH/CRITICAL, linux/amd64, DB 2026-10-09) reports 16 CRITICAL and 84 HIGH fixable for bitnamilegacy/postgresql:latest, and 1 CRITICAL and 24 HIGH fixable for postgres:14.

What changes:

  • New global.postgresClientImage: {repository, tag}, pinned by digest. Every site, including Keycloak's wait-for-db (rendered through common.tplvalues.render), reads it. Each subchart carries the same default so it still renders standalone. keycloak.waitForDb.image is removed in favour of the global.
  • replicated/openhands.yaml sets the repository online (images.r9.all-hands.dev/proxy/<appSlug>/docker.io/...) and in the HasLocalRegistry block (<LocalRegistryHost>/<LocalRegistryNamespace>/...). The tag comes from the chart, and the builder render picks the image up for the airgap bundle.
  • Keycloak's wait-for-db has no pod-level runAsUser, and the Bitnami image used to supply uid 1001. The new image defaults to root, so the container now sets runAsUser: 1001 / runAsNonRoot: true to keep the same identity.
  • scripts/test_postgres_client_image.py renders every site, including the non-default branches, and asserts that each psql/pg_isready container uses the mirrored, digest-pinned image in online and airgap modes.

The image is chosen in two commits so reviewers can keep or drop the second:

  1. docker.io/library/postgres:16.15-alpine@sha256:7218…080ea is the Docker Official image, at the bundled server's major (Bitnami 16.4).
  2. Optional: docker.io/alpine/psql:18.6@sha256:08f3…64b8 ships only the client tools. The official image's remaining 25 fixable findings all sit in the Go stdlib of its gosu binary, which none of these containers execute. Drop this commit if policy requires Docker Official Images.
candidate (by digest) fixable C/H total C/H
bitnamilegacy/postgresql:latest (old) 16 / 84 19 / 148
postgres:14 (old) 1 / 24 2 / 87
postgres:16.15-alpine 1 / 24 1 / 24
postgres:16.15-bookworm 1 / 24 4 / 89
postgres:16.15-trixie 1 / 24 2 / 87
alpine/psql:18.6 0 / 0 0 / 0
ghcr.io/cloudnative-pg/postgresql:16-minimal-trixie 0 / 0 1 / 63

Neither new image adds a fixable finding that the old images didn't have.

Testing

  • Per-site docker run: each site's exact rendered command and env (secrets substituted) ran against a throwaway bitnamilegacy/postgresql:16.4.0-debian-12-r14 server, under the rendered pod's uid/gid: 42420 for openhands, automation and integrations-hub; 1000 for runtime-api; 1001 for plugin-directory and Keycloak. Every site ran twice to cover upgrade reruns. That's 12 distinct containers × 2 passes, with the old images, postgres:16.15-alpine, alpine/psql:18.6 and the CNPG image. All 24 runs exited 0 for each image, and each new image's output was byte-identical to the old images' output. The final databases, roles and grants matched.
  • External servers: with both new images, the wait-for-db, create-db and three create-db-user commands also ran against postgres:13 and postgres:18 with PGSSLMODE=require, twice each. All exited 0, and the connections negotiated TLSv1.3.
  • Environment notes: No site uses bash, and alpine/psql has none. No site sets readOnlyRootFilesystem. With an arbitrary uid, HOME=/, and psql runs non-interactively, so it writes no history. There were no locale warnings.
  • Render diff: helm template against origin/main (defaults, and with every site enabled) changes only the image lines and Keycloak's new securityContext. helm lint is clean. The CI helm-unittest suites all pass. pytest scripts -k 'not keycloak' reports 631 passed, against 626 on origin/main; the 5 extra are the new tests.

A 0.71.x backport of the same change exists on dj/c3-0713-pgclient.

Helm Chart Checklist

  • I have tested the chart upgrade path from the previous version (site commands are idempotent across reruns; see above)
  • I have verified backwards compatibility with existing values.yaml configurations (keycloak.waitForDb.image is removed; use global.postgresClientImage)
  • I have updated the chart's README.md if there are any breaking changes or new required values

Additional Notes

keycloak.waitForDb.image was only set by replicated/openhands.yaml in this repo. A direct Helm user who overrode it should set global.postgresClientImage.repository instead.

… and jobs

Every wait-for-db, create-db and create-db-user container hardcoded either bitnamilegacy/postgresql:latest or postgres:14. All but Keycloak's bypassed the Replicated proxy and the airgap registry rewrite, and :latest was unpinned.

Route all of them through global.postgresClientImage, pinned by digest to postgres:16.15-alpine, and mirror it in the online and local-registry Replicated values. Keycloak's wait-for-db keeps uid 1001, which the Bitnami image used to supply.
Optional on top of the previous commit: drop it to stay on the Docker Official postgres:16.15-alpine image.

postgres:16.15-alpine still reports 1 CRITICAL and 24 HIGH fixable findings, all in the Go stdlib of its gosu binary, which none of these containers run. alpine/psql 18.6 ships only the client tools and scans clean (0 CRITICAL/HIGH). psql 18 supports servers back to 9.2, and the image runs every site's command with output identical to the old images against PostgreSQL 13, 16 and 18.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant