Skip to content

fix(openhands): rebuild 0.71.3 images without Copa - #1387

Draft
dylan-openhands wants to merge 13 commits into
release/0.71from
dj/c3-0713-chart
Draft

dylan-openhands wants to merge 13 commits into
release/0.71from
dj/c3-0713-chart

Conversation

@dylan-openhands

@dylan-openhands dylan-openhands commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Draft. Do not merge as-is. The in-house images are pinned to hotfix PR candidate digests in the TEST-ONLY commit, and agent-canvas's candidate carries the staging PostHog key. Replace them with the rescanned -r1 release digests before merging.

What

0.71.3 for the 0.71.x hotfix line. It starts from 0.71.1's charts/ and replicated/, so none of 0.71.2's Copa-patched images carry forward, then re-pins each image to a properly rebuilt or newer same-line build. .github/ (including the Beta deploy guard) and the chart version (0.71.2 → release-please proposes 0.71.3) are kept from release/0.71.

Commits

Commit Item Change
fix(openhands): restore charts and replicated to 0.71.1 images + test: expect 0.71.1's MinIO image refs again 0 git restore --source=openhands/0.71.1 charts replicated (chart version kept at 0.71.2); the MinIO test goes back to 0.71.1's refs
ci: accept -rN hotfix suffix in the agent-server sync check A check_agent_server_sync.py maps <ver>-rN[@sha256:…] to release <ver> (one regex, 3 tests)
ci: keep image-loader and crd-check on their release tags A The sync check also reads 1.49.5-r1-python as 1.49.5-python, so charts/image-loader and charts/crd-check stay untouched (changing them would make release-please propose releases of those charts on release/0.71)
TEST-ONLY: hotfix candidate pins (replace with -r1 release digests) A 7 in-house images → <ver>-r1@sha256:<hotfix PR candidate> on the default ghcr.io/openhands/* repos
fix(infra): bump cert-manager chart to 1.20.4 F cert-manager v1.20.2 → v1.20.4 (patch; 1.20.3 narrows the cert-manager-edit aggregate role, GHSA-8rvj-mm4h-c258)
fix(infra): rebuild trust-pkg default package image at 20230311-deb12u1.7 F Same CA data, rebuilt image
fix(openhands): bump TLS preflight probe image to alpine/openssl 3.5.9 F Preflight collector only
fix(openhands): bump rancher/kubectl to v1.33.13 C Same minor (also #1385 on main)
fix(openhands): bump litellm-database to 1.100.5 D Same minor, no schema/migration changes between 1.100.1 and 1.100.5; scripts/test_litellm_budget_cache.py asserts the pinned ref
fix(openhands): bump Bitnami postgresql and redis image tags E postgresql 16.4.0-r14 → 16.4.0-r28 (same app version), redis 7.4.1 → 7.4.3 (patch)
fix(openhands): pin the postgres client image used by init containers and jobs B 11 hardcoded bitnamilegacy/postgresql:latest / postgres:14 sites → one global.postgresClientImage, digest-pinned, through the Replicated proxy and in the airgap block; Keycloak wait-for-db gets runAsUser: 1001 to keep its old uid; new scripts/test_postgres_client_image.py (also #1386 on main)
fix(openhands): [optional] use alpine/psql for the postgres client image B Optional: postgres:16.15-alpine (1/24 fixable, all in unused gosu) → alpine/psql:18.6 (0/0). Drop this commit to stay on the Docker Official image

Vulnerability result

The set is every image a default KOTS install runs: the openhands chart, the infra-cert-manager / sysbox / trust-manager charts, and the sandbox image. Scanned with Trivy 0.75 on one frozen DB (2026-10-09T19:06Z), linux/amd64, counting unique (CVE, package).

Fixable critical Fixable high Fixable C+H Total C Total H
0.71.1 144 1827 1971 173 3056
0.71.2 (Copa) 63 1368 1431 95 2664
this PR 92 1022 1114 115 2116

This PR is below 0.71.1 everywhere and below 0.71.2 on high and on C+H, but not on critical. The criticals left are concentrated in Bitnami (Keycloak 23, PostgreSQL 17, Redis 17), MinIO (6 + 3) and sysbox (9). None has a same-line upgrade that scans better; see "Needs a decision".

Third-party images: dispositions

Image Disposition
bitnamilegacy/postgresql:latest, postgres:14 (client sites) Changed → one pinned client image
rancher/kubectl Changed v1.33.0 → v1.33.13
litellm-database Changed 1.100.1 → 1.100.5
Bitnami PostgreSQL Changed 16.4.0-r14 → r28; remaining 17C/67H need a move off Bitnami
Bitnami Redis Changed 7.4.1 → 7.4.3; remaining 17C/74H need Valkey or similar
Bitnami Keycloak 26.3.0 No better option (26.3.1–26.3.3 add new fixable JRE highs); ≥26.7.2 is a one-way DB migration, needs a decision
ohe-minio / ohe-minio-mc No better option (single tag each); MinIO replacement needs a decision
cert-manager (×5), trust-pkg, alpine/openssl Changed (patch)
trust-manager v0.22.1 No better option (only v0.22.0/.1 exist)
alpine/k8s 1.30.0 No better option (all 13 1.30.x tags scanned; none better)
sysbox-deploy-k8s v0.7.0-0 Blocked: no v0.7.0-1; v0.7.1-0 is a sysbox version change (clears 1 critical)
debian:bookworm-slim Kept floating (already the newest published digest)
replicated-sdk 1.19.7 No better option (Trivy finds no packages in 1.19.7 or 1.19.11)

Needs a decision

  • Keycloak ≥26.7.2, Redis → Valkey, PostgreSQL off Bitnami, and a MinIO replacement.
  • Whether to reinstate specific 0.71.2 Copa images, if critical must also beat 0.71.2.
  • Keep or drop the optional alpine/psql commit.

CI status

The three Scan (openhands/…:<ver>-r1@sha256:…) jobs fail with manifest unknown. scripts/wait_for_image_manifest.py looks up the tag in name:tag@digest, and the -r1 tags don't exist yet. The same digest without the tag resolves. Pods pull by digest, so this doesn't affect a running install, but it stays red until the -r1 releases are published and pinned. Every other check passes. Check agent-server sync only runs on the release-please PR; it passes locally against the real enterprise repo.

Before merging (handoff)

  • Publish the -r1 releases (see each component PR). Confirm each tag resolves to a digest, rescan it on the same DB, pin <ver>-r1@sha256:<release digest>, and rerun CI.
  • The enterprise-server candidate is not a pure rebuild of 1.64.0. Node moves from 24.x to 26.x (to clear npm CVEs), the system pip is removed, Python goes 3.13.7 → 3.13.13, and 5 dependencies are bumped. SDK pins are unchanged. Smoke-test it before tagging 1.64.0-r1.
  • Keep ghcr.io/openhands/patched/agent-server@sha256:c143… published. Sandboxes created on 0.71.2 keep that image when resumed after the upgrade.
  • The optional fleet upgrade test (0.71.2 → this) covers what local smoke can't: real PVCs, the Replicated proxy, and login and a conversation.

Testing

  • helm lint passes on every chart, and all 8 helm unittest suites pass. pytest scripts -k 'not keycloak': 540 passed (531 on the restore commit, plus 9 new tests).
  • KOTS-rendered manifests at default config, diffed against 0.71.1, change only image refs, cert-manager labels and RBAC, and Keycloak wait-for-db's securityContext.
  • Every postgres-client site's rendered command was run with docker run as its pod's uid, against a Bitnami 16.4 server and against external postgres:13 and postgres:18 over TLS. Install and rerun-on-upgrade passes produced output identical to the old images.
  • Upgrade smoke in containers (rendered env, uid 1001, read-only root fs). Bitnami PostgreSQL r28 starts on a data dir written by 0.71.2's Copa image and by 0.71.1's r14, with data intact, and rolling back to either works. Redis 7.4.3 runs the rendered start script, and its auth and liveness/readiness scripts pass. litellm 1.100.1 → 1.100.5 on the same DB reports "No pending migrations" (the same 161 migrations), virtual and master keys keep working, and rolling back to 1.100.1 works.
  • Not done: a fleet install/upgrade test.

Base for 0.71.3: 0.71.2's Copa image pins don't carry forward. charts/ and
replicated/ are restored from openhands/0.71.1; .github/ and the chart
version (0.71.2) are kept so release-please cuts 0.71.3.
Follows the restore: 0.71.2 pointed this test at the Copa-patched paths.
Rebuilt hotfix releases (1.64.0-r1, 1.49.5-r1-python) name the release they rebuild; release_tag() now strips -rN as it already strips -patched and digest pins.
Pins the PR-build candidate digests of the 7 in-house images, default ghcr.io/openhands/* repositories. The -r1 tags do not exist yet; the tag text is ignored because a digest is present. Replace each digest with the rescanned -r1 release digest once the hotfix PRs are tagged.

agent-canvas carries the STAGING PostHog key (OpenHands#18258 candidate build) and must never ship: it needs the manual cancel-and-dispatch release before this pin is final.

agent-server tag keeps 1.49.5 as its first segment so enterprise's get_agent_server_image() rewrite (default repository only) leaves it unchanged.
… and jobs

Every wait-for-db, create-db and create-db-user container hardcoded either bitnamilegacy/postgresql:latest or postgres:14. All but Keycloak's bypassed the Replicated proxy and the airgap registry rewrite, and :latest was unpinned.

Route all of them through global.postgresClientImage, pinned by digest to postgres:16.15-alpine, and mirror it in the online and local-registry Replicated values. Keycloak's wait-for-db keeps uid 1001, which the Bitnami image used to supply.
Optional on top of the previous commit: drop it to stay on the Docker Official postgres:16.15-alpine image.

postgres:16.15-alpine still reports 1 CRITICAL and 24 HIGH fixable findings, all in the Go stdlib of its gosu binary, which none of these containers run. alpine/psql 18.6 ships only the client tools and scans clean (0 CRITICAL/HIGH). psql 18 supports servers back to 9.2, and the image runs every site's command with output identical to the old images against PostgreSQL 13, 16 and 18.
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Chart appVersion has drifted from the image tag

The openhands chart's appVersion no longer matches the deployed image tag. These are meant to stay in sync.

Location Value
charts/openhands/Chart.yaml → appVersion 1.64.0
charts/openhands/values.yaml → image.tag 1.64.0-r1@sha256:6c137216f1c6afe1d560300e92990a52050064a1358472c314285ae8e0bf8307

Fix: set appVersion to 1.64.0-r1@sha256:6c137216f1c6afe1d560300e92990a52050064a1358472c314285ae8e0bf8307 in charts/openhands/Chart.yaml.

This is a notice, not a blocking check.

@github-actions github-actions Bot added the type: fix A bug fix label Oct 10, 2026
Changing either chart makes release-please propose a release of it on
release/0.71. The sync check now reads 1.49.5-r1-python as 1.49.5-python,
so image-loader can stay on the release tag, as it did for 0.71.2. The
crd-check edit was a doc comment (the chart is consumed as OCI 0.1.0).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant