feat(clp-package): Parameterize clp-package Docker image by Ubuntu codename; auto-detect host version. by junhaoliao · Pull Request #2273 · y-scope/clp

junhaoliao · 2026-05-11T04:00:34Z

Description

Previously, the clp-package Docker image was hardcoded to ubuntu:jammy. Binaries built on an Ubuntu 24.04 (Noble) host (glibc 2.39) cannot run inside a Jammy container (glibc 2.35), causing task to fail on Noble hosts at the package-image stage. This affects both the C++ binaries (which require GLIBC_2.38) and Python packages with native extensions (e.g., _mysql_connector.so, psutil, greenlet) that are compiled on the host and linked against its glibc.

This PR makes two changes to tools/docker-images/clp-package/:

Dockerfile — The base image is now parameterized via ARG UBUNTU_VERSION_CODENAME=jammy so FROM ubuntu:${UBUNTU_VERSION_CODENAME} resolves to the host-matching Ubuntu version. Additionally, userdel -r ubuntu is called before useradd clp-user to handle the default ubuntu user present in Noble cloud images (UID 1000), which would otherwise conflict with the clp-user UID.
build.sh — Auto-detects the host's Ubuntu codename from /etc/os-release (VERSION_CODENAME) and passes it as --build-arg UBUNTU_VERSION_CODENAME=<codename>. Defaults to jammy when detection fails, preserving backward compatibility.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Scenario 1: Full build on Noble host

Task

Run task on an Ubuntu 24.04 (Noble) host to verify the build succeeds end-to-end.

Command

task

Output

task: [docker-images:package] ./build.sh
...
#4 [internal] load metadata for docker.io/library/ubuntu:noble
...
#30 exporting to image
#30 exporting layers done
...
task: [package] echo '0.12.1-dev' > '/home/ubuntu/workspace/clp/build/clp-package/VERSION'

Explanation

The Docker image build resolved the base image to ubuntu:noble, and the full task pipeline completed successfully on a Noble host.

Scenario 2: build.sh auto-detects the host Ubuntu codename

Task

Verify that build.sh reads the host's codename from /etc/os-release and passes it as a build-arg.

Command

bash -x tools/docker-images/clp-package/build.sh 2>&1 | grep -E "ubuntu_version|host_codename|UBUNTU_VERSION"

Output

+ ubuntu_version_codename=jammy
+ host_codename=noble
+ ubuntu_version_codename=noble
+ build_cmd+=(--build-arg "UBUNTU_VERSION_CODENAME=${ubuntu_version_codename}")
+ docker build --pull ... --build-arg UBUNTU_VERSION_CODENAME=noble ...

Explanation

The script defaults to jammy but correctly overrides with the host's VERSION_CODENAME=noble, ensuring the runtime image's glibc matches the host.

Scenario 3: Package Dockerfile builds with UBUNTU_VERSION_CODENAME=noble

Task

Verify the Dockerfile accepts --build-arg UBUNTU_VERSION_CODENAME=noble and pulls ubuntu:noble as the base image.

Command

docker build --check --build-arg UBUNTU_VERSION_CODENAME=noble -f tools/docker-images/clp-package/Dockerfile . 2>&1

Output

Check complete, no warnings found.

Explanation

Docker resolved the base image to docker.io/library/ubuntu:noble, confirming the build-arg is correctly substituted into the FROM directive.

Scenario 4: Package Dockerfile syntax check with default (jammy)

Task

Verify backward compatibility — the Dockerfile still works with the default jammy codename.

Command

docker build --check --build-arg UBUNTU_VERSION_CODENAME=jammy -f tools/docker-images/clp-package/Dockerfile . 2>&1

Output

Check complete, no warnings found.

Explanation

The Dockerfile passes syntax validation with the default jammy value, confirming no breaking change for existing builds.

Scenario 5: Container runs Noble and has correct user setup

Task

Verify the built container is based on Ubuntu Noble and the ubuntu user is removed while clp-user exists with UID 1000.

Command

docker run --rm <image> cat /etc/os-release | head -5
docker run --rm <image> id clp-user
docker run --rm <image> id ubuntu

Output

PRETTY_NAME="Ubuntu 24.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.4 LTS (Noble Numbat)"
VERSION_CODENAME=noble

uid=1000(clp-user) gid=1000(clp-user) groups=1000(clp-user)

id: 'ubuntu': no such user

Explanation

The container runs Ubuntu Noble; the default ubuntu user (present in Noble cloud images, UID 1000) was removed, and clp-user was created with UID 1000, avoiding the UID conflict that would otherwise occur.

Scenario 6: Start CLP package on Noble host

Task

Verify start-clp.sh starts all CLP services successfully.

Command

cd build/clp-package && ./sbin/start-clp.sh

Output

2026-05-11T03:36:54.358 INFO [controller] Setting up environment for compression_scheduler...
...
Container clp-package-ec9d-webui-1 Healthy
...
2026-05-11T03:37:06.327 INFO [controller] Started CLP.

Explanation

All services (database, queue, redis, results-cache, compression worker/scheduler, query worker/scheduler, api-server, webui, reducer, garbage-collector) started and became healthy.

Scenario 7: Compress logs

Task

Verify log compression works end-to-end in the Noble-based package.

Command

cd build/clp-package && ./sbin/compress.sh --timestamp-key timestamp ~/samples/postgresql.jsonl

Output

2026-05-11T03:55:03.410 INFO [compress] Compression job 5 submitted.
2026-05-11T03:55:05.415 INFO [compress] Compressed 389.02MB into 10.05MB (38.70x). Speed: 203.97MB/s.
2026-05-11T03:55:05.916 INFO [compress] Compression finished.
2026-05-11T03:55:05.916 INFO [compress] Compressed 389.02MB into 10.05MB (38.70x). Speed: 184.50MB/s.

Explanation

Compression completed successfully with a 38.70x compression ratio, confirming that the Noble-based runtime image correctly executes the CLP binaries built on the Noble host.

Scenario 8: Search logs

Task

Verify log search works end-to-end in the Noble-based package.

Command

cd build/clp-package && ./sbin/search.sh --count "error"

Output

(Container started, search completed, exit code 0)

Explanation

Search completed successfully, confirming the query pipeline works correctly with the Noble-based package.

Summary by CodeRabbit

Chores
- Docker image build now supports selecting the Ubuntu base image at build time for greater flexibility.
- Build script auto-detects the host Ubuntu codename and forwards it to the image build to improve compatibility.
- CI workflow and build action updated to accept and pass the base image codename.
- Image user initialization improved to detect and handle existing user IDs, failing safely if conflicts cannot be resolved.

…me; auto-detect host version.

coderabbitai · 2026-05-11T04:00:47Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 883e97f9-e2f3-4036-a4bf-f9f161579c17

📥 Commits

Reviewing files that changed from the base of the PR and between 7782bc1 and 1da7e74.

📒 Files selected for processing (3)

.github/actions/clp-build-runtime-image/action.yaml
.github/workflows/clp-artifact-build.yaml
tools/docker-images/clp-package/Dockerfile

Walkthrough

Docker build is parameterized by UBUNTU_VERSION_CODENAME; Dockerfile now removes any existing account with the requested UID before creating the runtime user. The build script detects the host Ubuntu codename and passes it as a build-arg, and the GitHub Action/workflow expose and forward the same codename.

Changes

Docker Build Parameterization

Layer / File(s)	Summary
Dockerfile base image parameterization `tools/docker-images/clp-package/Dockerfile`	`ARG UBUNTU_VERSION_CODENAME=jammy` added and used in `FROM ubuntu:${UBUNTU_VERSION_CODENAME}` to make the base image configurable.
Dockerfile user initialization `tools/docker-images/clp-package/Dockerfile`	Pre-`useradd` step added: lookup existing owner of `${UID}` and run `userdel --remove` (build fails if deletion cannot be performed) before creating `${USER}`.
Build script Ubuntu version detection `tools/docker-images/clp-package/build.sh`	Detect host Ubuntu codename from `/etc/os-release` (default `jammy`) and append `--build-arg UBUNTU_VERSION_CODENAME=...` to `docker build`.
GitHub Action input and workflow wiring `.github/actions/clp-build-runtime-image/action.yaml`, `.github/workflows/clp-artifact-build.yaml`	Add required action input `ubuntu_version_codename` and pass it as a Docker build-arg; workflow invocation supplies `ubuntu_version_codename: "jammy"`.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: parameterizing the Ubuntu codename and auto-detecting the host version in the clp-package Docker image.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Bill-hbrhbr · 2026-05-16T14:22:47Z

    USER="clp-user"

-RUN useradd --uid ${UID} --shell /bin/bash --home-dir ${CLP_HOME} ${USER}
+RUN userdel -r ubuntu 2>/dev/null || true \


Instead of hardcoding ubuntu, we should resolve the username from the UID, using something like getent passwd "${UID}" | cut -d: -f1.

Also, is it possible to raise an error when encountering a UID conflict?

sg. i tried to address it in 5af1885 . let me know if i understand correctly

Bill-hbrhbr · 2026-05-16T14:37:43Z


+ubuntu_version_codename="jammy"
+if [[ -f /etc/os-release ]]; then
+    host_codename="$(. /etc/os-release && echo "$VERSION_CODENAME")"


Right now, task package builds a clp-package image using the same Ubuntu release as the host invoking the task command. However, I believe a 22.04 host should still be able to build a 24.04 clp-package image.

Should we expose the Ubuntu version codename as a Task argument or environment variable so the package target is not tied to the host release?

dynamic ubuntu version detection on host should make more sense. the main reason why we want to avoid hardcoding a version of the ubuntu image is because whatever (Python wheels, glibc dependent binaries, etc.) a non-Jammy host builds do not run inside the Jammy-based image container.

the long-term plan is to migrate the base image from Ubuntu to Muslinux / Manylinux as @jackluo923 proposed. we (or whoever has bandwidth) can work on it in the next month. also, i think we shall explore setting up https://containers.dev/ too.

for now, this PR can unblock our developers to develop on different versions of Ubuntu hosts, without use of Ubuntu Jammy build containers

let me know what you think

just realized that tools/docker-images/clp-package/Dockerfile copies built artifacts instead of building the package itself. Yes I agree we should keep host and host-built container image consistent.

…_version_codename

Bill-hbrhbr

Need to update docs:

clp/docs/src/dev-docs/building-package.md

Line 8 in 137af2c

* An x86_64 Ubuntu 22.04 (Jammy) machine or container

Bill-hbrhbr · 2026-05-19T18:15:20Z

    default: "amd64"
    required: false


Suggested change

default: "amd64"

required: false

required: true

Maybe open an issue to make this required?

Bill-hbrhbr · 2026-05-19T18:27:23Z


+ubuntu_version_codename="jammy"
+if [[ -f /etc/os-release ]]; then
+    host_codename="$(. /etc/os-release && echo "$VERSION_CODENAME")"


just realized that tools/docker-images/clp-package/Dockerfile copies built artifacts instead of building the package itself. Yes I agree we should keep host and host-built container image consistent.

Bill-hbrhbr · 2026-05-19T18:30:23Z

+RUN existing_user="$(getent passwd "${UID}" | cut --delimiter=: --fields=1)" || true; \
+    [ -z "${existing_user}" ] || userdel --remove "${existing_user}" \
+    || { echo "ERROR: Cannot delete user '${existing_user}' that occupies UID ${UID}"; exit 1; }
 RUN useradd --uid ${UID} --shell /bin/bash --home-dir ${CLP_HOME} ${USER}


To make the user handling more coherent, we can maybe merge these two inside a single RUN. Your choice.

feat(package): Parameterize clp-package Docker image by Ubuntu codena…

7782bc1

…me; auto-detect host version.

junhaoliao requested a review from a team as a code owner May 11, 2026 04:00

junhaoliao requested a review from Bill-hbrhbr May 11, 2026 04:00

junhaoliao changed the title ~~feat(package): Parameterize clp-package Docker image by Ubuntu codename; auto-detect host version.~~ feat(clp-package): Parameterize clp-package Docker image by Ubuntu codename; auto-detect host version. May 11, 2026

junhaoliao requested a review from jackluo923 May 12, 2026 10:12

junhaoliao force-pushed the main branch from 4505f8c to 8977316 Compare May 13, 2026 22:48

Merge branch 'main' into fix-noble

351588b

Bill-hbrhbr requested changes May 16, 2026

View reviewed changes

junhaoliao added 2 commits May 17, 2026 23:51

fix(package): Dynamically detect and remove user occupying target UID

5af1885

refactor(package): Simplify runtime image action inputs to use ubuntu…

1da7e74

…_version_codename

junhaoliao requested a review from Bill-hbrhbr May 18, 2026 04:06

Bill-hbrhbr requested changes May 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(clp-package): Parameterize clp-package Docker image by Ubuntu codename; auto-detect host version.#2273

feat(clp-package): Parameterize clp-package Docker image by Ubuntu codename; auto-detect host version.#2273
junhaoliao wants to merge 4 commits into
y-scope:mainfrom
junhaoliao:fix-noble

junhaoliao commented May 11, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Bill-hbrhbr May 16, 2026

Uh oh!

junhaoliao May 18, 2026

Uh oh!

Bill-hbrhbr May 16, 2026

Uh oh!

junhaoliao May 18, 2026

Uh oh!

Bill-hbrhbr May 19, 2026

Uh oh!

Bill-hbrhbr left a comment

Uh oh!

Bill-hbrhbr May 19, 2026

Uh oh!

Bill-hbrhbr May 19, 2026

Uh oh!

Bill-hbrhbr May 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

junhaoliao commented May 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Scenario 1: Full build on Noble host

Task

Command

Output

Explanation

Scenario 2: build.sh auto-detects the host Ubuntu codename

Task

Command

Output

Explanation

Scenario 3: Package Dockerfile builds with UBUNTU_VERSION_CODENAME=noble

Task

Command

Output

Explanation

Scenario 4: Package Dockerfile syntax check with default (jammy)

Task

Command

Output

Explanation

Scenario 5: Container runs Noble and has correct user setup

Task

Command

Output

Explanation

Scenario 6: Start CLP package on Noble host

Task

Command

Output

Explanation

Scenario 7: Compress logs

Task

Command

Output

Explanation

Scenario 8: Search logs

Task

Command

Output

Explanation

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bill-hbrhbr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

junhaoliao commented May 11, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 11, 2026 •

edited

Loading