I think this repo should have a bit more structure/procedures

[cgwalters]

Hi, thank you for maintaining shared infrastructure in the CNCF.

We are hitting issues around disk space with the default GHA runners, and I went to investigate the state of the CNCF provided runners - see bootc-dev/bootc#2306

I think that will work for us.

However, the first thing that I found a bit of a minor problem is that the naming of the runners like cncf-ubuntu-8-32-x86 replicates a problem from stock Github runners - it's missing the operating system major version which is a very relevant thing.

Still to this day, ubuntu-latest on Github is not latest, because they can't change it because it would break too many things (see actions/runner-images#10636 )

It seems to be that "ubuntu" here is really ubuntu-24.04 - which is fine, however, it's already 2026...and from their side 26.04 is already out...

---

OK so the next thing I went to do is find the pull request that introduced these runners...and then (my agent) noticed there wasn't a pull request at all - b785890 AFAICS just got directly pushed to main.

I think this repository should change to meet basic guidelines:

• All changes require a pull request
• All changes require a reviewer
• (optional but a really good idea) All changes have an AI review by default

I think those of us who are maintaining projects in the CNCF could probably get pinged periodically to review PRs here if that was helpful.

[cgwalters]

cc @koksay

[koksay]

@cgwalters Thanks for raising this. I understand and agree with the general guideline. When we initially implemented this, we chose the direct commit approach to unblock the work and keep the implementation moving, especially given the coordination overhead across time zones.

We'll discuss this internally and determine the most appropriate path forward.

[koksay]

btw, you can always ping us on the #cncf-ci-infra channel on Slack.

[jeefy]

@cgwalters Great feedback and I think we're at a point where we can meet those repo-guidelines. Early on we were kinda pushing hard to get stable and onboard folks but now we can slow down.

As for runner naming, we should be able to support multiple OS versions by using runner tags. That way, the "default" still mirrors GitHub. For example, runs-on: cncf-ubuntu-8-32-x86 would go to our ubuntu-latest, but runs-on: cncf-ubuntu-8-32-x86,ubuntu-26.04 would allow you to target the shape and a specific Ubuntu version we offer.

Thoughts on that?

ref: https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/use-in-a-workflow#using-custom-labels-to-route-jobs

[cgwalters]

Hi @jeefy thanks for your reply and working on this.

@cgwalters Great feedback and I think we're at a point where we can meet those repo-guidelines. Early on we were kinda pushing hard to get stable and onboard folks but now we can slow down.

I totally understand. Some days it definitely feels annoying to have a relatively simple PR just sit there for 3 days because the other maintainer missed it or was on PTO etc...

(Also on bootc we take a step further and enable the rule that pushes to a PR reset approvals)

But I also think at least with LLMs some of this is mitigatable - whole other topic there.

For example, runs-on: cncf-ubuntu-8-32-x86 would go to our ubuntu-latest,

I would argue against replicating the ubuntu-latest problem. What just works better is having stable versioned streams, with Renovate/Dependabot style bumps of those.

Since in the CNCF it's a closed ecosystem, we could actually automatically deploy a renovate-style thing that auto-bumped people's GHA using our runners to new major versions.

[Luap99]

I would argue against replicating the ubuntu-latest problem. What just works better is having stable versioned streams, with Renovate/Dependabot style bumps of those.

Since in the CNCF it's a closed ecosystem, we could actually automatically deploy a renovate-style thing that auto-bumped people's GHA using our runners to new major versions.

I agree with that, we (Podman) got broken by image updates before, see https://cloud-native.slack.com/archives/C08P4HUFQ6M/p1781269458107629

In general we find these totally custom ubuntu images unsuited for most integration level testing anyway (outdated packages, no selinux, etc...) and resort to building static qcow2 images with all our dependencies included (to get rid of flaky/slow runtime installs) and then running them via lima on the ubuntu runner instead. These are our build scripts: https://github.com/podman-container-tools/automation/tree/main/images
Which however means we need nested virt support and arm does not offer it so we cannot test the same there. And then the other problem since neither lima nor qemu directly is part of the ubuntu base image we still have to rely runtime installs which flake...

It would be much better if we could provide our own custom oracle cloud image of our own choice with all the tools we need. Then we do not need nested VMs and no runtime installs. While more work on the infrastructure side it should help to make the CI jobs faster and more stable and thus reduce the overall job runtime. Yes it does mean you cannot keep hot spares for each and every image ready but since we already have to wait for the VM boot time in our workflow waiting for that ahread of time should not make the total CI any worse.

[Luap99]

Another thing is I see changes like f1de260 which seems to downgrade the performance of these runners without communicating such change in advance.

That at least caused another CI problem for us in container-libs as all of the sudden many tests hit the 20 minute job timeout. That timeout exists because the default 6 hour timeout from github is totally unreasonable and waste CI time if a test actually ever hangs or loops and on the other side because we want "fast" CI feedback, most jobs should not get that long in general so we rather fail them if someone adds a really slow test case.

So now to work around the bad performance we now need larger runners instead likely making the costs worse as a some steps are single threaded.
podman-container-tools/container-libs#1026

Now I get that some changes must be made and performance is hard to know in advance but at the minimum I would like such changes to be communicated in some way, at least in the cncf-ci-infra slack channel. Otherwise you ask all users of the runners to chase down why their test times got worse all the sudden.

[cgwalters]

Another thing is I see changes like f1de260 which seems to downgrade the performance of these runners without communicating such change in advance.

Right, that's another good example.

That at least caused another CI problem for us in container-libs as all of the sudden many tests hit the 20 minute job timeout. That timeout exists because the default 6 hour timeout from github is totally unreasonable and waste CI time if a test actually ever hangs or loops and on the other side because we want "fast" CI feedback, most jobs should not get that long in general so we rather fail them if someone adds a really slow test case.

(Personally I'd manage that via soft not hard failures, i.e. trigger a warning and have a periodic (LLM-assisted most likely) flow that triages and looks for slow tests)

[Luap99]

(Personally I'd manage that via soft not hard failures, i.e. trigger a warning and have a periodic (LLM-assisted most likely) flow that triages and looks for slow tests)

Well sure but that can only solve the second concern, the primary reason for timeouts is to not have test hang and consume the runner resources for 6 hours. The only way to solve that is a hard failure. And if I have a hard limit I might as well set it to something I think the job must never exceed.