CNTRLPLANE-2938: KMS plugin certification requirements by kevinrizza · Pull Request #2062 · openshift/enhancements

kevinrizza · 2026-07-09T20:20:18Z

Update enhancement proposal to include plugin certification requirements for image signature verification and testing binary requirements.

Supercedes #2029

openshift-ci-robot · 2026-07-09T20:20:22Z

@kevinrizza: This pull request references CNTRLPLANE-2938 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Update enhancement proposal to include plugin certification requirements for image signature verification and testing binary requirements.

Supercedes #2029

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-07-09T20:20:40Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dgoodwin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2026-07-09T20:33:47Z

@kevinrizza: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

tjungblu · 2026-07-10T07:30:02Z

+
+###### Overriding Configuration
+
+There are cases where a user needs the ability to override this configuration in order to be able to use an unsupported plugin image -- for example if there is a hotfix that needs to be handed to a user urgently or even in certain testing  scenarios. We will enable this by adding an override field to the UnsupportedConfigOverrides cluster API that will disable the deployment of the ClusterImagePolicy. This will put the cluster in an unsupported state that will need to be resolved before upgrading the cluster, but it will allow a user a temporary workaround to running unsupported or unverified plugin images.


if the customer manages the ClusterImagePolicy, they also have two options:

remove the ClusterImagePolicy

update the public key to match the hotfix version

Not sure we would need to create an override in this case

tjungblu · 2026-07-10T07:33:49Z

+      matchPolicy: MatchRepository
+```
+
+In order for the API Server Operators to validate that the installed KMS plugin is certified and supported by OCP, it must be defined in a well known way. The fields that match must include the following:


I get that we want to also validate that this has been created, it's certainly better than "trust me bro"...

We would need 1:n support (for the transition case below), so we would need to hardcode a set of policy names (?) we expect? Or is this just structural - in which case any image policy would match?

Can an image have more that one .sig tag ? If yes, then maybe key rotation could be backward compatible.

We just need to validate that there is a policy specified, not the specific policy type. As long as we are verifying that there is a signature required to pull the image we should satisfy the problem. The only other requirement is that the scope is set to a well known path so that we restrict the location that we are pulling the image from just to a place that we know is supported. That field we will compare to a well defined value.

just in plain terms:

admin configures a vault kms image in the encryption cfg

aggregate apiserver operator(s) list all cluster image policies

... and verify that there is a policy.scope that is matching the configured image

is that it?

tjungblu · 2026-07-10T07:36:47Z

+
+###### Disconnected Concerns
+
+When a cluster is disconnected from external networks or airgapped, the ClusterImagePolicy defined by the cluster will not be sufficient because the cluster will not have access to an external registry to pull the image. OpenShift has tools and APIs that enable core component images and layered operators to be mirrored into disconnected environments in mirror registries (oc-mirror, IDMS, ITMS). In a future iteration, we will update oc-mirror to understand how to mirror KMS plugins and provide the correct set of manifests to apply to the cluster so that plugins can be run out of the box in a seamless experience. For now, we are putting that work out of scope of this initial enhancement.


@sherine-k sorry to tag you directly and I hope you're still working on oc-mirror - is this something that is feasible from your end?

Hi @tjungblu, I'm from the oc-mirror team. Are KMS plugins packaged as container images? Are they components of an Openshift release or are they part of operator catalog or helm charts?

Hey Alex! In the current version of this proposal, they are container images that are configured to run alongside the API Servers. That data would be packaged somehow as part of the OpenShift payload. Part of this proposal is about defining where and how that metadata is stored / configured.

If this is going to follow the same standard as the other components and it will be referenced in the image-references, oc-mirror will be capable of recognizing it and mirror its container images.

@aguidirh AFAIK, oc-mirror already generates IDMS/ITMS and is capable of mirroring signatures alongside their images. Right?
Does oc-mirror generate ClusterImagePolicy for disconnected clusters already? or is this the gap that probably will be needed?

oc-mirror does not generate ClusterImagePolicies today, but an openshift ClusterImagePolicy is baked into the OCP payload and applied by the cluster-version operator today, to cover scopes relevant to OCP payloads. The mechanics are discussed in an existing enhancement. The hard-coded canonical pullspecs in the openshift ClusterImagePolicy and in the Sigstore signatures themselves work in disconnected/restricted-network environments, because the image is retrieved via its canonical pullspec, and CRI-O consumes the ImageDigestMirrorSets to find those images in the local mirrors (which must also host the Sigstore signatures). oc-mirror creates those ImageDigestMirrorSets, and helps ensure that the local mirrors have both the OCP-payload images and the Sigstore images that sign those OCP-payload images.

tjungblu · 2026-07-10T07:39:20Z

+After successful completion, vendors should provide:
+1. Full test output log (`kms-test-output.log`)
+2. JUnit XML files from the `junit/` directory
+3. OpenShift version and KMS plugin version tested
+4. Any test failures with root cause analysis and remediation plans


talking about "trust me bro", how do we prevent folks from just submitting all junit tests as PASSED?

p0lyn0mial · 2026-07-10T11:46:10Z

+
+Additionally, users must set the `spec.policy.rootOfTrust.publicKey.keyData` field to the base64 encoded public key that the image was signed with. Users should refer to the vendor's official documentation for the most current public key to use in their `ClusterImagePolicy`.
+
+When vendors release images signed with a new key, users must create a new `ClusterImagePolicy` resource with the updated public key. During the transition period, both old and new `ClusterImagePolicy` resources must exist in the cluster until the previous `ClusterImagePolicy` is no longer needed (i.e when migration to the new plugin image is finished).


I got the impression that the operator will ensure that the proper key is used in the policy, which comes from the release payload.

However, this paragraph suggests that there will be cases where the operator does not actually validate the origin of a plugin.

Is that correct?

We don't really need to validate the key matches what we expect, we just need to validate that there is a key and that it is used for a specific image repository. Previously we discussed the idea that we would have to package it into the operator, but that was for mechanical reasons (we wanted the UX of this getting generated automatically so a user doesn't have to deal with it). Since that can't happen either way because of the problems that poses to disconnected workflows, we might as well let a user specify it at runtime (or, eventually, by a mirroring tool automatically).

See here for my concerns about simultanous-ClusterImagePolicies as a key-rotation process. High level summary is that my current read of the code is that you'd need the signer to backsign existing images with the new key and to continue to sign new images with the old key for however long it took the fleet to transition to the new images. I think a convenient solution would be to propose an extension to the (Cluster)ImagePolicy API to support "signed by any key from a set" semantics, which RHEL/CRI-O already support since RHEL-22610 / containers/image#2526.

p0lyn0mial · 2026-07-10T12:00:45Z

+
+Additionally, users must set the `spec.policy.rootOfTrust.publicKey.keyData` field to the base64 encoded public key that the image was signed with. Users should refer to the vendor's official documentation for the most current public key to use in their `ClusterImagePolicy`.
+
+When vendors release images signed with a new key, users must create a new `ClusterImagePolicy` resource with the updated public key. During the transition period, both old and new `ClusterImagePolicy` resources must exist in the cluster until the previous `ClusterImagePolicy` is no longer needed (i.e when migration to the new plugin image is finished).


Can an image have more that one .sig tag ? If yes, then maybe key rotation could be backward compatible.

p0lyn0mial · 2026-07-10T12:10:08Z

+
+When that override is set, users will be able to apply KMS plugin configuration to the cluster without image verification.
+
+###### Disconnected Concerns


it seems like the operator could read the IDMS/ITMS resources to discover the mirror path, which would allow the operator to fully own the policy.

isn't IDMS/ITMS the standard way to configure image mirroring in OpenShift?

do users already create these resources as part of a disconnected installation?

@sherine-k maybe you could advice on ^

if we require IDMS/ITMS and have the operator read them, then the operator has everything it needs to fully manage the ClusterImagePolicy.

p0lyn0mial · 2026-07-10T12:12:32Z

+```
+metadata.name
+spec.scopes
+spec.policy.rootOfTrust.policyType


do i understand correctly that the operator will also verify keyData against the pub key from the release payload ?

No, the operator just needs to know that a policy is specified and that a key is required. We will rely on the policy itself to enforce that it is valid.

p0lyn0mial · 2026-07-10T12:27:48Z

+
+Additionally, users must set the `spec.policy.rootOfTrust.publicKey.keyData` field to the base64 encoded public key that the image was signed with. Users should refer to the vendor's official documentation for the most current public key to use in their `ClusterImagePolicy`.
+
+When vendors release images signed with a new key, users must create a new `ClusterImagePolicy` resource with the updated public key. During the transition period, both old and new `ClusterImagePolicy` resources must exist in the cluster until the previous `ClusterImagePolicy` is no longer needed (i.e when migration to the new plugin image is finished).


Since the metadata.name is unique we cannot have two policies with the same name.

sherine-k · 2026-07-13T17:23:12Z

+    config.openshift.io/type: kms
+spec:
+  scopes:
+    - mirror.internal.registry.com:5000/internal/mirrored-vault-plugin-kms # mirrored KMS plugin image


Question: Here we're not talking about fetching the publicKey for the rootOfTrust from any KMS or anything, correct?
And we are talking about fetching images for the KMS plugin only, right?
We're not talking about online signature verification (i.e. providing a way to reach the sigstore components (Rekor, TUF, Fulcio, etc...) for signature verification: by default the signature verification is offline and doesn't require connections to sigstore.)

sherine-k · 2026-07-13T17:37:04Z

+
+###### Disconnected Concerns
+
+When a cluster is disconnected from external networks or airgapped, the ClusterImagePolicy defined by the cluster will not be sufficient because the cluster will not have access to an external registry to pull the image. OpenShift has tools and APIs that enable core component images and layered operators to be mirrored into disconnected environments in mirror registries (oc-mirror, IDMS, ITMS). In a future iteration, we will update oc-mirror to understand how to mirror KMS plugins and provide the correct set of manifests to apply to the cluster so that plugins can be run out of the box in a seamless experience. For now, we are putting that work out of scope of this initial enhancement.


@aguidirh AFAIK, oc-mirror already generates IDMS/ITMS and is capable of mirroring signatures alongside their images. Right?
Does oc-mirror generate ClusterImagePolicy for disconnected clusters already? or is this the gap that probably will be needed?

sherine-k

Hi
see inline comments

flavianmissi and others added 2 commits July 8, 2026 13:11

update kms EP with plugin image certification and signature verification

f3079e0

kms: require signature verification for plugin deployment

4467ba1

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 9, 2026

openshift-ci Bot requested review from jaypoulz and travier July 9, 2026 20:20

tjungblu reviewed Jul 10, 2026

View reviewed changes

p0lyn0mial reviewed Jul 10, 2026

View reviewed changes

sherine-k reviewed Jul 13, 2026

View reviewed changes


		###### Overriding Configuration

		There are cases where a user needs the ability to override this configuration in order to be able to use an unsupported plugin image -- for example if there is a hotfix that needs to be handed to a user urgently or even in certain testing scenarios. We will enable this by adding an override field to the UnsupportedConfigOverrides cluster API that will disable the deployment of the ClusterImagePolicy. This will put the cluster in an unsupported state that will need to be resolved before upgrading the cluster, but it will allow a user a temporary workaround to running unsupported or unverified plugin images.


		###### Disconnected Concerns

		When a cluster is disconnected from external networks or airgapped, the ClusterImagePolicy defined by the cluster will not be sufficient because the cluster will not have access to an external registry to pull the image. OpenShift has tools and APIs that enable core component images and layered operators to be mirrored into disconnected environments in mirror registries (oc-mirror, IDMS, ITMS). In a future iteration, we will update oc-mirror to understand how to mirror KMS plugins and provide the correct set of manifests to apply to the cluster so that plugins can be run out of the box in a seamless experience. For now, we are putting that work out of scope of this initial enhancement.


		Additionally, users must set the `spec.policy.rootOfTrust.publicKey.keyData` field to the base64 encoded public key that the image was signed with. Users should refer to the vendor's official documentation for the most current public key to use in their `ClusterImagePolicy`.

		When vendors release images signed with a new key, users must create a new `ClusterImagePolicy` resource with the updated public key. During the transition period, both old and new `ClusterImagePolicy` resources must exist in the cluster until the previous `ClusterImagePolicy` is no longer needed (i.e when migration to the new plugin image is finished).


		When that override is set, users will be able to apply KMS plugin configuration to the cluster without image verification.

		###### Disconnected Concerns

Uh oh!

Conversation

kevinrizza commented Jul 9, 2026

Uh oh!

openshift-ci-robot commented Jul 9, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Jul 9, 2026

Uh oh!

openshift-ci Bot commented Jul 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tjungblu Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sherine-k left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

openshift-ci-robot commented Jul 9, 2026 •

edited by openshift-ci Bot

Loading

tjungblu Jul 10, 2026 •

edited

Loading