CNTRLPLANE-2938: KMS plugin certification requirements

enhancements/kube-apiserver/kms-encryption-foundations.md

@@ -724,6 +724,309 @@ These rollup conditions (`KMSPluginsDegraded`, and any future `KMSPluginsAvailab
 
 The plan is to extend the existing [`conditionController`](https://github.com/openshift/library-go/blob/master/pkg/operator/encryption/controllers/condition_controller.go) in library-go's encryption controllers, which already emits the `Encrypted` condition on the same operator CR. It sits in the right call path (operator CR → ClusterOperator) and runs on the informer set the rollup needs. If extending it turns out to be a poor fit (conflicting sync triggers, unrelated dependencies that make the rollup hard to reason about), a dedicated controller will be introduced instead.
 
+#### KMS Plugin Image Verification and Certification
+
+Vendors must sign their images using cosign, and run the certification test suite described in this document before Red Hat can officially support the vendor's KMS plugin. In order to provide official support for KMS plugin configuration, we will include a pointer to officially vendor supported and signed images in the OCP payload. This will ensure that only plugin images that can prove their validity and are officially supported can run on the OCP platform.
+
+##### Image Signature and Verification
+
+We advise vendors to sign their images with cosign using public key encryption. OpenShift must be able to verify image signatures in air-gapped clusters, and public key encryption is best suited for this scenario.
+
+###### Signing Images with cosign v3
+
+cosign v3 defaults to storing signatures as OCI 1.1 referrers (which are not visible as tags). To create signatures compatible with OpenShift, vendors must use the `--new-bundle-format=false` and `--use-signing-config=false` flags to force creation of legacy `.sig` tags.
+
+**Step-by-step signing process:**
+
+1. **Generate cosign key pair**:
+   ```bash
+   cosign generate-key-pair
+   # Creates cosign.key (private, keep secure) and cosign.pub (public, distribute to customers)
+   ```
+
+2. **Build and push your KMS plugin image**:
+   ```bash
+   # Build the image
+   podman build -t registry.example.com/kms-plugin:v1.0.0 .
+
+   # Push to registry
+   podman push registry.example.com/kms-plugin:v1.0.0
+   ```
+
+3. **Get the image digest**:
+   ```bash
+   DIGEST=$(skopeo inspect docker://registry.example.com/kms-plugin:v1.0.0 | jq -r '.Digest')
+   echo "Image digest: $DIGEST"
+   ```
+
+4. **Sign the image with compatibility flags**:
+   ```bash
+   COSIGN_PASSWORD=<your-password> cosign sign \
+     --key cosign.key \
+     --new-bundle-format=false \
+     --use-signing-config=false \
+     registry.example.com/kms-plugin@${DIGEST} \
+     --yes
+   ```
+
+5. **Verify the .sig tag was created**:
+   ```bash
+   # Check for signature tag (should show sha256-<digest>.sig)
+   skopeo list-tags docker://registry.example.com/kms-plugin | jq -r '.Tags[]' | grep ".sig"
+   ```
+
+To generate the base64-encoded public key for the `keyData` field that the KMS encryption API in OCP will need:
+```bash
+cat cosign.pub | base64 -w 0
+```
+
+**Why these flags are required:**
+- `--new-bundle-format=false`: Forces cosign to use the legacy tag-based signature format instead of OCI 1.1 referrers
+- `--use-signing-config=false`: Required when using `--new-bundle-format=false` to disable TUF-provided signing configuration
+
+**Note:** These flags will be removed in cosign v4. When containers/image adds OCI 1.1 referrers support in future OpenShift versions, vendors can use cosign v3+ default behavior without compatibility flags.
+
+###### ImagePolicy Cluster Configuration
+
+The cluster operators will ensure that only verified images can be pulled and run from the cluster. To ensure that, when the KMS plugin is configured, the operators will check the cluster for a scoped ClusterImagePolicy with a well formed name, scope and policy.
+
+To do this, when the controllers detect that KMS mode is configured and the operator is configuring and deploying the plugin, it will first detect which supported KMS plugin is being rolled out. It will then fetch the correct well known ClusterImagePolicy that matches the metadata for the plugin it expects. If the ClusterImagePolicy match the criteria that is embedded in the OCP payload, installation of the KMS plugin will continue.
+
+**Note:**  All supported plugin metadata, including registry image path and public key data, will be hardcoded and included in the OpenShift payload. That data will be provided by KMS plugin vendors to OCP Engineering.
+
+###### Customer Configuration
+
+Users must create a `ClusterImagePolicy` resource to configure OpenShift to verify the image signature every time CRI-O pulls a plugin image. Users should obtain the vendor's public key from the vendor's official documentation (e.g., HashiCorp's documentation for the Vault KMS plugin).
+
+`ClusterImagePolicy` example:
+```yaml
+apiVersion: config.openshift.io/v1
+kind: ClusterImagePolicy
+metadata:
+  name: vault-kms-plugin
+  labels:
+    config.openshift.io/type: kms
+spec:
+  scopes:
+    - docker.io/hashicorp/vault-plugin-kms # this must match the kmsPluginImage in the APIServer config
+  policy:
+    rootOfTrust:
+      policyType: PublicKey
+      publicKey:
+        # base64-encoded cosign public key (cosign.pub)
+        keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFLi4uCi0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQo=
+    signedIdentity:
+      matchPolicy: MatchRepository
+```
+
+In order for the API Server Operators to validate that the installed KMS plugin is certified and supported by OCP, it must be defined in a well known way. The fields that match must include the following:
+
+```
+metadata.name
+spec.scopes
+spec.policy.rootOfTrust.policyType
+spec.policy.matchPolicy
+```
+
+As part of our KMS documentation, we will publish a mapping of the fields that are required for each KMS plugin.
+
+Additionally, users must set the `spec.policy.rootOfTrust.publicKey.keyData` field to the base64 encoded public key that the image was signed with. Users should refer to the vendor's official documentation for the most current public key to use in their `ClusterImagePolicy`.
+
+When vendors release images signed with a new key, users must create a new `ClusterImagePolicy` resource with the updated public key. During the transition period, both old and new `ClusterImagePolicy` resources must exist in the cluster until the previous `ClusterImagePolicy` is no longer needed (i.e when migration to the new plugin image is finished).
+
+###### Overriding Configuration
+
+There are cases where a user needs the ability to override this configuration in order to be able to use an unsupported plugin image -- for example if there is a hotfix that needs to be handed to a user urgently or even in certain testing  scenarios. We will enable this by adding an override field to the UnsupportedConfigOverrides cluster API that will disable the deployment of the ClusterImagePolicy. This will put the cluster in an unsupported state that will need to be resolved before upgrading the cluster, but it will allow a user a temporary workaround to running unsupported or unverified plugin images.
+
+`UnsupportedConfigOverrides` in cluster config API example:
+```yaml
+apiVersion: operator.openshift.io/v1
+kind: KubeAPIServer
+metadata:
+  name: cluster
+spec:
+  unsupportedConfigOverrides:
+    encryption:
+      kms:
+        ignoreImagePolicy: true
+```
+
+When that override is set, users will be able to apply KMS plugin configuration to the cluster without image verification.
+
+###### Disconnected Concerns
+
+When a cluster is disconnected from external networks or airgapped, the ClusterImagePolicy defined by the cluster will not be sufficient because the cluster will not have access to an external registry to pull the image. OpenShift has tools and APIs that enable core component images and layered operators to be mirrored into disconnected environments in mirror registries (oc-mirror, IDMS, ITMS). In a future iteration, we will update oc-mirror to understand how to mirror KMS plugins and provide the correct set of manifests to apply to the cluster so that plugins can be run out of the box in a seamless experience. For now, we are putting that work out of scope of this initial enhancement.
+
+To manually configure the mirror, the ClusterImagePolicy API has the `spec.policy.signedIdentity.remapIdentity` field that allows users to point the signed image to a mirrored registry.
+
+`ClusterImagePolicy` example for disconnected:
+```yaml
+apiVersion: config.openshift.io/v1
+kind: ClusterImagePolicy
+metadata:
+  name: vault-kms-plugin
+  labels:
+    config.openshift.io/type: kms
+spec:
+  scopes:
+    - mirror.internal.registry.com:5000/internal/mirrored-vault-plugin-kms # mirrored KMS plugin image
+  policy:
+    rootOfTrust:
+      policyType: PublicKey
+      publicKey:
+        # base64-encoded cosign public key (cosign.pub)
+        keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFLi4uCi0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQo=
+    signedIdentity:
+      matchPolicy: MatchRepository
+      remapIdentity:
+        prefix: mirror.internal.registry.com:5000/internal/mirrored-vault-plugin-kms # mirrored KMS plugin image
+        signedPrefix: docker.io/hashicorp/vault-plugin-kms # this must match the kmsPluginImage in the APIServer config
+```
+
+Once a user has applied the ClusterImagePolicy the KMS plugin installation can continue.
+
+##### Certification
+
+KMS plugin certification follows the CSI driver certification model. A dedicated test suite will be added to the `openshift-tests` binary, which vendors can extract from an OpenShift release payload and run in a test cluster with their KMS plugin installed.
+
+The certification test suite validates:
+- KMS plugin compliance with the Kubernetes KMS v2 API
+- Encrypt/decrypt operations under various conditions
+- Plugin behavior during upgrades and migrations
+
+###### Extracting the Test Binary and Running the Test Suite
+
+The KMS encryption test suite may take several hours to finish and is marked as disruptive (it may affect cluster stability during testing). Vendors are advised to run it in a dedicated test cluster within a CI/CD environment where test output is persisted. The test output is required to validate KMS plugin compatibility with OpenShift.
+
+**Prerequisites:**
+- An OpenShift cluster (version matching the target certification version)
+- KMS plugin configured via OpenShift APIServer configuration API
+- `oc` CLI tool installed and authenticated to the cluster
+- At least 1GB of disk space for test binary extraction and test output
+
+**Step 1: Extract the openshift-tests binary from the release payload**
+
+```bash
+# Set the release image you want to certify against
+RELEASE_IMAGE="quay.io/openshift-release-dev/ocp-release:4.22.0-x86_64"
+
+# Get the tests image reference from the release payload
+TESTS_IMAGE=$(oc adm release info ${RELEASE_IMAGE} --image-for=tests)
+echo "Tests image: ${TESTS_IMAGE}"
+
+# Extract the openshift-tests binary
+mkdir -p ./kms-certification
+oc image extract ${TESTS_IMAGE} \
+  --path /usr/bin/openshift-tests:./kms-certification \
+  --confirm
+
+# Make the binary executable
+chmod +x ./kms-certification/openshift-tests
+
+# Verify extraction
+./kms-certification/openshift-tests version
+```
+
+**Step 2: Run the KMS certification test suite**
+
+```bash
+# Set your kubeconfig (if not already set)
+export KUBECONFIG=/path/to/your/kubeconfig
+
+# Run the full KMS test suite
+# Note: This may take several hours and is disruptive to the cluster
+./kms-certification/openshift-tests run openshift/kms \
+  --junit-dir=./kms-certification/junit \
+  -o ./kms-certification/kms-test-output.log
+
+# The exit code indicates success (0) or failure (non-zero)
+echo "Test run exit code: $?"
+```
+
+**Step 3: Review test results**
+
+```bash
+# Check the test output log
+less ./kms-certification/kms-test-output.log
+
+# Review JUnit XML results (for CI integration)
+ls -lh ./kms-certification/junit/
+
+# Count passed/failed tests
+grep -c "PASS:" ./kms-certification/kms-test-output.log
+grep -c "FAIL:" ./kms-certification/kms-test-output.log
+```
+
+**Test Suite Configuration:**
+
+The `openshift/kms` test suite is configured with the following parameters:
+- **Parallelism:** 1 (tests run sequentially to avoid conflicts)
+- **Timeout:** 4 hours
+- **Stability:** Disruptive (may affect cluster operations)
+- **Scope:** Aggregates all KMS encryption tests across kube-apiserver, openshift-apiserver, and oauth-apiserver
+- **Exclusions:** Automatically excludes tests marked as `[Flaky]` or `[Disabled:]`
+
+**What the Test Suite Validates:**
+
+The test suite performs comprehensive validation including:
+
+1. **KMS v2 API Compliance:**
+   - Status, Encrypt, and Decrypt gRPC method correctness
+   - Proper error handling and timeout behavior
+   - Health check responses
+
+2. **Encryption Operations:**
+   - Successful encryption and decryption of secrets and configmaps
+   - Concurrent encrypt/decrypt operations under load
+   - Round-trip data integrity verification
+
+3. **Migration Scenarios:**
+   - identity → KMS migration
+   - KMS → identity migration
+   - KMS → KMS migration
+   - aesgcm/aescbc ↔ KMS migrations
+
+4. **Operational Behaviors:**
+   - Plugin startup and initialization
+   - Graceful handling of KMS service unavailability
+   - API server restart with KMS encryption enabled
+   - Multi-plugin scenarios during migration
+
+5. **OpenShift Integration:**
+   - Sidecar container lifecycle
+   - Credential management from Secrets
+   - ConfigMap integration for CA bundles
+   - Pre-flight checker validation
+
+**Submitting Certification Results:**
+
+After successful completion, vendors should provide:
+1. Full test output log (`kms-test-output.log`)
+2. JUnit XML files from the `junit/` directory
+3. OpenShift version and KMS plugin version tested
+4. Any test failures with root cause analysis and remediation plans
+
+**Troubleshooting Common Issues:**
+
+```bash
+# If tests fail to connect to the cluster
+export KUBECONFIG=/path/to/correct/kubeconfig
+oc whoami  # Verify authentication
+
+# If tests timeout
+# Increase timeout with: --timeout=6h
+./kms-certification/openshift-tests run openshift/kms --timeout=6h
+
+# To run a subset of tests for debugging
+# First, list the tests with --dry-run, then filter specific ones
+./kms-certification/openshift-tests run openshift/kms \
+  --dry-run | grep "specific-test-name" | \
+  ./kms-certification/openshift-tests run -f -
+
+# For verbose output during test run
+./kms-certification/openshift-tests run openshift/kms -v=4
+```
+
 ### Risks and Mitigations
 
 **Risk: KMS Plugin Unavailable During Controller Sync**