Red Hat ISV Operator Certification Pipelines

Red Hat OpenShift pipelines for certifying ISV Operator Bundles.

Getting Started

Refer to the developer guide.

Note: This documentation is intended for pipeline developers/maintainers only.

Partners/users should refer to this documentation instead.

Usage

Operator CI Pipeline

The Operator CI pipeline is a Tekton pipeline that can be triggered by a partner using on-premise infrastructure. The pipeline validates an Operator Bundle, builds it and installs it to an OpenShift environment. After installation, pre-flight tests are executed which validate that the Operator meets minimum requirements for Red Hat OpenShift Certification. If all preceding tasks pass, the CI pipeline optionally uploads results and submits a pull request to trigger the next stages of the operator certification workflow.

Note: Execution of the CI pipeline is NOT required in the overall certification workflow.

If using the default internal registry, the CI pipeline can be triggered using the tkn CLI like so:

tkn pipeline start operator-ci-pipeline \
  --use-param-defaults \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_branch=main \
  --param bundle_path=operators/kogito-operator/1.6.0-ok \
  --param env=prod \
  --workspace name=pipeline,volumeClaimTemplateFile=templates/workspace-template.yml \
  --showlog

If using an external registry, the CI pipeline can be triggered using the tkn CLI like so:

tkn pipeline start operator-ci-pipeline \
  --use-param-defaults \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_branch=main \
  --param bundle_path=operators/kogito-operator/1.6.0-ok \
  --param env=prod \
  --param registry=quay.io \
  --param image_namespace=redhat-isv \
  --workspace name=pipeline,volumeClaimTemplateFile=templates/workspace-template.yml \
  --workspace name=registry-credentials,secret=registry-dockerconfig-secret \
  --showlog

A subset of tasks in the pipeline requires privilege escalation which is no longer supported with OpenShift Pipelines 1.9. Thus a new SCC needs to be created and linked with pipeline service account. Creating SCC requires user with cluster-admin privileges.

# Create a new SCC
oc apply -f ansible/roles/operator-pipeline/templates/openshift/openshift-pipelines-custom-scc.yml
# Add SCC to a pipeline service account
oc adm policy add-scc-to-user pipelines-custom-scc -z pipeline

To enable opening the PR and uploading the pipeline logs (visible to the certification project owner in Red Hat Connect), pass the following argument:

--param submit=true

To open the PR with submission, upstream repository name must be supplied (eg. test-org/test-repo):

--param upstream_repo_name=<repository_name>

To enable digest pinning, pass the following arguments:

--param pin_digests=true \
--param git_repo_url=<github_repo_ssh_url> \
--param git_username=<github_user_name> \
--param git_email=<github_email> \
--workspace name=ssh-dir,secret=github-ssh-credentials

Note: The git_repo_url param needs an SSH URL to commit the pinned digests.

If any of bundle's related images are stored in a private registry the user needs to provide tokens to pull from those registries. See more details about how to provide registry tokens in the pipeline environment setup documentation.

Operator Hosted Pipeline

The hosted pipeline is used to certify the Operator bundles. It’s an additional (to CI pipeline) layer of validation that has to run within the Red Hat infrastructure. It is intended to be triggered upon the creation of a bundle pull request and successfully completes with merging it (configurable).

Note: Execution of the hosted pipeline is ALWAYS required in the overall certification workflow. Prior execution of the CI pipeline may influence its behavior if results were submitted. Preflight testing may be skipped in such a case.

The hosted pipeline can be triggered using the tkn CLI like so:

tkn pipeline start operator-hosted-pipeline \
  --use-param-defaults \
  --param git_pr_branch=test-PR-ok \
  --param git_pr_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test/pull/31 \
  --param git_fork_url=https://github.com/MarcinGinszt/operator-pipelines-test.git \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_username=foo@redhat.com \
  --param git_commit=0aeff5f71e4fc2d4990474780b56d9312554da5a \
  --param git_base_branch=main \
  --param env=prod \
  --workspace name=repository,volumeClaimTemplateFile=templates/workspace-template-small.yml \
  --workspace name=results,volumeClaimTemplateFile=templates/workspace-template.yml \
  --workspace name=registry-credentials-all,volumeClaimTemplateFile=templates/workspace-template-small.yml \
  --workspace name=registry-credentials,secret=hosted-pipeline-registry-auth-secret \
  --showlog

To ignore the results of the publishing checklist, pass the following argument:

  --param ignore_publishing_checklist=true

⚠️ Only quay-based registries are supported by the hosted pipeline. There are some quay specific tasks for configuring the repositories where the bundle and index images are pushed.

Merge base lane guard (ISV-7093)

Before gh pr merge, the hosted pipeline can block the merge when the default branch tip has advanced after the run recorded its snapshot and the operator/catalog paths in the guard lane changed at the new tip (file-level fingerprint via git ls-tree, not a second IIB or opm validation). The snapshot baseline is the live target-branch tip at snapshot time (shallow clone of git_repo_url at git_base_branch when the tip differs from git_commit_base, otherwise the existing base clone), while detect-changes still compares PR head to the clone at git_commit_base. The lane is operator_path plus each path from added_or_modified_catalog_operators (normalized to catalogs/<ocp_version>/<operator>), not whole OCP version trees under catalogs/. If the branch tip is unchanged between snapshot and merge, the check is a no-op. Limitations: the guard can prevent merge in situations with no actual conflict (e.g. different channels), making the process more restrictive. If the guard fails due to base drift, either restart the hosted pipeline (when your change still fits the current base tip) or rebase or merge the latest target branch into the PR branch and re-run for a fresher baseline. Pick the option that matches your situation.

Operator Release Pipeline

The release pipeline is responsible for releasing a bundle image which has passed certification. It's intended to be triggered by the merge of a bundle pull request by the hosted pipeline. It successfully completes once the bundle has been distributed to all relevant Operator catalogs and appears in the Red Hat Ecosystem Catalog.

Note: Execution of the release pipeline is ALWAYS required in the overall certification workflow.

tkn pipeline start operator-release-pipeline \
  --use-param-defaults \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_commit=3ffff387caac0a5b475f44c4a54fb45eebb8dd8e \
  --param git_pr_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test/pull/31 \
  --param dest_image_namespace=redhat-isv-operators \
  --workspace name=repository,volumeClaimTemplateFile=templates/workspace-template.yml \
  --workspace name=results,volumeClaimTemplateFile=templates/workspace-template-small.yml \
  --workspace name=image-data,volumeClaimTemplateFile=templates/workspace-template-small.yml \
  --workspace name=registry-pull-credentials,secret=release-pipeline-registry-auth-pull-secret \
  --workspace name=registry-push-credentials,secret=release-pipeline-registry-auth-push-secret \
  --workspace name=registry-serve-credentials,secret=release-pipeline-registry-auth-serve-secret \
  --showlog

Using a Custom Pipeline Image

All the pipelines share a common pipeline image for many of the steps. This image can be overridden by passing the following to any tkn pipeline start command.

--param pipeline_image=<image-pull-spec>

Testing

The repository comes with a default configuration for integration tests. In order to execute tests locally a user needs an access to OCP cluster with operator-pipelines pre-installed.

The tests are orchestrated using Ansible and contain a several steps:

• Configuration of OCP cluster
• Preparation of test data (new version of operator bundle)
• Execution of operator pipelines
• Evaluation of pipelines statuses

To run test locally running the Makefile is recommended.

# Set a version of a new operator based on the latest successful test run in
# Github action
export OPERATOR_VERSION_RELEASE="402-1"

make build-and-test-isv

# or
make build-and-test-community
# or
make build-and-test-isv-fbc-bundle
# or
make build-and-test-isv-fbc-catalog

Running test in PR workflow

A repository is configured to automatically run all integration tests on pull request. This makes sure a code change doesn't break existing workflow. Since a tests are running OCP cluster and requires some resources a tests are execute only when ready-for-testing label is added to a PR. Adding this label executes additional workflow.

All PRs should pass a tests before merging.

CI/CD for external contributors

The repository contains Github Actions workflows that automatically run all CI steps including linting, building and testing the operator pipelines.

In case of a pull request from a forked repository the tests are not executed automatically due to security reasons. To run the tests an owner of the repository must review the pull request to make sure it doesn't contain any malicious code then approve the build-and-test workflow environment.

The manuall approval is not needed for contributors who have write access to the repository and not using a forked repository.

Additional Documentation

• OpenShift cluster configuration
• Index signature verification