Skip to content

[cifmw_helpers] Fix CRC certificate renewal wait using kubeconfig#3989

Merged
openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
Valkyrie00:fix/cifmw-helpers-crc-start-expired-cert
Jun 10, 2026
Merged

[cifmw_helpers] Fix CRC certificate renewal wait using kubeconfig#3989
openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
Valkyrie00:fix/cifmw-helpers-crc-start-expired-cert

Conversation

@Valkyrie00

@Valkyrie00 Valkyrie00 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Problem

Since ~2026-06-03, all CRC-based molecule jobs (ci_local_storage, env_op_images, etc.) fail consistently during the prepare phase, stuck on TASK [cifmw_helpers : Login to the OpenShift when certificate is expired] for 15 minutes before timing out with 401 Unauthorized.

Root cause

The oc login -u kubeadmin command in crc_start.yml never included the p flag. In non-interactive CI the empty password always returns 401, regardless of cluster state.

Fix

  • Replace oc login with oc get nodes --kubeconfig=... which uses the CRC kubeconfig file directly, avoiding the OAuth dependency and the need for credentials during certificate renewal.
  • Increase retries from 90 to 150 (~25 min) to cover the observed cluster recovery time.

Refs:

@openshift-ci

openshift-ci Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@Valkyrie00 Valkyrie00 marked this pull request as ready for review June 9, 2026 22:04
@Valkyrie00 Valkyrie00 changed the title [DNM] [cifmw_helpers] Fix CRC certificate renewal wait using kubeconfig [cifmw_helpers] Fix CRC certificate renewal wait using kubeconfig Jun 9, 2026
Comment thread roles/ci_local_storage/molecule/default/prepare.yml Outdated
@danpawlik

danpawlik commented Jun 10, 2026

Copy link
Copy Markdown
Contributor
The oc login command in crc_start.yml was missing the -p (password)
flag, causing every retry to fail with 401 Unauthorized regardless
of cluster health.

that's not true :)

The generated kubeconfig contains few ids, no password needed due not via password authorization is done :)

Probably with new Zuul CI, new Ansible module are installed so it can not find kubeconfig in proper localization.

@Valkyrie00

Valkyrie00 commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author
The oc login command in crc_start.yml was missing the -p (password)
flag, causing every retry to fail with 401 Unauthorized regardless
of cluster health.

that's not true :)

The generated kubeconfig contains few ids, no password needed due not via password authorization is done :)

Probably with new Zuul CI, new Ansible module are installed so it can not find kubeconfig in proper localization.

  stdout: |-
    WARNING: Using insecure TLS client config. Setting this option is not supported!

    Console URL: https://api.crc.testing:6443/console
    Authentication required for https://api.crc.testing:6443 (openshift)
    Username: kubeadmin
    Password: Login failed (401 Unauthorized)
    Verify you have provided the correct credentials.
  stdout_lines: <omitted>

That's the error, that i found. If you have any suggestions, please feel free to share them.

The oc login command in crc_start.yml was missing the -p (password)
flag, causing every retry to fail with 401 Unauthorized regardless
of cluster health. This code path was never exercised until ~2026-06-03
when the CRC image certificates expired, breaking all CRC-based
molecule jobs.

Replace oc login with oc get nodes using the CRC kubeconfig file.
This avoids the OAuth dependency during certificate renewal and does
not require credentials. Also increase retries from 90 to 150 (~25
minutes) to cover the observed recovery time.

Signed-off-by: Vito Castellano <vcastell@redhat.com>
@nemarjan

Copy link
Copy Markdown
Contributor

/lgtm
/approve

@openshift-ci

openshift-ci Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nemarjan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@bogdando bogdando left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nemarjan nemarjan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@openshift-merge-bot openshift-merge-bot Bot merged commit 592d089 into openstack-k8s-operators:main Jun 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants