SPLAT-2668: remove CCM test skipps from Hypershift TP jobs#79439
SPLAT-2668: remove CCM test skipps from Hypershift TP jobs#79439mtulio wants to merge 3 commits into
Conversation
openshift-ci-robot
added
the
jira/valid-reference
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
@mtulio: This pull request references SPLAT-2668 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Skipping CI for Draft Pull Request. |
|
/test all |
WalkthroughAdds two AWS OVN conformance CCM e2e jobs (regular and techpreview) to hypershift CI and periodics, clears/condenses TEST_SKIPS for those jobs across 4.23 and 5.0 configs, and schedules the techpreview periodic as daily for 5.0. ChangesHypershift CI jobs and periodics
Sequence Diagram(s)sequenceDiagram
participant Developer
participant CI-Operator
participant ci_config_main as "openshift-hypershift-main.yaml"
participant periodics_5_0 as "openshift-hypershift-release-5.0__periodics.yaml"
participant periodics_4_23 as "openshift-hypershift-release-4.23__periodics.yaml"
Developer->>CI-Operator: add/modify job definitions and periodics
CI-Operator->>ci_config_main: insert new e2e-aws-ovn-conformance-ccm jobs (PUBLIC_ONLY, TEST_SKIPS, GUEST_FEATURE_SET)
CI-Operator->>periodics_5_0: register `@monthly` and `@daily` periodics for new jobs
CI-Operator->>periodics_4_23: update schedules and clear TEST_SKIPS
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 12✅ Passed checks (12 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/ph-rehearse periodic-ci-openshift-hypershift-release-5.0-e2e-aws-ovn-conformance-ccm-techpreview |
1 similar comment
|
/ph-rehearse periodic-ci-openshift-hypershift-release-5.0-e2e-aws-ovn-conformance-ccm-techpreview |
|
Another manual since yesterday failed due outdated image: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-5.0-e2e-aws-ovn-conformance-ccm-techpreview/2056746936064544768 |
mtulio
force-pushed
the
SPLAT-2668-validate-OCPBUGS-85414
branch
from
b879e72 to
501cf43
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml`:
- Around line 159-177: The periodic job entries for "as:
e2e-aws-ovn-conformance-ccm" and "as: e2e-aws-ovn-conformance-ccm-techpreview"
are missing the required mirrored sections from the main config; copy the full
set of variant sections (base_images, build_root, images, promotion, and
releases) from the main config into each of these periodic job blocks so their
structure matches the main config exactly, ensuring you place the copied
build_root, images, promotion, base_images and releases under the same
steps/job-level scope as the other fields for those two jobs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 4c60cda5-c9dd-46a2-a881-d997f083c3b3
⛔ Files ignored due to path filters (2)
ci-operator/jobs/openshift/hypershift/openshift-hypershift-main-presubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (3)
ci-operator/config/openshift/hypershift/openshift-hypershift-main.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
💤 Files with no reviewable changes (1)
- ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
mtulio
force-pushed
the
SPLAT-2668-validate-OCPBUGS-85414
branch
from
501cf43 to
1ad913a
Compare
openshift-ci
Bot
added
the
needs-rebase
|
@mtulio, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
mtulio
force-pushed
the
SPLAT-2668-validate-OCPBUGS-85414
branch
from
1ad913a to
64535e9
Compare
|
/test all |
openshift-ci
Bot
removed
the
needs-rebase
There was a problem hiding this comment.
♻️ Duplicate comments (1)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml (1)
159-175: 🛠️ Refactor suggestion | 🟠 MajorPeriodic variant is missing required mirrored sections from main config.
This
__periodics.yamlfile still lacksbuild_root,images, andpromotionsections at the file level. While this is a pre-existing structural issue (not introduced by the current PR), it should be addressed to comply with variant file structure requirements.As per coding guidelines: "Variant periodic configuration files should include
base_images,build_root,images,promotion, andreleasessections copied from main config, then add onlytests:entries withinterval:orcron:scheduling"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml` around lines 159 - 175, This periodic variant is missing required top-level sections copied from the main config; add the base_images, build_root, images, promotion, and releases sections from the main config into this variant file and keep only tests entries (like the existing e2e-aws-ovn-conformance-ccm and e2e-aws-ovn-conformance-ccm-techpreview blocks) with their cron/interval scheduling; ensure the inserted sections match the main config exactly (including any base_images keys and build_root settings) so the variant structure follows the guideline that only tests are varied while build/images/promotion/releases remain identical.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml`:
- Around line 159-175: This periodic variant is missing required top-level
sections copied from the main config; add the base_images, build_root, images,
promotion, and releases sections from the main config into this variant file and
keep only tests entries (like the existing e2e-aws-ovn-conformance-ccm and
e2e-aws-ovn-conformance-ccm-techpreview blocks) with their cron/interval
scheduling; ensure the inserted sections match the main config exactly
(including any base_images keys and build_root settings) so the variant
structure follows the guideline that only tests are varied while
build/images/promotion/releases remain identical.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 3c47be1d-c435-4d22-a084-1aa181d0879d
⛔ Files ignored due to path filters (3)
ci-operator/jobs/openshift/hypershift/openshift-hypershift-main-presubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.23-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (4)
ci-operator/config/openshift/hypershift/openshift-hypershift-main.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
💤 Files with no reviewable changes (1)
- ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
mtulio
changed the title
mtulio
force-pushed
the
SPLAT-2668-validate-OCPBUGS-85414
branch
from
64535e9 to
39202cd
Compare
|
Just rebase it, since openshift/cluster-cloud-controller-manager-operator#464 is merged, we are green to go rehearsing jobs whe the merge is available in the pipeline: /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm-techpreview |
|
@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
rehearsal job may not include the CCCMO pr yet, I will try again later tomorrow. for now testing job config readiness /test all |
|
interim update, 4.23.0-0.ci-2026-06-04-111742 is cooking with expected changes: cluster-cloud-controller-manager-operator Observing it to finish before triggering a new job
|
|
Release is in Ready phase, let's see if rehearsal can pick this one: /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm-techpreview |
|
@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
The job caught the last Accepted, which does not include the fix yet :( |
trying again: /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm-techpreview |
|
@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
job canceled to save infra as it didn't picked the expected release image yet :( |
|
Release 4.23.0-0.ci-2026-06-04-111742 is /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm-techpreview |
|
@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
mtulio
force-pushed
the
SPLAT-2668-validate-OCPBUGS-85414
branch
from
39202cd to
60d2380
Compare
|
Success validation showing 5/6 are passing, and 1/6 is skipped with prefix 1 test is also failing on CSI, which is unrelated and I can't find it on Sippy, skipping for now: I updated TESTS_FIX with those skips as it is not the goal of fixing by this PR as well linked bug 85414. /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm-techpreview |
|
@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/test all |
|
CSI tests is still not skipped, I am checking the exactly pattern, meanwhile, looks like this CSI test has already tracking issues (details here): Current Test Health (Sippy) Per Sippy, this test currently has a 100% pass rate across all releases (5.0, 4.22, presubmits) over the last 14 days — zero failures, zero flakes, and no Component Readiness regressions detected. It's passing on AWS, GCP, Azure, Metal, vSphere, and OpenStack across HA, SNO, blocking, and informing tiers. Existing Tracked Bugs There are several related OCPBUGS, all currently Verified:
Root Cause The underlying issue in these bugs is a race condition in the csi-hostpath-groupsnapshot driver where CreateVolumeGroupSnapshot does not return complete responses on repeated calls, leading to incomplete VolumeSnapshotInfoList data. The fix has been merged and is present in current nightlies. |
Review CCM tests by keeping only main presubmit, and adding 5.0 periodics for jobs required to ensure CCM e2e readines on Hypershift. Removing TEST_SKIPS which is manually tested and working in local dev env in HCP clusters: openshift/cluster-cloud-controller-manager-operator#464 (comment)
mtulio
force-pushed
the
SPLAT-2668-validate-OCPBUGS-85414
branch
from
60d2380 to
f396815
Compare
v |
/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm-techpreview |
|
@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
rehearsal are passing, I will clean up the PR. /test all |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mtulio The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Clean up done:
/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm-techpreview periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance-ccm-techpreview periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance-ccm periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance |
|
@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance-techpreview |
|
@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
A total of 612 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@mtulio: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
This PR updates Hypershift CI configuration in the openshift/release repository to debug removal of skipped Cloud Controller Manager (CCM) tests on Hypershift (refs SPLAT-2668 / OCPBUGS-85414).
What changed (practical terms)
@monthlywith TEST_SKIPS: ""@dailywith GUEST_FEATURE_SET: TechPreviewNoUpgrade and TEST_SKIPS: ""These schedule the CCM conformance variants for automatic runs on the 5.0 release branch.
@monthlyto@dailyand clears its TEST_SKIPS as well — increasing frequency and ensuring the CCM variants exercise previously-skipped tests on 4.23.Impact
Other notes