Skip to content

CNTRLPLANE-3576: add RestartDateAnnotation propagation tests#8733

Open
mgencur wants to merge 1 commit into
openshift:mainfrom
mgencur:CNTRLPLANE-3576_on_demand_cp_restart
Open

CNTRLPLANE-3576: add RestartDateAnnotation propagation tests#8733
mgencur wants to merge 1 commit into
openshift:mainfrom
mgencur:CNTRLPLANE-3576_on_demand_cp_restart

Conversation

@mgencur

@mgencur mgencur commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

  • Convert an E2E test from openshift-tests-private into a unit test that covers the missing functionality as follows:
    • Verify that setDefaultOptions propagates RestartDateAnnotation from HCP to the pod template when set, and omits it when absent. This covers the mechanism that triggers rolling restarts via pod template annotation changes. Rolling out the new ReplicaSet is then Kubernetes' concern. The annotation propagation from HostedCluster to HostedControlPlane is already tested in TestReconcileHostedControlPlaneAnnotations.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3576

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Tests
    • Enhanced test coverage and robustness for restart date annotation propagation in control plane components, with improved test structure and error handling.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 15, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 15, 2026

Copy link
Copy Markdown

@mgencur: This pull request references CNTRLPLANE-3576 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

This PR is WIP because it builds on top of #8731 and will need a rebase once this PR is meged

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3576

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-area labels Jun 15, 2026
@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

TestSetDefaultOptions in support/controlplane-component/defaults_test.go receives two changes. First, the scheme initialization block now calls hyperv1.AddToScheme, corev1.AddToScheme, and appsv1.AddToScheme with t.Fatalf on any registration error, replacing the previous silent discard of errors. Second, restart-date annotation behavior is extracted into a new annotationTests table-driven block of parallel subtests, each exercising setDefaultOptions and asserting that hyperv1.RestartDateAnnotation is either propagated to or omitted from the workload pod template annotations based on its presence on the HostedControlPlane.

Suggested Reviewers

  • devguyio
  • Nirshal
🚥 Pre-merge checks | ✅ 10 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Test Structure And Quality ⚠️ Warning The test file uses standard Go testing, not Ginkgo. However, the tests fail quality requirement #4 (assertion messages): most assertions (12 without messages vs 1 with) lack meaningful failure mess... Add descriptive failure messages to assertions, especially to lines 553, 554-555, 706, 709-712. Example: g.Expect(err).NotTo(HaveOccurred(), "failed to set default options").
✅ Passed checks (10 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main change: adding RestartDateAnnotation propagation tests to the test file, which aligns with the refactored test logic and table-driven subtests described in the summary.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The file uses standard Go testing (func Test*, t.Run), not Ginkgo. No Ginkgo test constructs (It, Describe, Context, When) exist. Test names in annotation tests are static strings without dynamic v...
Topology-Aware Scheduling Compatibility ✅ Passed This PR modifies only test files (defaults_test.go) with unit tests for annotation propagation. No deployment manifests, operator code, or controller modifications that introduce scheduling constra...
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This PR modifies a standard Go unit test file using testing.T and table-driven tests, not Ginkgo e2e tests. The custom check applies only to Ginkgo e2e tests, so it is not applicable here.
No-Weak-Crypto ✅ Passed No weak cryptography patterns detected. The PR only modifies test code for annotation propagation and fixes a nil-map bug; no crypto imports or operations are present.
Container-Privileges ✅ Passed PR contains only unit test additions with no container/K8s manifests. No privileged settings (privileged:true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation:true, root without...
No-Sensitive-Data-In-Logs ✅ Passed No sensitive data exposed in logging. Only error logging for test setup with generic error messages and test data uses dummy values (test images, component names, and generic test resource identifi...

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from devguyio and jparrill June 15, 2026 08:58
@openshift-ci openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Jun 15, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@support/controlplane-component/defaults_test.go`:
- Around line 526-528: In support/controlplane-component/defaults_test.go at
lines 526-528, replace each `_ = ...AddToScheme(scheme)` statement (for hyperv1,
corev1, and appsv1) by capturing the error return value and checking it
explicitly: if the error is not nil, call t.Fatalf with an appropriate failure
message. In
control-plane-operator/controllers/hostedcontrolplane/v2/cno/component_test.go
at line 83, apply the same fix to the appsv1.AddToScheme call by capturing its
error return value and checking it with t.Fatalf on failure. This ensures test
initialization errors in scheme registration are properly caught and reported
rather than silently ignored.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 340b6197-2c17-4dde-8101-7a1e8f3c4713

📥 Commits

Reviewing files that changed from the base of the PR and between 712ba58 and fe42d7c.

📒 Files selected for processing (3)
  • control-plane-operator/controllers/hostedcontrolplane/v2/cno/component.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/cno/component_test.go
  • support/controlplane-component/defaults_test.go

Comment thread support/controlplane-component/defaults_test.go Outdated
@codecov

codecov Bot commented Jun 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.84%. Comparing base (f0536b8) to head (686cc08).
⚠️ Report is 170 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8733      +/-   ##
==========================================
+ Coverage   41.69%   41.84%   +0.14%     
==========================================
  Files         758      759       +1     
  Lines       93945    94073     +128     
==========================================
+ Hits        39175    39364     +189     
+ Misses      52025    51954      -71     
- Partials     2745     2755      +10     

see 17 files with indirect coverage changes

Flag Coverage Δ
cmd-support 35.14% <ø> (+0.11%) ⬆️
cpo-hostedcontrolplane 44.10% <ø> (+0.09%) ⬆️
cpo-other 43.45% <ø> (ø)
hypershift-operator 52.02% <ø> (+0.31%) ⬆️
other 31.56% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mgencur mgencur force-pushed the CNTRLPLANE-3576_on_demand_cp_restart branch from fe42d7c to 8b1e0ae Compare June 15, 2026 09:26
@mgencur mgencur force-pushed the CNTRLPLANE-3576_on_demand_cp_restart branch from 8b1e0ae to 928c70f Compare June 16, 2026 09:26
@mgencur mgencur changed the title [WIP] CNTRLPLANE-3576: add RestartDateAnnotation propagation tests CNTRLPLANE-3576: add RestartDateAnnotation propagation tests Jun 16, 2026
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 16, 2026

@jparrill jparrill left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped some comments. Thanks!

Comment thread control-plane-operator/controllers/hostedcontrolplane/v2/cno/component_test.go Outdated
Comment thread control-plane-operator/controllers/hostedcontrolplane/v2/cno/component_test.go Outdated
@csrwng

csrwng commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

/approve

@openshift-ci

openshift-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng, mgencur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 17, 2026
@mgencur

mgencur commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

/hold

I will remove the test for SetRestartAnnotationAndPatch because it's being removed in #8751

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 18, 2026
…ests

Verify that setDefaultOptions propagates RestartDateAnnotation from HCP
to the pod template when set, and omits it when absent. This covers the
mechanism that triggers rolling restarts via pod template annotation changes.

Rolling out the new ReplicaSet is then Kubernetes' concern.
The annotation propagation from HostedCluster to HostedControlPlane is
already tested in TestReconcileHostedControlPlaneAnnotations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mgencur mgencur force-pushed the CNTRLPLANE-3576_on_demand_cp_restart branch from 928c70f to 686cc08 Compare June 18, 2026 07:12
@mgencur

mgencur commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

/unhold

Removed the test for SetRestartAnnotationAndPatch and also the bug fix in that function, which also addresses both comments from @jparrill that were related to that part.

@openshift-ci openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 18, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
support/controlplane-component/defaults_test.go (1)

677-679: Consider adding loop variable rebinding for consistency with defensive coding practices.

Line 677-679 uses a range variable in a parallel subtest. While this is safe on Go 1.22+ (the minimum supported version), adding test := test before the t.Run call follows the pattern recommended in 100go.co for defensive clarity and consistency. However, this is optional refactoring—the file shows this pattern inconsistently applied (lines 618–625 have an identical pattern without rebinding), and the project's .golangci.yml does not enforce the loopvar linter.

Optional adjustment
 for _, test := range annotationTests {
+	test := test
 	t.Run(test.name, func(t *testing.T) {
 		t.Parallel()
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@support/controlplane-component/defaults_test.go` around lines 677 - 679, In
the annotationTests loop (line 677), add a loop variable rebinding line `test :=
test` immediately after the for statement and before the `t.Run` call to create
a local copy of the loop variable for the parallel subtest closure. This follows
defensive coding practices for consistency, even though it is safe on Go 1.22+
and the project does not enforce the loopvar linter.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@support/controlplane-component/defaults_test.go`:
- Around line 677-679: In the annotationTests loop (line 677), add a loop
variable rebinding line `test := test` immediately after the for statement and
before the `t.Run` call to create a local copy of the loop variable for the
parallel subtest closure. This follows defensive coding practices for
consistency, even though it is safe on Go 1.22+ and the project does not enforce
the loopvar linter.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: d75e2f71-8f9d-4239-8a4b-05f6d830e9ee

📥 Commits

Reviewing files that changed from the base of the PR and between 928c70f and 686cc08.

📒 Files selected for processing (1)
  • support/controlplane-component/defaults_test.go

@jparrill

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 18, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@cwbotbot

cwbotbot commented Jun 18, 2026

Copy link
Copy Markdown

Test Results

e2e-aks

e2e-aws

@mgencur

mgencur commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@hypershift-jira-solve-ci

Copy link
Copy Markdown

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2067861110131789824 | Cost: $2.7264704999999996 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report


Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6

@mgencur

mgencur commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@celebdor celebdor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/verified by "unit test covers the functionality it claims"

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 29, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@celebdor: This PR has been marked as verified by "unit test covers the functionality it claims".

Details

In response to this:

/lgtm
/verified by "unit test covers the functionality it claims"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 2d2b2d0 and 2 for PR HEAD 686cc08 in total

@openshift-ci

openshift-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@mgencur: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown

Test Failure Analysis Complete

Job Information

  • Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
  • Build ID: 2071530849425166336
  • Target: e2e-aws
  • Cluster: build01
  • State: error
  • Duration: 30m 0s (09:46:56 → 10:16:56 UTC, 2026-06-29)

Test Failure Analysis

Error

Pod scheduling timeout.

0/98 nodes are available: 1 Insufficient cpu, 12 node(s) didn't match Pod's node
affinity/selector, 3 Insufficient memory, 82 node(s) had untolerated taint(s). no new
claims to deallocate, preemption: 0/98 nodes are available: 4 No preemption victims found
for incoming pod, 94 Preemption is not helpful for scheduling.

Summary

The CI pod for this job was never scheduled onto a node in the build01 cluster. It remained in Pending state for the full 30-minute scheduling timeout, then was automatically killed by Prow. No test code ever executed — no artifacts, build logs, JUnit results, or cluster were produced. This is a CI infrastructure capacity issue entirely unrelated to the PR's code changes (CNTRLPLANE-3576: add RestartDateAnnotation propagation tests).

Root Cause

The Prow CI pod could not be scheduled on any of the 98 nodes in the build01 cluster for the entire 30-minute timeout window:

  • 82 nodes (83.7%) had taints the pod could not tolerate. The pod only tolerates node-role.kubernetes.io/ci-prowjobs-worker, node.kubernetes.io/not-ready, node.kubernetes.io/unreachable, and node.kubernetes.io/memory-pressure. The remaining 82 nodes carry other taints (likely for different workload classes — builds, tests, etc.).
  • 12 nodes (12.2%) did not match the pod's node affinity/selector. The pod requires ci-workload=prowjobs nodeSelector and kubernetes.io/arch in [amd64, arm64] affinity.
  • 3 nodes (3.1%) had insufficient memory. The pod requests ~106 MiB of memory.
  • 1 node (1.0%) had insufficient CPU. The pod requests only 10m CPU.

This means only ~4 nodes were potentially eligible (the ones that failed on resource constraints), but all 4 were fully loaded at the time. Preemption was also not possible — no suitable victims were found.

This is a transient CI infrastructure capacity issue — the build01 cluster's ci-prowjobs-worker node pool was saturated. The PR's code changes are not involved in any way.

Recommendations
  1. Retest the PR — simply trigger /retest or /test e2e-aws on the PR. This is a transient infrastructure issue and will likely succeed when cluster capacity frees up.
  2. No code changes needed — the PR's test code (RestartDateAnnotation propagation tests) was never compiled or executed. The failure has zero correlation with the PR content.
  3. If retests continue to fail with the same scheduling timeout, escalate to the CI infrastructure team (DPTP) as a capacity issue on the build01 cluster's ci-prowjobs-worker node pool.
Evidence
Evidence Detail
Job state error (not failure — job infrastructure problem, not test failure)
Status description Pod scheduling timeout.
Pod phase Pending — never reached Running
Pod condition PodScheduled=False, reason: Unschedulable
Pod created 2026-06-29T09:46:56Z
Pod deleted 2026-06-29T10:16:56Z (exactly 30m timeout)
Scheduling message 0/98 nodes are available: 1 Insufficient cpu, 12 node(s) didn't match Pod's node affinity/selector, 3 Insufficient memory, 82 node(s) had untolerated taint(s)
Artifacts produced None — no artifacts/ directory, no build-log.txt, no JUnit XML
Target cluster build01
Node selector ci-workload: prowjobs
Pod resource requests CPU: 10m, Memory: ~106 MiB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants