Skip to content

fix(recipes): NFD worker-DaemonSet check asserts omitempty numberUnavailable (fails on healthy cluster) #1248

Closed as not planned

Description

@ArangoGutierrez

Summary

recipes/checks/nfd/health-check.yaml gates the NFD worker DaemonSet rollout on a numberUnavailable == 0 assertion. DaemonSetStatus.NumberUnavailable is omitempty, so on a fully-healthy DaemonSet (0 unavailable) the field is absent from the serialized object. Chainsaw/kyverno-json then evaluates that comparison as null == 0 → false, the assert fails, and the NFD health check fails on an otherwise-healthy cluster.

Discovered during review of #1245, which deliberately avoided this pattern for its aws-efa / nvidia-dra-driver-gpu DaemonSet checks.

Evidence

  • recipes/checks/nfd/health-check.yaml — the validate-worker-daemonset step asserts that numberUnavailable equals 0 (see the Fix section for the exact chainsaw expression).
  • vendor/k8s.io/api/apps/v1/types.go:722 declares NumberUnavailable with a json:"numberUnavailable,omitempty" struct tag — the omitempty is the trap. By contrast DesiredNumberScheduled (L698) and NumberReady (L702) have no omitempty, so they are always serialized.
  • kyverno-json scalar matching: a missing field projects to null, and null == 0 is false.

Impact

The NFD worker-DaemonSet rollout gate is structurally unsatisfiable when the DaemonSet is healthy — the field it tests is omitted exactly when the condition it wants (== 0) is true. The master/GC Deployment asserts, the phase checks, and the container-state checks are unaffected; this is isolated to the worker-DaemonSet rollout assertion.

Fix

Match the corrected pattern #1245 uses for its two DaemonSets (numberReady and desiredNumberScheduled are always serialized, so this is robust):

- assert:
    resource:
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: node-feature-discovery-worker
        namespace: node-feature-discovery
      status:
        # Guard against vacuous pass when 0 nodes match the DaemonSet.
        (desiredNumberScheduled > `0`): true
        # Full rollout: every scheduled pod is ready.
        (numberReady == desiredNumberScheduled): true

Related

Note

#1245 carries comments in aws-efa / nvidia-dra-driver-gpu that cite this NFD file as "same rationale" — those cross-references are flagged for correction in that PR (NFD uses a different, broken mechanism).

Activity

  1. github-actions commented on Jun 9, 2026

    @github-actions
    Contributor

    Welcome to AICR, @ArangoGutierrez! Thanks for opening your first issue.

    A maintainer will triage this shortly. In the meantime:

  2. self-assigned this
    on Jun 9, 2026
  3. ArangoGutierrez commented on Jun 9, 2026

    @ArangoGutierrez
    ContributorAuthor

    Closing as invalid — this is already fixed on main.

    NFD's worker-DaemonSet check uses (numberReady == desiredNumberScheduled): true (with the desiredNumberScheduled > 0 guard) on both main and PR #1245's head fcb5bdc, and the file carries an inline comment documenting the numberUnavailable omitempty trap. It was fixed by commit d7e68a24 as part of #1235's chainsaw-runtime work.

    I filed this off a stale local checkout (branch pr-997, which predates d7e68a24) — I read the wrong copy of the file. The underlying mechanism described here is real (numberUnavailable is omitempty, so null == 0 is false on a healthy DaemonSet) but it was already addressed. No action needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions