Add Prometheus + Alertmanager StatefulSet readiness to kube-prometheus-stack check #1247

Summary

Follow-up to PR #1245 (closed #1222, closed epic #660). Add
StatefulSet readiness assertions for the Prometheus and
Alertmanager workloads shipped by the kube-prometheus-stack
chart, which PR #1245 deferred because the chart's exact StatefulSet
names and label conventions need verification.

Background

PR #1245 enhanced recipes/checks/kube-prometheus-stack/health-check.yaml
with universal container-state error coverage but stopped short of
asserting the StatefulSet workloads the stack ships:

• Prometheus (via the prometheus.monitoring.coreos.com CR; the
  operator creates a StatefulSet named per chart fullname template)
• Alertmanager (same pattern via the alertmanager.monitoring.coreos.com CR)

Today's check only asserts the kube-prometheus-operator Deployment

• label-scoped pod-phase / container-state errors in the monitoring
  namespace. A scenario where the operator is healthy but Prometheus
  itself is stuck (Pending pods, PVC binding failures, scrape config
  errors) would silently pass.

Why deferred

The chart's StatefulSet names depend on the Prometheus / Alertmanager
CR names (e.g., prometheus-<cr-name>, alertmanager-<cr-name>),
which in turn depend on the chart's prometheus.prometheusSpec.name
/ alertmanager.alertmanagerSpec.name values or the chart's default
naming. Without live-cluster confirmation (or a chart-template render
against the pinned values file), guessing the names risks a typo'd
assertion that fails on real deployments.

Approach

1. Render the chart at its pinned version with AICR's
   recipes/components/kube-prometheus-stack/values.yaml applied:
   helm template <release> prometheus-community/kube-prometheus-stack --version <pin> -f values.yaml (release name kube-prometheus
   matches AICR's ComponentRef Name).
2. Enumerate the actual StatefulSet and CR metadata.name values
   from the rendered output.
3. Add asserts to recipes/checks/kube-prometheus-stack/health-check.yaml:
   • Prometheus CR exists and status.conditions[?type == 'Available'].status == "True" (or the chart's equivalent
     readiness condition).
   • Alertmanager CR same.
   • StatefulSet readyReplicas >= replicas for each, as a deeper
     workload-level signal that the CRs actually reconciled into
     running pods.
4. Live-cluster validation against any recipe that enables
   kube-prometheus-stack (e.g., conformance overlays).

Acceptance Criteria

• recipes/checks/kube-prometheus-stack/health-check.yaml asserts
  both the Prometheus and Alertmanager StatefulSets reach full
  rollout (readyReplicas >= replicas with a vacuous-pass guard).
• Optional but recommended: also assert the corresponding
  Prometheus / Alertmanager CRs report a reconciled condition.
• Allowlist sweep + registry lint guard remain green.
• Live-cluster validation documented in the PR description.

Related

• PR feat(recipes): deepen 21 chainsaw health checks; close epic #660 #1245 (closed Enhance existing Chainsaw health checks (#660 PR 4) #1222, closed epic [EPIC] Reuse Chainsaw health checks in aicr validate --phase deployment #660) — the source of this deferral
• Epic [EPIC] Reuse Chainsaw health checks in aicr validate --phase deployment #660 — the broader chainsaw-deployment work
• recipes/components/prometheus-operator-crds — the sibling
  CRD-only component this check depends on (already asserts the 8
  prometheus-operator CRDs Established)