Skip to content

Commit b49eaaa

Browse files
authored
feat(nvsentinel): add opt-in event exporter mixin (#3371)
Signed-off-by: Varun Ramesh <varamesh@nvidia.com>
1 parent df40bdc commit b49eaaa

25 files changed

Lines changed: 2195 additions & 305 deletions

‎.github/workflows/kwok-nvsentinel-remediation.yaml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@
1919
# carries nvsentinel-health-events-analyzer, which
2020
# CheckNVSentinelRemediationPipelineCoherent requires beside live remediation.
2121
# It also composes quarantine with nvsentinel-slinky-drainer onto the EKS Slurm
22-
# leaf, which adds the plugin and switches node-drainer to customDrain.
22+
# leaf, which adds the plugin and switches node-drainer to customDrain, and
23+
# the event exporter onto the observe step.
2324

2425
name: KWOK NVSentinel Remediation
2526

@@ -112,6 +113,9 @@ jobs:
112113
- mixin: nvsentinel-quarantine,nvsentinel-slinky-drainer
113114
deployer: helm
114115
recipe: h100-eks-ubuntu-training-slurm
116+
# The event exporter (#2621), which reads the datastore a step adds.
117+
- mixin: nvsentinel-observe,nvsentinel-event-exporter
118+
deployer: helm
115119
steps:
116120
- name: Checkout Code
117121
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Runtime Kind test for the nvsentinel-event-exporter mixin.
16+
#
17+
# Checks the bundle gate refuses the exporter without a usable sink, installs
18+
# the observe step with the exporter via the bundle's own install.sh beside an
19+
# in-cluster mock OAuth token endpoint and CloudEvents sink, sends synthetic
20+
# health events into platform-connectors, and asserts what reaches the sink:
21+
# a client-credentials token taken with the Secret's client secret, the
22+
# mixin's filter, the CloudEvent shape and metadata, resume tokens in the
23+
# datastore's database, and delivery after a restart. Not wired into
24+
# `make qualify`: it needs a Kind cluster.
25+
#
26+
# Path filter kept narrow for the same reason as nvsentinel-preflight-e2e.yaml.
27+
28+
name: NVSentinel Event Exporter E2E
29+
30+
on:
31+
push:
32+
branches: [main]
33+
paths: &paths
34+
- 'recipes/mixins/nvsentinel-observe.yaml'
35+
- 'recipes/mixins/nvsentinel-event-exporter.yaml'
36+
- 'recipes/registry.yaml'
37+
- 'recipes/components/nvsentinel/**'
38+
- 'recipes/components/nvsentinel-mongodb/**'
39+
- 'recipes/components/psmdb-operator/**'
40+
# The tested overlay inherits kind.yaml, which inherits base.yaml, and
41+
# base.yaml is where nvsentinel enters the chain at all.
42+
- 'recipes/overlays/h100-kind-training.yaml'
43+
- 'recipes/overlays/kind.yaml'
44+
- 'recipes/overlays/base.yaml'
45+
- 'recipes/components/cert-manager/**'
46+
- 'recipes/components/prometheus-operator-crds/**'
47+
- 'pkg/bundler/validations/nvsentinel_event_exporter.go'
48+
- 'tests/e2e/nvsentinel-event-exporter/**'
49+
- 'pkg/bundler/deployer/localformat/templates/**'
50+
# run.sh sources tools/common and is invoked through a make target.
51+
- 'tools/common'
52+
- 'Makefile'
53+
- '.github/workflows/nvsentinel-event-exporter-e2e.yaml'
54+
- '.github/actions/setup-build-tools/**'
55+
- '.github/actions/prep-kind-runner/**'
56+
- '.github/actions/load-versions/**'
57+
- '.settings.yaml'
58+
pull_request:
59+
branches: [main]
60+
paths: *paths
61+
workflow_dispatch: {}
62+
63+
permissions:
64+
contents: read
65+
66+
concurrency:
67+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
68+
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
69+
70+
jobs:
71+
test:
72+
name: NVSentinel Event Exporter E2E
73+
# mongod 8.0 before 8.0.35 exits at startup on the ubuntu-24.04 image's
74+
# 7.0 kernel (https://jira.mongodb.org/browse/SERVER-131779). Return to
75+
# ubuntu-24.04 once the Percona MongoDB pin is 8.0.35 or newer.
76+
runs-on: ubuntu-22.04
77+
# The script's bounded waits sum to ~92 minutes worst case (node Ready 2,
78+
# mock 5, cert-manager 10, Percona datastore 15, nvsentinel rollout 15,
79+
# event pods 12, event delivery 12, filter settle 1, stop and restart 10,
80+
# proto fetch 1), before kind create, go build and the installs. A healthy
81+
# run takes a fraction of that.
82+
timeout-minutes: 110
83+
steps:
84+
- name: Checkout Code
85+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
86+
with:
87+
persist-credentials: false
88+
89+
- name: Load versions
90+
id: versions
91+
uses: ./.github/actions/load-versions
92+
93+
- name: Prep Kind runner
94+
uses: ./.github/actions/prep-kind-runner
95+
96+
- name: Set up Go
97+
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
98+
with:
99+
go-version: ${{ steps.versions.outputs.go }}
100+
cache: false
101+
102+
# run.sh's has_tools gate also needs jq and curl, which the runner image
103+
# already ships; installing only some of the rest fails the lane before
104+
# it exercises anything.
105+
- name: Install Kind, kubectl, Helm and yq
106+
uses: ./.github/actions/setup-build-tools
107+
with:
108+
install_kind: 'true'
109+
kind_version: '${{ steps.versions.outputs.kind }}'
110+
install_kubectl: 'true'
111+
kubectl_version: '${{ steps.versions.outputs.kubectl }}'
112+
install_helm: 'true'
113+
helm_version: '${{ steps.versions.outputs.helm }}'
114+
install_yq: 'true'
115+
yq_version: '${{ steps.versions.outputs.yq }}'
116+
117+
# KEEP_CLUSTER so the cluster survives for the diagnostics below; the
118+
# script's own EXIT trap would otherwise delete it before they run.
119+
- name: Run nvsentinel-event-exporter e2e
120+
env:
121+
KEEP_CLUSTER: 'true'
122+
run: make nvsentinel-event-exporter-e2e
123+
124+
# run.sh writes its kubeconfig inside a mktemp dir and removes that dir on
125+
# exit, so re-export before querying anything. cancelled() because a job
126+
# timeout cancels rather than fails.
127+
- name: Collect diagnostics on failure
128+
if: ${{ failure() || cancelled() }}
129+
run: |
130+
set +e
131+
cluster=aicr-nvsentinel-ee-e2e
132+
export KUBECONFIG="${RUNNER_TEMP}/ee-diag.kubeconfig"
133+
kind export kubeconfig --name "$cluster" --kubeconfig "$KUBECONFIG" || exit 0
134+
kubectl get pods -A
135+
kubectl -n nvsentinel get events --sort-by=.lastTimestamp
136+
kubectl -n nvsentinel get pvc,statefulset,deployment,perconaservermongodb,certificate
137+
kubectl -n nvsentinel describe pods -l app.kubernetes.io/name=event-exporter
138+
kubectl -n nvsentinel logs deployment/event-exporter --tail=200
139+
kubectl -n ee-mock logs deployment/ee-mock --tail=100
140+
kubectl -n nvsentinel logs daemonset/platform-connectors --tail=200
141+
kubectl -n nvsentinel logs statefulset/nvsentinel-mongodb-rs0 -c mongod --tail=100
142+
143+
- name: Delete Kind cluster
144+
if: ${{ always() }}
145+
run: kind delete cluster --name aicr-nvsentinel-ee-e2e || true

‎.github/workflows/nvsentinel-observability-render-check.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,9 @@
2424
# action map, the quarantine rule sets and the drain mode, the slinky-drainer
2525
# one asserts node-drainer's customDrain wiring to the plugin, the
2626
# health-events-analyzer one asserts the workload, its datastore wiring and
27-
# that the mixin names every rule the chart ships, and the CRD
27+
# that the mixin names every rule the chart ships, the event-exporter one
28+
# asserts the workload, its OIDC Secret and datastore wiring and its config,
29+
# and the CRD
2830
# exclusion one asserts the chart still ships mongodb-store's CRDs where
2931
# nvsentinel's ownsCRDsExcludeSubcharts expects them.
3032
# All are gated on testing.Short() -- `make
@@ -100,6 +102,7 @@ jobs:
100102
TestNVSentinelCRDExclusionMatchesChart
101103
TestNVSentinelSlinkyDrainerChartRender
102104
TestNVSentinelHealthEventsAnalyzerChartRender
105+
TestNVSentinelEventExporterChartRender
103106
)
104107
pattern="^($(IFS='|'; echo "${tests[*]}"))$"
105108
out=$(go test ./pkg/bundler/validations/... -run "$pattern" -v -count=1 2>&1) && rc=0 || rc=$?

‎.github/workflows/vuln-scan-images.yaml‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -332,8 +332,9 @@ jobs:
332332
# AICR does not build the images below. They ship only with the
333333
# opt-in nvsentinel-object-monitor (#2612), nvsentinel-preflight
334334
# (#2610), nvsentinel-nic-health-monitor, nvsentinel-slurm-drain-monitor
335-
# (#2611), remediation, nvsentinel-slinky-drainer (#2615) and
336-
# nvsentinel-health-events-analyzer (#2618) mixins, and no other job
335+
# (#2611), remediation, nvsentinel-slinky-drainer (#2615),
336+
# nvsentinel-health-events-analyzer (#2618) and
337+
# nvsentinel-event-exporter (#2621) mixins, and no other job
337338
# scans a mixin-gated image, so nothing else would surface a CVE in
338339
# them.
339340
# Because they are not built by this workflow they carry their
@@ -353,6 +354,7 @@ jobs:
353354
- ghcr.io/nvidia/nvsentinel/gpu-reset
354355
- ghcr.io/nvidia/nvsentinel/slinky-drainer
355356
- ghcr.io/nvidia/nvsentinel/health-events-analyzer
357+
- ghcr.io/nvidia/nvsentinel/event-exporter
356358
# Percona, from the remediation mixins' separately bundled datastore
357359
# (psmdb-operator, nvsentinel-mongodb). Pinned by AICR, by digest.
358360
- docker.io/percona/percona-server-mongodb-operator
@@ -378,8 +380,9 @@ jobs:
378380
# TestNICHealthMonitorImagePinnedEverywhere,
379381
# TestSlurmDrainMonitorImagePinnedEverywhere,
380382
# TestRemediationImagesPinnedEverywhere,
381-
# TestSlinkyDrainerImagePinnedEverywhere and
382-
# TestHealthEventsAnalyzerImagePinnedEverywhere fail if these drift
383+
# TestSlinkyDrainerImagePinnedEverywhere,
384+
# TestHealthEventsAnalyzerImagePinnedEverywhere and
385+
# TestEventExporterImagePinnedEverywhere fail if these drift
383386
# apart.
384387
- image: ghcr.io/nvidia/nvsentinel/kubernetes-object-monitor
385388
tag: v1.28.0
@@ -412,6 +415,8 @@ jobs:
412415
tag: v1.28.0@sha256:bf6ccaf1d6c035aa5f86e70a929dc6d9a18c5c05b10104d23c69aacff35bcb14
413416
- image: ghcr.io/nvidia/nvsentinel/health-events-analyzer
414417
tag: v1.28.0
418+
- image: ghcr.io/nvidia/nvsentinel/event-exporter
419+
tag: v1.28.0
415420
# AICR's own pins in recipes/components/{psmdb-operator,nvsentinel-mongodb};
416421
# TestRemediationImagesPinnedEverywhere fails if these drift apart.
417422
- image: docker.io/percona/percona-server-mongodb-operator

‎Makefile‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -450,6 +450,12 @@ nvsentinel-health-events-analyzer-e2e: ## Live Kind test: nvsentinel-health-even
450450
echo "Running nvsentinel-health-events-analyzer mixin e2e test..."; \
451451
tests/e2e/nvsentinel-health-events-analyzer/run.sh
452452

453+
.PHONY: nvsentinel-event-exporter-e2e
454+
nvsentinel-event-exporter-e2e: ## Live Kind test: nvsentinel-event-exporter delivers filtered CloudEvents with an OIDC token to a mock sink (#2621)
455+
@set -e; \
456+
echo "Running nvsentinel-event-exporter mixin e2e test..."; \
457+
tests/e2e/nvsentinel-event-exporter/run.sh
458+
453459
.PHONY: npd-nvsentinel-object-monitor-e2e
454460
npd-nvsentinel-object-monitor-e2e: ## Live Kind test: an NPD Node condition becomes an NVSentinel health event (#2614)
455461
@set -e; \

‎docs/contributor/recipe.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -347,7 +347,7 @@ Mixin files currently in the tree: `os-ubuntu`, `os-talos`, `os-amazonlinux`, `o
347347
`nvsentinel-object-monitor`, `nvsentinel-nic-health-monitor`,
348348
`nvsentinel-preflight`, `nvsentinel-slurm-drain-monitor`, `nvsentinel-observe`,
349349
`nvsentinel-quarantine`, `nvsentinel-remediation`, `nvsentinel-slinky-drainer`,
350-
`nvsentinel-health-events-analyzer`, `npd`.
350+
`nvsentinel-health-events-analyzer`, `nvsentinel-event-exporter`, `npd`.
351351

352352
The OS-scoped mixins (`os-ubuntu`, `os-talos`, `os-amazonlinux`, `os-sles`) are
353353
mutually exclusive:

‎docs/contributor/validator.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1402,10 +1402,11 @@ unrecognized value, which keeps a typo'd severity non-silent.
14021402
| `CheckDriverOwnershipCoherence` | GPU driver-ownership coherence on the final effective values (recipe merge + `--set`/`--set-json`/`--set-file` under canonical names and registry aliases): a recipe whose snapshot observed no NVIDIA driver (`metadata.gpuDriverState: absent`) must not bundle with the preinstalled-driver assumption. When GPU Operator manages the driver, `nvidia-dra-driver-gpu.nvidiaDriverRoot` must equal `gpu-operator hostPaths.driverInstallDir`; with a preinstalled driver, the DRA root must avoid the unpopulated operator container root and may intentionally differ from `hostPaths.driverInstallDir` ([#1087](https://github.com/NVIDIA/aicr/issues/1087), [#1757](https://github.com/NVIDIA/aicr/issues/1757)). Wired at `severity: error`. |
14031403
| `CheckMariaDBOperatorOwnershipCoherence` | MariaDB Operator installation safety for AICR-provided Slurm accounting: `metadata.mariaDBOperatorState` values `crs-detected` and `unknown` block bundling, `api-detected` or omitted evidence warns, and `absent` proceeds silently. Wired at `severity: warning` so warning results remain non-blocking while returned errors still fail the bundle. |
14041404
| `CheckNVSentinelDatastorePrerequisites` | Cluster prerequisites of `nvsentinel-mongodb` from snapshot evidence. `metadata.defaultStorageClassState`: `absent` and `unknown` block unless `replsets.rs0.volumeSpec.pvc.storageClassName` or `--storage-class` names a class, `multiple` warns, `present` proceeds. `metadata.perconaOperatorState`: `crs-detected`, `operator-detected`, and `unknown` block, `api-detected` warns that psmdb-operator replaces the existing CRDs, `absent` and `aicr-owned` proceed. Omitted evidence warns for either field. |
1405-
| `CheckNVSentinelRemediationPipelineCoherent` | NVSentinel's fault pipeline can work as configured: no quarantine, drain or remediation stage with the embedded `mongodbStore` or without a supported `global.datastore.provider`; each stage has the one before it (`node-drainer` needs `fault-quarantine`, `fault-remediation` needs `node-drainer`), `global.dryRun` is a boolean when `fault-remediation` is on, a live `fault-remediation` has `janitor`, `janitor` has `janitor-provider`, and `fault-remediation` has `metadata-collector`; janitor's GPU reset Job reads the driver from the root the GPU Operator populates, with a clean `driverRoot`, a RuntimeClass coherent with the enabled GPU Operator variant (none under CDI with NRI), and a service manager that waits only for operands the GPU Operator runs, in its namespace, with a custom spec's timeouts set; the GPU reset is rejected on GKE COS. With the `nvsentinel-mongodb-uri` datastore it also requires x509 auth with the `nvsentinel-mongodb-app-client` certificate and both Percona components deployed, enabled and in NVSentinel's namespace. Rejects `--dynamic` on any path it verifies. Wired at `severity: error`. |
1405+
| `CheckNVSentinelRemediationPipelineCoherent` | NVSentinel's fault pipeline can work as configured: no quarantine, drain or remediation stage, `health-events-analyzer` or `event-exporter` with the embedded `mongodbStore` or without a supported `global.datastore.provider`; each stage has the one before it (`node-drainer` needs `fault-quarantine`, `fault-remediation` needs `node-drainer`), `global.dryRun` is a boolean when `fault-remediation` is on, a live `fault-remediation` has `janitor`, `janitor` has `janitor-provider`, and `fault-remediation` has `metadata-collector`; janitor's GPU reset Job reads the driver from the root the GPU Operator populates, with a clean `driverRoot`, a RuntimeClass coherent with the enabled GPU Operator variant (none under CDI with NRI), and a service manager that waits only for operands the GPU Operator runs, in its namespace, with a custom spec's timeouts set; the GPU reset is rejected on GKE COS. With the `nvsentinel-mongodb-uri` datastore it also requires x509 auth with the `nvsentinel-mongodb-app-client` certificate and both Percona components deployed, enabled and in NVSentinel's namespace. Rejects `--dynamic` on any path it verifies. Wired at `severity: error`. |
14061406
| `CheckNVSentinelMongoDBCoherent` | `nvsentinel-mongodb` is usable by NVSentinel: NVSentinel is deployed, TLS is `requireTLS`, sharding is off, the `$external` x509 app user has `readWrite` on `HealthEventsDatabase` and the `nvsentinelTTLIndex` role, `psmdb-operator` is deployed with `watchAllNamespaces: false` and `disableTelemetry: true`, and `crVersion` and `initImage` are at the operator image's minor. Dependencies are read from the declared component set, so a `bundlers=` subset still sees them. Rejects `--dynamic` on any path it verifies. Wired at `severity: error`. |
14071407
| `CheckNVSentinelSlurmDrainMonitorRequiresSlinky` | NVSentinel's `slurm-drain-monitor` is not enabled without `slinky-slurm` (absent or disabled), or watching a blank namespace or one `slinky-slurm` does not deploy to: the monitor reads the `SlurmNodeStateDrain` condition off Slinky worker pods, which only `slinky-slurm`'s NodeSets create, so it would run and never see a pod. Rejects `--dynamic` on the monitor's toggle and namespace and on `slinky-slurm`'s `enabled`, even with the monitor off. Wired at `severity: error`. |
14081408
| `CheckNVSentinelSlurmCustomDrainCoherent` | NVSentinel drains Slurm nodes through the slinky-drainer plugin: a recipe with `slinky-slurm` and a live (not dry-run) `fault-remediation` has `node-drainer.customDrain` on, while quarantine alone passes; `customDrain` is only on with a live `node-drainer`, a boolean `global.dryRun` that is false, `slinky-slurm`, the `slinky-drainer-crds` component, and the `slinky-drainer` component in NVSentinel's namespace watching the namespace `slinky-slurm` deploys to; and its values match the plugin's contract (DrainRequest kind, resource and `DrainComplete` condition, the `slinky-drain-template` template, NVSentinel's namespace, a positive whole-seconds `timeout`, and empty `userNamespaces` and `podDrainPolicies` with an empty `nodeSelector`, since a scope could leave slurmd nodes on pod eviction); and the slurmd NodeSet and every Slurm control pod in the Slurm namespace (slinky-slurm's controller, REST API, login and enabled accounting, and `slurm-accounting-mariadb` there) render non-empty node selectors (modelled on the bundler's order: `--set`, then node-selector injection unless the overlay opts out with an empty or null value, then `--set-json`), each control pod's giving some label key a different value from slurmd's, since the plugin waits for every Ready pod in that namespace on a drained node to report Slurm DRAIN. Rejects `--dynamic` on any path a rule that can still fire reads, so a monitoring-only Slurm recipe keeps its `--dynamic` paths. Wired at `severity: error`. |
1409+
| `CheckNVSentinelEventExporterCoherent` | NVSentinel's Event Exporter can deliver: `sink.endpoint` and `oidc.tokenUrl` are `http` or `https` URLs with a host that is not an `example.com`, `.org` or `.net` placeholder; `oidc.clientId`, `oidc.scope`, `metadata.cluster` (not the chart's `my-cluster`) and `metadata.environment` are set by AICR's values as strings; no metadata value holds a double quote, backslash or control character other than tab, and every metadata key is a TOML bare key, since the chart writes them into its config unescaped; both `insecureSkipVerify` settings are `false`; and `oidcSecretName` is a valid Secret name. A no-op unless `global.eventExporter.enabled`; then rejects `--dynamic` on any `event-exporter` value. Wired at `severity: error`. |
14091410
| `CheckNPDNotDuplicatingProviderNPD` | `node-problem-detector` is bundled only on an allowlist of verified platforms — never where the provider already runs its own NPD (GKE, AKS), on OpenShift (no privileged SCC binding), or on Talos (restricted namespace) |
14101411
| `CheckGKETCPXOInterfacesCoherence` | The final resolved kubeflow-trainer `tcpxoInterfaces` value matches the mapping the recipe records in `configuration.gke.tcpxoInterfaces` |
14111412
| `CheckNVSentinelDriverLabelDetectable` | Where the GPU Operator does not own the driver and no driver pod exists (driver ships in the node image), NVSentinel sets `labeler.assumeDriverInstalled`, so its driver-label-gated DaemonSets do not come up half-rolled-out |

0 commit comments

Comments
 (0)