[BUG] AutoPause can pause from stale probe results after a same-name probe update

[AiRanthem]

What happened:

On a Running Sandbox with the AutoPauseController feature gate enabled, updating an existing named exec probe can leave the controller evaluating the old Pod condition in the same reconcile. EnsureProbe patches the Pod's kruise.io/podprobe annotation for the new spec.probes value, but then immediately mirrors the already-read Pod.Status.Conditions into Sandbox.Status.Conditions. Conditions are selected by probe name, with no configuration generation/version check.

If the old Active probe had reported True with message idle, and the probe is updated in the same spec patch to run printf active, an accompanying policy with probe: Active, messageRegex: ^idle$, and thresholdDuration: 0 can make the same reconcile call handleAutoPause with the stale idle result. The Sandbox can therefore be paused before the new probe configuration has produced a result.

This is a static code-path counterexample from the baseline source; it has not been reproduced on a live cluster.

What you expected to happen:

After a named probe's configuration changes, AutoPause should wait for a result produced by the new configuration. It should not treat a condition produced by the previous command as evidence that the updated probe is idle. With thresholdDuration: 0, pausing should be immediate only after the new probe reports a matching idle result.

How to reproduce it (as minimally and precisely as possible):

1. Use a Running, claimed Sandbox with the AutoPauseController feature gate enabled. Start with an Active exec probe whose command is equivalent to printf idle, and with no pause policy so the old result cannot pause the Sandbox before the update.

2. Wait until the Pod condition for agents.kruise.io/Active is True with message idle.

3. Apply one Sandbox.spec update that both replaces the same-named probe command with printf active and sets this pause policy:

   probes:
   - name: Active
     exec:
       command: ["sh", "-c", "printf active"]
   autoPausePolicy:
     pause:
       whenProbedIdleState:
         probe: Active
         messageRegex: '^idle$'
         thresholdDuration: 0s

4. On the first reconcile after this update, before the runtime has emitted the new probe result, the baseline path is:

   • EnsureProbe patches the Pod annotation, then calls syncConditions on the old in-memory Pod object and finds the old Active=True/idle condition;
   • handleAutoPause evaluates that mirrored condition; calculatePauseTime adds zero to its old LastTransitionTime and considers the pause due;
   • tryPause patches spec.paused=true.

   Relevant baseline source at commit cb37dce59730f1bf6a9a34fafa3e07c27b906e09: EnsureProbe, syncConditions, handleAutoPause, calculatePauseTime, and tryPause.

Anything else we need to know?:

• This affects the auto-pause behavior introduced by commit 7a16b1562b9eb72681e67099ffa454e475361b05 (Sandbox Auto-Pause and Resume, Sandbox Auto-Pause and Resume #612). The affected runtime files are unchanged from the baseline commit above; this report is independent of the current SandboxClaim auto-pause-policy change.
• Related context: found while reviewing #955; the affected code path is already present at the baseline commit above.
• The example uses a legal zero threshold. Repository validation explicitly treats zero as “pause as soon as the probe reports idle”; negative thresholds are rejected.
• No Kubernetes version, installation details, or live-cluster observation are available for this report.

Environment:

• OpenKruiseAgent version: Repository source at commit cb37dce59730f1bf6a9a34fafa3e07c27b906e09 (baseline)
• Kubernetes version (use kubectl version): Not reproduced on a live cluster
• Install details (e.g. helm install args): Not available
• Others: Static source analysis; AutoPauseController feature gate enabled in the reproduction scenario

[rajsingh1301]

@AiRanthem , traced the same path, can confirm it. Couple of things that might help:

Renaming a probe is already safe. The old condition gets removed and the new name starts at Unknown, so there's already a "wait for a fresh result" path in the code. It just doesn't kick in when the name stays the same.

The zero threshold isn't really needed for this. probeTransitionTime keeps the old LastTransitionTime when status, reason and message all match, so the old probe's idle time is already banked. A 30m threshold on a probe that's been idle for an hour would fire just as fast.

Also, re-reading the Pod after the patch won't help, since the PodProbeMarker hasn't run the new probe yet. Looks like the fix needs to track which probe config produced a result, maybe a hash of the probe spec, so a stale result can fall back to the same Unknown path renames already get.

[Gagansharma-code]

Hey @AiRanthem , I'd like to take this one, could you assign it to me?

Looking at the baseline source, the issue is that syncConditions mirrors whatever's currently in Pod.Status.Conditions keyed only by probe name, with nothing to distinguish a stale result (from the old probe command) from a fresh one produced by the updated config.

My plan is to track what produced a given condition — something like a hash of the probe spec (or a generation counter) stored alongside it — so EnsureProbe/syncConditions can tell when the current condition was produced by a since-replaced probe config, even when the name is unchanged, and fall back to Unknown in that case rather than reporting the stale result to handleAutoPause.

I'll dig into pod_probe.go and auto_pause.go and put up a PR against this.