fix(spurctld)!: refuse to bootstrap a second raft cluster

[pre]

Required merge order: main < #806 < #810 < #843 < #844

Related issue:

• feat(spurctld): support dynamic Raft membership changes (add/remove voter) #341.
  This change does not implement online growth, it refuses the unsafe transition that currently stands in for it. A replica with an empty store that finds a membership it does not belong to stops instead of bootstrapping a second cluster, so the replica count stays fixed at first apply. Dynamic membership through change_membership, tracked in that issue, is what removes the restriction.

Related pull requests:

• fix(spur-k8s): give SPUR_PEER_NODES resolvable pod names #807
• feat(spurctld): let a node outside the static peer list join a raft cluster #843
• feat(spurctld): add a raft membership admin API #844
• fix(spur-k8s): keep the cancel of a deleted SpurJob #808
• fix(k8s): recover a node the controller has forgotten #809
• fix(spurctld): stop serving when the raft core is dead #810

Motivation

Raising the replica count of a running spurctld StatefulSet silently destroys the
cluster. A replica that starts with an empty Raft store bootstraps a second
one-member cluster instead of joining the existing one. The two then meet, and:

• the replica that holds the data panics inside RaftCore on a higher term;
• the job id counter restarts at 1, so new jobs collide with accounting rows that
  already exist;
• both clusters answer behind the same headless Service, so a read returns
  whichever one the client reached.

Nothing in the current code stops this, and the operator gets no warning. The goal
is to make the unsafe transition impossible rather than merely documented.

Technical Details

A node with an empty store now asks every configured peer whether it already
belongs to a cluster, before it bootstraps.

raft_internal.proto gains a ClusterProbe RPC. It is internal
controller-to-controller plumbing, so slurm.proto is untouched and FFI and REST
are unaffected. The RPC is appended after the existing three, so a peer that
predates it answers UNIMPLEMENTED, which the caller reads as "unknown" and
treats as permission to continue.

The probe returns the peer's effective membership, voters and learners alike, so a learner that lost its volume is still read as a member. Three verdicts follow:

• no peer answers, or every answer is "unknown" — bootstrap, as before;
• a peer reports a membership that contains this node id — join it;
• a peer reports a membership that does not contain this node id — refuse to
  start, with a message that explains why and what to do instead.

The refusal is deliberate. Real online growth needs change_membership, which is
a feature and not a defect fix; until then the replica count is fixed at first
apply, and the example manifests and docs/deployment/kubernetes.rst now say so.

This is a breaking change: a controller that used to start now refuses to, in the
one case where starting corrupted the cluster. Hence the ! in the title.

The docs commit also documents the two execution backends (Pod mode and Agent
mode) as a choice, and corrects the spurd DaemonSet nodeSelector from
spur.amd.com/compute to spur.amd.com/managed, which is the operator's actual
default selector (crates/spur-k8s/src/main.rs). The old value matches no node.

Related:

• fix(spur-k8s): give SPUR_PEER_NODES resolvable pod names #807
• fix(spur-k8s): keep the cancel of a deleted SpurJob #808
• fix(k8s): recover a node the controller has forgotten #809
• fix(spurctld): stop serving when the raft core is dead #810

Test Plan

1. On a three node RKE2 cluster, deploy spurctld with 1 replica, submit jobs,
   then raise the replica count to 3 and observe what the new replicas do.
2. Confirm the surviving replica keeps its data and its job id counter.
3. Confirm the guard does not fire when it should not: roll a new image through a
   healthy three member cluster, and cold start three replicas from empty PVCs.
4. Unit tests for the three verdicts against a stub ClusterProbe server.

Test Result

Environment: three node RKE2 cluster on cloud VMs, 8 vCPU and 96 GiB each, no GPU.

• Before this change, the 1 to 3 scale-up produced exactly the failure above:
  spurctld-0 panicked in RaftCore, the next job id was 1 although jobs 1 and 2
  existed, and sinfo gave different answers depending on which replica served it.
  The panicked Pod stayed 1/1 Running with 0 restarts.
• After this change, the same scale-up leaves spurctld-1 and spurctld-2 in
  CrashLoopBackOff with the refusal message. spurctld-0 keeps both jobs, logs
  no panic, and the next job id is 3.
• No false refusal. A rolling image upgrade of a healthy three member cluster
  replaced all three replicas with no refusal. A cold start of three replicas from
  empty PVCs elected a leader, replicated, survived a leader kill, and readmitted
  the restarted member.
• cargo clippy --workspace --exclude spur-ffi --all-targets --locked reports
  nothing; cargo test --locked passes, including five new tests: three that
  drive the verdicts against a stub ClusterProbe gRPC server, one that reads
  a learner back from the probe of a real node, and one that lets an empty
  replica listed as a learner start.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.85714% with 12 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #806      +/-   ##
==========================================
+ Coverage   80.40%   80.44%   +0.04%     
==========================================
  Files         187      187              
  Lines       92752    92912     +160     
==========================================
+ Hits        74577    74741     +164     
+ Misses      18175    18171       -4     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.