ClientRegistration controller and AuthBridge webhook disagree on serviceAccountName: bare Deployments never get credentials Secret (silent, permanent)

[huang195]

Summary

With SPIRE enabled, any workload deployed as a plain Deployment without an
explicit dedicated serviceAccountName in its pod template never receives its
per-agent Keycloak client-credentials Secret. The pod stays Pending on
FailedMount permanently. Nothing surfaces an error: no Event, no status
condition, only an Info-level log line and a 30s requeue loop that never
converges.

This is deterministic and reproducible — not the intermittent Keycloak client
registration that e2e-kind.yaml works around with a wait loop and
continue-on-error, and not related to the k8s.keycloak.org CRD (this path uses
Keycloak's admin REST API directly).

Root cause: two halves of the feature read different objects

Component	Action	Value it sees
internal/webhook/injector/pod_mutator.go:191-198	Creates a dedicated SA and sets podSpec.ServiceAccountName = crName on the admitted Pod; pre-populates the …/keycloak-client-credentials-secret-name annotation	dedicated SA ✅
internal/controller/clientregistration_controller.go:247	Reads template.Spec.ServiceAccountName from the Deployment pod template, which the webhook never patches	"" → "default" ❌

resolveKeycloakClientID() (:449-464) then returns an error, and reconcileOne
logs and requeues:

"msg":"cannot resolve Keycloak client id yet"
"reason":"SPIRE enabled: set spec.template.spec.serviceAccountName to a dedicated
          ServiceAccount (not default) on the workload for a stable SPIFFE client ID"

So the webhook has already promised the pod a Secret (by name, via the annotation)
that the controller has just declined to create. internal/clientreg/names.go's
package doc states its purpose is keeping these two sides in lockstep — they agree
on the Secret name and disagree on the input.

The webhook's own fixup is the tell: it knows the SA must not be default, creates
one, and assigns it — but only on the Pod, which the controller never reads.

Reproduction

Verified on a Kind cluster, both directions.

1. Namespace with SPIRE_ENABLED=true in authbridge-config.
2. Apply a plain Deployment with no serviceAccountName, plus an
   AgentRuntime with targetRef pointing at it.
3. Observe: controller logs cannot resolve Keycloak client id yet with the reason
   above and requeues every 30s. Zero credentials Secrets in the namespace. Pod
   Pending, FailedMount on the annotation-named Secret. State is stable
   indefinitely.
4. kubectl patch deploy <name> --type=merge -p '{"spec":{"template":{"spec":{"serviceAccountName":"<name>"}}}}'
5. Secret appears within 20s — with the exact name hash the webhook had already
   pre-populated — and the pod reaches Ready ~10s later.

Contrast: 74-deploy-weather-agent.sh creates a dedicated SA explicitly, with the
comment "required by webhook for correct SPIFFE ID derivation". That is why
weather-service works while bare Deployments silently do not.

Impact

• Any consumer deploying an agent as a plain Deployment hits this with no usable
  signal. The visible symptom (FailedMount on a missing Secret) points at
  Keycloak or the webhook, not at a serviceAccountName default.
• It masked an entire e2e suite: all 13 tests in transparent_inbound skipped on
  every shared Kind run since they merged, and the failure looked like flaky client
  registration for weeks. Fixed on the test side by adding the SA
  (test(e2e): cover inbound mTLS and the ambient/HBONE path in transparent inbound rossoctl#2404), which is only the half a caller controls.

Suggested fixes

Any one of these closes the silent-permanent-failure class; the third is worth
doing regardless:

1. Controller resolves the effective SA the way the webhook does — fall back to
   the workload-name SA when the template says ""/default and SPIRE is on,
   rather than refusing.
2. Webhook patches the Deployment template, not just the pod, so the two sides
   read the same value.
3. Surface it. A permanent refusal should not be an Info line inside a 30s
   requeue. Emit an Event on the workload and/or set a status condition, so the
   symptom names its own cause.

Minor, related

Per-agent ServiceAccounts created by ensureServiceAccount
(pod_mutator.go:835) carry no ownerReference, so nothing garbage-collects them
when the workload goes away. Eight had accumulated in two namespaces over one
development cycle. (The credentials Secret is correctly owner-referenced to the
Deployment and does self-GC.)

Assisted-By: Claude Code

[huang195]

Closing this — my severity assessment was wrong, and the issue as filed overstates it.

The mechanism I described is real: the webhook sets serviceAccountName on the admitted
Pod (pod_mutator.go:191-198) while ClientRegistrationReconciler reads the
Deployment template (clientregistration_controller.go:247), so the webhook's fixup is
invisible to the controller. But it is latent, not an active defect — every path that
actually creates a workload here already sets serviceAccountName:

• backend agents: agents_manifests.py:379,581,723,832
• backend tools: tools.py:1330,1492
• simulation_manifests.py:253
• agents_migration.py:487, which carries the explicit comment "Ensure serviceAccountName
  is set so the webhook's SPIFFE identity…"
• the SA object itself is created at kubernetes.py:320
• the deploy scripts (74-deploy-weather-agent.sh, 72-deploy-weather-tool.sh)

I hit this with a hand-written Deployment in an e2e fixture — the one caller that did not
set an SA — and generalized from that single case to "any workload," which does not hold for
any supported path.

For the record, there is also no regression here: the default refusal has been present since
the commit that introduced this controller (9637c03, 2026-03-26), and spire.enabled has
defaulted to true since 60d0d6cf, 2025-11-06.

Two residual items I still think have merit, but neither justifies this issue's framing.
Happy to file either separately if they're wanted:

1. resolveKeycloakClientID failures are unsurfaced. They emit logger.Info("cannot resolve Keycloak client id yet") plus RequeueAfter: 30s indefinitely — no Event, no
   status condition — while the pod sits on FailedMount. This applies to all of that
   function's failure modes (missing KEYCLOAK_URL/KEYCLOAK_REALM, missing Keycloak admin
   Secret, a disabled clientRegistration gate), not just the SA one. The controller was
   printing the exact reason the whole time; nothing raised it.
2. Webhook-created SAs carry no ownerReference, so nothing garbage-collects them. Eight
   had accumulated across team1/team2 during development.

Apologies for the noise.