feat(sandbox-env): opt-in HPA, PDB and 502-retry EnvoyFilter for the preview gateway

[nicacioliveira]

Context

The preview gateway (Gateway API + Istio auto-deploy) runs as a single fixed replica with no HPA or PDB. Investigation on eks-serverless showed:

• Traffic is at <1% of capacity (peak 39 req/s, 14 millicores of a 2-core limit, zero throttling) — the intermittent 502s are not gateway saturation.
• 502s (~3k/day) carry response_flags="-": the sandbox daemon itself answers 502 while the dev server inside the sandbox is booting/restarting. Istio's default retry policy does not retry an upstream that answered 502.
• The single replica is still a real availability risk: it runs on a spot node, and the node rotation on 2026-06-29/30 took the whole preview path down while the pod rescheduled.

Changes (chart sandbox-env 0.9.0 → 0.10.0, all opt-in, default-off)

• previewGateway.autoscaling — HPA targeting the Istio-generated <previewName>-istio Deployment. This is the documented pattern for auto-deployed gateways: the controller never manages replicas, so the HPA owns replica count without conflict. Defaults: min 1 / max 3 / 80% CPU.
• previewGateway.pdb — PDB (minAvailable: 1) over the generated pods, selected by the gateway.networking.k8s.io/gateway-name label. Blocks voluntary evictions (Karpenter consolidation/drift) of the only preview ingress. Does not cover spot interruptions — raise minReplicas to 2 for that.
• previewGateway.retries — EnvoyFilter adding a retry policy for the preview data path:
  • retries connect-failure,refused-stream,reset + upstream 502 only (503 is the daemon's auto-refreshing "Connecting…" page — retrying it would just delay it)
  • GET-only via retriable_request_headers on :method — no body buffering, no replayed mutations
  • 2 attempts, 0.5s–2s backoff — bridges a short dev-server restart (~3–5s); longer outages still 502 after the budget

Scoping / blast radius

The EnvoyFilter workloadSelector is hard-coded to this gateway's gateway-name label and lands in the gateway's namespace — double barrier (namespace + label). It cannot apply to any other gateway (e.g. deco-gateway-istio in istio-system, which serves sites). Template-time validations reject:

• retries.enabled=true with namespace=istio-system (root-namespace EnvoyFilter selects workloads cluster-wide)
• retries.enabled=true with a non-Istio gatewayClassName
• retries.enabled=true when the EnvoyFilter CRD is absent

Testing

• helm lint clean
• helm template verified: default values render zero new resources; all three knobs enabled render HPA/PDB/EnvoyFilter correctly scoped; all three validation guards fail with clear messages
• EnvoyFilter is v1alpha3 and coupled to Envoy route internals — smoke-test on stg after Istio upgrades (noted in the template)

Rollout

Consumed via deco-apps-cd (studio-sandbox-{stg,prod}): follow-up PR bumps targetRevision to 0.10.0 and enables retries+PDB on stg first, prod after validation.

🤖 Generated with Claude Code

---

Summary by cubic

Adds opt-in HPA, PDB, and a GET-only 502 retry policy to the preview gateway to improve resilience and cut transient 502s during sandbox restarts. Bumps the sandbox-env chart to 0.10.0; defaults keep current behavior.

• New Features

  • previewGateway.autoscaling — HPA for the Istio-generated <previewName>-istio Deployment (min 1 / max 3 / 80% CPU). Validated to require gatewayClassName=istio.
  • previewGateway.pdb — PDB (minAvailable: 1) for the generated gateway pods to block voluntary evictions. Validated to require gatewayClassName=istio.
  • previewGateway.retries — EnvoyFilter that retries GETs on connection errors and 502s (2 attempts with backoff), scoped by namespace and gateway-name label. Validations: require gatewayClassName=istio, EnvoyFilter CRD present, and namespace not istio-system.

• Migration

  • Enable with previewGateway.autoscaling.enabled=true, previewGateway.pdb.enabled=true, previewGateway.retries.enabled=true.
  • For HA, set previewGateway.autoscaling.minReplicas=2.
  • Retries need the EnvoyFilter CRD and a non-istio-system namespace.

^{Written for commit 0691f0e. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="deploy/helm/sandbox-env/templates/sandbox-preview-hpa.yaml">

<violation number="1" location="deploy/helm/sandbox-env/templates/sandbox-preview-hpa.yaml:12">
P2: Autoscaling can silently be ineffective when `previewGateway.gatewayClassName` is not `istio`, because the HPA target Deployment name is fixed to `-istio`. Consider gating `previewGateway.autoscaling.enabled` to Istio (validation) or deriving the target from the selected gateway controller instead of hard-coding the suffix.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Autoscaling can silently be ineffective when previewGateway.gatewayClassName is not istio, because the HPA target Deployment name is fixed to -istio. Consider gating previewGateway.autoscaling.enabled to Istio (validation) or deriving the target from the selected gateway controller instead of hard-coding the suffix.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At deploy/helm/sandbox-env/templates/sandbox-preview-hpa.yaml, line 12:

<comment>Autoscaling can silently be ineffective when `previewGateway.gatewayClassName` is not `istio`, because the HPA target Deployment name is fixed to `-istio`. Consider gating `previewGateway.autoscaling.enabled` to Istio (validation) or deriving the target from the selected gateway controller instead of hard-coding the suffix.</comment>

<file context>
@@ -0,0 +1,30 @@
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: {{ $previewName }}-istio
+  namespace: {{ .Values.previewGateway.namespace }}
+  labels:
</file context>

[nicacioliveira]

Fixed — extended the existing retries-only validation to all three knobs: autoscaling/pdb/retries.enabled=true each fail at template time when gatewayClassName != istio. PDB had the same issue (the gateway.networking.k8s.io/gateway-name pod label is also Istio's controller convention), so it's gated too. Deriving the generated Deployment name per controller isn't worth it here — every other controller has different generated-workload conventions, and the chart's data path (per-claim HTTPRoutes minted by mesh) is only exercised against Istio anyway.

[cubic-dev-ai]

Thanks — the Istio-specific boundary is clear, and the PDB/gateway-name detail is useful. The original autoscaling point still stands, but the broader contract is that these preview-gateway resilience knobs should be gated together to Istio-only conventions.

Thanks for the feedback! I've saved this as a new learning to improve future reviews.