Document runc as the evaluated AKS sandbox configuration

Author: rajshah4

enterprise/k8s-install/aks.mdx

@@ -4,17 +4,30 @@ description: Prepare an Azure Kubernetes Service cluster to run OpenHands Enterp
 icon: /enterprise/images/azure-logo.svg
 ---
 
-Prepare Azure Kubernetes Service (AKS) for OpenHands Enterprise, then follow the
-[Helm installation](/enterprise/k8s-install/installation) for application configuration.
-This page covers Azure permissions, node pools, persistent storage, and ingress.
+Running OpenHands Enterprise on Azure Kubernetes Service (AKS) follows the standard
+[Helm installation](/enterprise/k8s-install/installation), with provider-specific
+choices for node pools, storage, ingress and the sandbox runtime. This guide covers
+preparing the cluster with standard runc sandboxes. Once it is ready, follow the
+Helm guide to deploy using the runtime values below. Skip
+[Installing Sysbox](/enterprise/k8s-install/sysbox). Wherever the Helm guide installs
+or checks Sysbox, use the runc values and checks on this page instead.
+
+<Warning>
+Sysbox is not yet supported for OpenHands Enterprise on AKS. This guide uses the
+node’s default runc runtime, evaluated with ordinary coding workflows (see Validation Scope). Docker builds
+and Docker Compose inside the sandbox are unavailable in this configuration.
+Other system-container workloads, such as systemd or nested containers, were not validated.
+Runc does not provide Sysbox’s additional system-container isolation; review the
+isolation requirements for your workloads before production adoption.
+</Warning>
 
 ## Cluster Requirements
 
 | Requirement | Recommendation |
 | --- | --- |
 | Access | An Azure subscription and permissions to create the resource group, cluster, node pools and networking |
-| AKS version | A supported AKS version compatible with [Sysbox](/enterprise/k8s-install/sysbox); verify the actual Ubuntu image and containerd version |
-| Sandbox OS | Ubuntu nodes with a working Sysbox runtime |
+| AKS version | An AKS-supported version; verify the actual Ubuntu image and containerd version against your target Enterprise release |
+| Sandbox OS | Ubuntu nodes using the default containerd/runc runtime |
 | Storage class | Azure Disk CSI, such as `managed-csi`, with expansion enabled |
 | Capacity | Sufficient regional and VM-family vCPU quota for application nodes, sandbox nodes and upgrades |
 
@@ -33,48 +46,94 @@ Use separate pools for application services and sandbox workloads:
 
 - **General pool:** runs OpenHands services and cluster add-ons. Keep application
   workloads here through node affinity or selectors.
-- **Sysbox pool:** an Ubuntu user node pool for agent sandboxes. Configure
-  `sysbox-install=yes` as a persistent node-pool label so new nodes receive the installer.
+- **Sandbox pool:** an Ubuntu user node pool for agent sandboxes. Set
+  `workload=openhands-sandbox` as a persistent node-pool label so new nodes match
+  the Runtime API selector. Do not install the Sysbox DaemonSet or add its installer label.
 
 The evaluation used Azure CNI overlay with Calico, two platform nodes and a sandbox
 pool with autoscaler bounds of one to two nodes. Size the pools using the
 [Sizing Guide](/enterprise/sizing-guide) and
 [Resource Limits](/enterprise/k8s-install/resource-limits), including node overhead,
 image storage and warm sandbox capacity. Validate scale-down behavior for active sessions.
 
-### Set Up the Sandbox Runtime
+AKS rejected a manual node-pool scale-down with `UnsatisfiablePDB` because of a
+sandbox PodDisruptionBudget with `maxUnavailable: 0`. Node-image and Kubernetes
+upgrades, which drain nodes, were not tested. Validate maintenance before relying
+on it; do not delete or patch PodDisruptionBudgets to force it.
 
-1. Follow [Installing Sysbox](/enterprise/k8s-install/sysbox) to install Sysbox on
-   your Ubuntu sandbox pool.
-2. Before installing OpenHands, run a Sysbox test pod with an Azure Disk workspace
-   using the [AKS installation skill and companion checks](https://github.com/OpenHands/OpenHands-Cloud/tree/832e57028298f3ad564e6ee7461ee1d9d2ee6d62/aks-install).
-   Confirm the pod starts and can write and read a file.
-3. After installing OpenHands, start a conversation and ask it to run `pwd`.
-   Confirm it returns the workspace directory.
+### Configure runc Sandboxes
 
-<Warning>
-Some AKS node images need an additional Sysbox setup step. If the test pod fails
-with an error mentioning `sysbox-runc`, contact OpenHands support before continuing.
-The AKS installation skill includes the workaround used for the evaluation.
-</Warning>
+Override the Sysbox defaults in the Enterprise chart through Helm values:
 
-Before enabling sandbox-pool autoscaling, confirm that each newly created node can
-start a sandbox. A setup correction applied to existing nodes must also be applied
-to replacement and new nodes. Validate this with your support team before relying
-on automatic scale-out or node upgrades.
+```yaml
+runtime-api:
+  env:
+    RUNTIME_CLASS: ""
+    SET_HOST_USERS: "false"
+    RUNTIME_NODE_SELECTOR: '{"workload":"openhands-sandbox"}'
+    RUNTIME_TOLERATIONS: '[]'
+```
 
-<Accordion title="Troubleshooting: Sysbox is installed but sandbox pods do not start">
-The failure was observed on AKS 1.35.8, Ubuntu 24.04.5 LTS and containerd 2.3.3-2
-with Sysbox installer v0.7.1-0. The installer registered the runtime under the old
-containerd plugin key, so its success message did not establish a working runtime.
-See [upstream issue 997](https://github.com/nestybox/sysbox/issues/997).
+An empty `RUNTIME_CLASS` omits `runtimeClassName` from new sandbox pods, using
+containerd’s default runtime. `SET_HOST_USERS: "false"` omits the `hostUsers` override;
+it does not request the Sysbox user-namespace configuration. Save these values in
+`values-aks-runc.yaml` and pass that file as the **last** `-f` argument on every
+install and upgrade, after your base values and feature overlays. For example:
 
-The evaluation workaround corrected only the Sysbox registration and restarted
-containerd on each affected sandbox node. It requires a reviewed per-node operation
-and does not automatically cover nodes added by autoscaling, reimaging or upgrades.
-The [AKS install skill](https://github.com/OpenHands/OpenHands-Cloud/blob/832e57028298f3ad564e6ee7461ee1d9d2ee6d62/.agents/skills/aks-install.md)
-contains the diagnostic and correction procedure. Verify a real test pod afterward.
-</Accordion>
+```bash
+helm upgrade --install openhands <licensed-chart-source> --version 0.74.0 \
+  --namespace openhands --create-namespace \
+  -f values.yaml -f values-automation.yaml -f values-aks-runc.yaml --timeout 10m
+```
+
+Include `values-automation.yaml` only when enabling automations. Follow the Helm
+guide for namespaces, Secrets and other prerequisites. Omitting these overrides
+can restore the chart’s Sysbox defaults and prevent new sandboxes from starting.
+If you add a sandbox-pool taint, configure a matching toleration instead of the empty list.
+
+After deployment, create a conversation and verify that its pod is in the
+sandbox pool with no `runtimeClassName` or `hostUsers` override:
+
+```bash
+kubectl get pod <sandbox-pod> -n <runtime-namespace> \
+  -o jsonpath='{.spec.nodeName}{"\t"}{.spec.runtimeClassName}{"\t"}{.spec.hostUsers}{"\n"}'
+```
+
+Expect a sandbox-pool node name followed by two empty fields. Use the configured
+runtime namespace (`openhands-runtimes` in the evaluation). The evaluated
+sandboxes ran without privileged mode, a host Docker socket or added capabilities.
+Confirm the agent executes commands and writes and rereads a workspace marker.
+Python, Node/npm, Git (public clone; authenticated pushes not tested), browser
+navigation/screenshots, HTTP previews and workspace
+persistence passed the runc evaluation. Docker-in-sandbox failed; an installed
+Docker CLI or Compose binary does not establish daemon functionality.
+See [Docker in the agent sandbox](/enterprise/docker-in-sandbox) for that feature’s requirements.
+
+### Keep Startup Capacity Ready
+
+Keep enough Ready sandbox nodes with the agent image cached to accommodate the
+expected concurrent sessions. Pool minimum counts alone do not establish available
+capacity: account for existing pods, memory requests, disk limits and node overhead.
+The tested defaults requested 500m CPU and 3 GiB memory per sandbox; a tested
+4-vCPU/16-GiB node accommodated three such sandboxes after Kubernetes reservations.
+Size production capacity using your own resources and workload measurements.
+
+In the October 6 evaluation, six starts on ready/cached capacity all succeeded in
+38–51 seconds. A separate three-start burst forcing a zero-node pool to scale up
+failed around 120 seconds: the new node became Ready at 143 seconds and sandbox
+pods at approximately 234–247 seconds. The failed startup tasks did not recover
+automatically after their pods became healthy. Three fresh starts on that same node
+with its image cached then all succeeded in 53–55 seconds.
+
+This locates the observed failure before sandbox execution, in cold provisioning
+and startup orchestration. It is not a runc-versus-Sysbox performance benchmark.
+Nodes that were already Ready with the image cached avoided this failure in testing.
+Pre-pulling images onto newly added nodes was not tested and would not remove the
+node-readiness delay. No startup deadline or recovery fix was validated. Do not
+rely on the autoscaler adding nodes, including scale-from-zero, to absorb interactive
+startups in the tested release. Validate scale-out, node replacement and maintenance
+for your target release. Pause orphaned sandboxes from failed startups through
+supported OpenHands APIs or the UI’s Stop Runtime action before retrying.
 
 ## Persistent Storage
 
@@ -84,9 +143,12 @@ Inspect the Azure Disk CSI StorageClass:
 kubectl get storageclass managed-csi -o yaml
 ```
 
-The evaluated class uses `disk.csi.azure.com`, `WaitForFirstConsumer`,
-`ReadWriteOnce` and volume expansion. Configure stateful components and workspace
-storage explicitly; the GKE `standard-rwo` and AWS gp3 examples do not apply.
+The evaluated class uses `disk.csi.azure.com`, `WaitForFirstConsumer`, reclaim
+policy `Delete` and volume expansion. The evaluated Azure Disk PVCs use
+`ReadWriteOnce`. Configure stateful components and workspace
+storage explicitly, including persistence for any in-cluster object store; the GKE
+`standard-rwo` and AWS gp3 examples do not apply. Add `STORAGE_CLASS` to the same
+`runtime-api.env` map as the runc values above; do not create duplicate YAML keys.
 
 ```yaml
 runtime-api:
@@ -100,7 +162,9 @@ postgresql:
 
 Account for node disk-attachment limits and availability-zone topology. Validate
 mounting, persisted content after reattachment, and expansion on a disposable PVC.
-The evaluation passed these checks, including expansion from 1 GiB to 2 GiB.
+The runc evaluation passed workspace read/write, stop/reopen persistence and
+cross-node disk reattachment. Expansion was not validated on the runc configuration;
+test it on a disposable PVC.
 See [Azure Disk CSI provisioning](https://learn.microsoft.com/en-us/azure/aks/create-volume-azure-disk).
 
 ## Object Storage
@@ -109,21 +173,23 @@ Conversation/session storage is separate from workspace PVCs. Helm chart 0.74.0
 supports S3-compatible and GCS filestore configuration; it does not expose an Azure
 Blob backend. Do not substitute Azure Blob credentials into the S3 settings.
 
-The evaluation used the chart's optional RustFS store on Azure Disk CSI. For production,
-select a supported durable object store and validate backup, restore and availability.
-When enabling automations, also provision the separate automation package bucket.
+The tested configuration used the chart's optional RustFS store on Azure Disk CSI.
+Select an object store that meets your durability and availability requirements,
+and verify backup and restore for that configuration.
+When enabling automations, configure a separate automation package bucket. The
+automation service can create it at startup when bucket creation is enabled.
 
 ## Database
 
-Use the [External PostgreSQL](/enterprise/external-postgres) guide for production
+Use the [External PostgreSQL](/enterprise/external-postgres) guide for managed
 database requirements and Helm values. If choosing Azure Database for PostgreSQL,
 verify compatibility, TLS and network access against those requirements before deployment.
-The AKS evaluation used bundled PostgreSQL; a managed Azure database was not validated.
+The tested configuration used bundled PostgreSQL.
 
 ## Ingress
 
 Run an ingress controller on the general pool. Traefik was validated with an Azure
-public LoadBalancer:
+public LoadBalancer. In the Traefik Helm values (chart `41.6.0` was tested):
 
 ```yaml
 service:
@@ -138,20 +204,27 @@ Follow [DNS and TLS](/enterprise/k8s-install/dns-and-tls) for trusted certificat
 and hostname configuration. Use flat runtime hostnames with Traefik standard Ingress.
 If provisioning certificates manually, assign renewal and Secret-update ownership.
 
-## Installation Skill
+## Validation Scope
 
-For agent-assisted evaluation setup, the
-[AKS installation skill](https://github.com/OpenHands/OpenHands-Cloud/blob/832e57028298f3ad564e6ee7461ee1d9d2ee6d62/.agents/skills/aks-install.md)
-and [companion files](https://github.com/OpenHands/OpenHands-Cloud/tree/832e57028298f3ad564e6ee7461ee1d9d2ee6d62/aks-install)
-provide Azure commands, values templates and storage/automation checks.
-The skill is currently proposed in [PR #1339](https://github.com/OpenHands/OpenHands-Cloud/pull/1339).
+<Note>
+Validated October 6, 2026 with Helm chart `0.74.0`, OpenHands `1.67.0`, Runtime API
+`0.10.0` and agent-server `1.49.6-python` on AKS `1.35.8`, Ubuntu `24.04.5 LTS`
+and containerd `2.3.3-2`. The configuration used bundled PostgreSQL and RustFS.
+GitHub login, trusted HTTPS, ordinary coding/browser workflows, workspace persistence,
+cross-node disk movement and execution on an autoscaler-added node, after it
+became Ready, passed. Newly added nodes ran sandboxes without per-node runtime setup.
+
+Cold autoscaling exceeded the startup deadline. An earlier resume-related HTTP 401
+and a PodDisruptionBudget issue blocking manual pool scale-down remain unresolved.
+Automations and PVC expansion were not validated on this runc configuration. Managed Azure PostgreSQL, Azure Blob, private
+endpoints, backup/restore, cross-zone recovery, tenant isolation and network
+isolation between sandboxes were not covered. No NetworkPolicy targeted the
+sandbox namespace in the evaluation. This is evaluation evidence, not production acceptance.
+</Note>
 
 ## Next Steps
 
 <CardGroup cols={2}>
-  <Card title="Installing Sysbox" icon="cube" href="/enterprise/k8s-install/sysbox">
-    Configure and verify the sandbox runtime.
-  </Card>
   <Card title="DNS and TLS" icon="lock" href="/enterprise/k8s-install/dns-and-tls">
     Configure hostnames and trusted certificates.
   </Card>

enterprise/k8s-install/sysbox.mdx

@@ -4,8 +4,15 @@ description: Install the Sysbox runtime so agent sandboxes can run securely
 icon: cube
 ---
 
-OpenHands runs each agent session in a sandbox that uses [Sysbox](https://github.com/nestybox/sysbox)
-for isolation. This guide covers installing Sysbox.
+OpenHands Enterprise can use [Sysbox](https://github.com/nestybox/sysbox)
+for sandbox isolation. This guide covers installing Sysbox.
+
+<Warning>
+Sysbox is not yet supported for OpenHands Enterprise on **Azure Kubernetes Service
+(AKS)**. Do not use this procedure for OpenHands Enterprise on AKS. See the
+[Azure AKS guide](/enterprise/k8s-install/aks) for the evaluated runc configuration
+and its Docker-in-sandbox, isolation and startup limitations.
+</Warning>
 
 ## Node Requirements