Provide recipes and a skill for multi-node deployment on Kubernetes clusters

.agents/skills/deploy-afd-k8s/SKILL.md

@@ -0,0 +1,87 @@
+---
+name: deploy-afd-k8s
+description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is read directly off the recipe's own top-of-file comment block (however that block describes it, strict or prose): a recipe whose comment describes a per-pod placement deploys exactly that many pods with exactly that per-pod setup; a recipe without one deploys as a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e).
+---
+
+# Deploy an AFD GPU recipe on Kubernetes
+
+Stand up a serve pod (or pods) running a local AFD recipe script on a
+Kubernetes/OpenShift cluster and leave a reachable OpenAI-compatible
+endpoint behind.
+
+## Contract
+
+On success this skill guarantees, in the target namespace:
+
+| | |
+|---|---|
+| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`) |
+| Serving | the model named by `MODEL_ID` |
+| Pod(s) / Service | `vllm-pod`, or one pod per pod the recipe's header comment describes (named `vllm-pod-<pod>`); Service `vllm-service`, labels `app=afd-recipe,role=serve` |
+| Model cache | `PVC_NAME`, mounted at `/models`, `HF_HOME=/models/.hf_home` |
+| Left running | yes, deliberately -- so weights stay warm for follow-up runs |
+
+Whenever a recipe's header spans more than one pod, two more internal-only
+Services may also exist (`vllm-ffn-p2p-service` always; `vllm-attn-dp-service`
+and/or `vllm-ffn-dp-service` only if that role is itself split across more
+than one pod) -- callers never talk to these directly. See
+[services.md](references/services.md).
+
+## Scope
+
+- Recipes under `recipe/gpu/P2pNcclAFDConnector/**` (GPU only, not NPU),
+  **`prefill_decode_colocation` only** (attention + FFN workers, no prefill
+  split).
+  `prefill_decode_disaggregation` recipes are **out of scope**: they need a
+  NIXL-enabled image plus a proxy, and resolve a `SCRIPT_DIR`-relative path
+  that this skill's flat script mount (step 4b) breaks. Ask the user for a
+  different workflow for disaggregation.
+- Recipes are hand-written shell scripts, not JSON configs -- there is no
+  generator step and no manifest to read.
+- A recipe is **placement-driven** (more than one pod) exactly when its
+  top-of-file comment block describes a per-pod placement -- read that
+  block, don't infer shape from flags. Today's recipes spell it as one
+  `# For pod N: POD=...` line per pod, but the comment isn't guaranteed to
+  use that exact grammar going forward. See
+  [resolve-recipe.md](references/resolve-recipe.md).
+- Not for `tools/benchmarks/decode_bench_server.sh` (local, non-k8s).
+
+## Requirements
+
+- A live, authenticated `kubectl`/`oc` session able to create/delete Pods,
+  Services, ConfigMaps, and PVCs in the target namespace.
+- `envsubst` (from `gettext`) locally.
+- An image, pushed where the cluster can pull it. Ask the user whether to
+  use an image they already have, or to build one from `docker/Dockerfile.ci`
+  and push it to a registry they provide:
+
+  ```bash
+  IMAGE=<registry>/<repo>:<tag>
+  docker build -t "$IMAGE" -f docker/Dockerfile.ci .
+  docker push "$IMAGE"
+  ```
+
+  The image supplies only the `afd-plugin` install. The recipe script is
+  mounted fresh from local disk on every run (step 4b), so editing or adding
+  a recipe never requires a rebuild.
+- An `hf-token-secret` Secret with a `token` key, needed when `MODEL_ID` is
+  a HF Hub repo id (not needed when weights are already staged on the PVC):
+  ```bash
+  kubectl create secret generic hf-token-secret --from-literal=token=<hf_token>
+  ```
+- Nodes with enough free GPUs for `GPU_COUNT` (per pod).
+
+## Workflow
+
+Load these in order; each covers one phase and nothing else. Services must
+exist **before** any Pod is deployed -- see
+[services.md](references/services.md) for why:
+
+| Reference | Covers |
+|---|---|
+| [resolve-recipe.md](references/resolve-recipe.md) | Step 1: read the recipe's serve-block fields and model id; read its placement (or lack of one) from its header comment -- never ask. |
+| [deploy.md](references/deploy.md) | Steps 2-4f: GPU_COUNT, confirm, PVC, ConfigMap, fsGroup, Services (step 4d -- which of the four, see [services.md](references/services.md)), pod(s) via `templates/pod.yaml`, wait for Running and for each pod's own readiness line. |
+| [report-and-teardown.md](references/report-and-teardown.md) | Step 5: report the endpoint/model/nodes. Step 6: teardown (pods/Services only -- never the PVC unless asked). |
+
+All k8s manifests live in `templates/*.yaml`, applied via `envsubst | kubectl
+apply -f -`; no manifest is inlined in these docs.

.agents/skills/deploy-afd-k8s/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Deploy AFD on Kubernetes"
+  short_description: "Deploy an AFD GPU recipe on k8s and expose a ready endpoint"
+  default_prompt: "Use $deploy-afd-k8s to deploy an AFD GPU colocation recipe on Kubernetes and wait until the endpoint is ready."

.agents/skills/deploy-afd-k8s/references/deploy.md

@@ -0,0 +1,201 @@
+# 2-4. Compute GPU_COUNT, confirm, and deploy
+
+## 2. GPU_COUNT
+
+Plain single-pod recipe (no plan from [resolve-recipe.md](resolve-recipe.md)):
+union of every literal `CUDA_VISIBLE_DEVICES=` in the script.
+
+```bash
+GPU_COUNT="$(grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' "$RECIPE_SCRIPT_PATH" \
+  | cut -d= -f2 | tr ',' '\n' | sort -u | wc -l)"
+```
+
+Per plan entry: count the devices in the literal `CUDA_VISIBLE_DEVICES=` inside
+the recipe's `if` block that tests this pod's role (`ATTENTION` or `FFN` --
+whichever prefix `POD` starts with). Every pod sharing a role shares that
+literal, so this is a plain text extraction, not a per-pod computation:
+
+```bash
+role_block() {  # $1=script $2=ATTENTION|FFN
+  awk -v role="$2" '
+    /if.*\$POD/ && $0 ~ role { inblock=1; next }
+    inblock && /^fi/ { inblock=0 }
+    inblock { print }
+  ' "$1"
+}
+devices="$(role_block "$RECIPE_SCRIPT_PATH" "$ROLE" \
+  | grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' | head -1 | cut -d= -f2)"
+GPU_COUNT=$(( $(grep -o ',' <<<"$devices" | wc -l) + 1 ))
+```
+
+## 3. Confirm before touching the cluster
+
+State and confirm: target cluster/namespace, `AFD_PLUGIN_IMAGE`, `MODEL_ID`,
+`PVC_NAME`, `CLIENT_PORT`, `RECIPE_SCRIPT_PATH`, and either `GPU_COUNT`
+(plain recipe) or the plan as a table of `pod_name | role | GPU_COUNT`
+(placement-driven recipe, straight from resolve-recipe.md -- no shape to
+negotiate, just restate what was parsed). Flag that step 4 **deletes any
+existing Pod(s)** with the same name(s) about to be deployed.
+
+```bash
+AFD_PLUGIN_IMAGE=<image>
+MODEL_ID=<model-id>
+PVC_NAME=<pvc-name>
+CLIENT_PORT=<n>   # from the script's --port; default 18305
+RECIPE_SCRIPT_PATH=<local-recipe-script-path>
+```
+
+**Node placement is left to the scheduler by default.** Only add
+`podAntiAffinity` forcing Pods onto separate nodes if the caller explicitly
+asks to exercise cross-node placement -- add this to every Pod's
+`metadata.labels` and `spec` in `templates/pod.yaml` before applying:
+
+```yaml
+  labels:
+    afd-multinode-group: afd-multinode
+  spec:
+    affinity:
+      podAntiAffinity:
+        requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchExpressions:
+                - {key: afd-multinode-group, operator: In, values: ["afd-multinode"]}
+            topologyKey: kubernetes.io/hostname
+```
+
+## 4a. Model PVC
+
+Size for the model being served. An existing undersized PVC is reused
+silently and fails mid-download, so set `MODEL_PVC_SIZE` deliberately. `PVC_ACCESS_MODE` is derived, not asked:
+`ReadWriteMany` whenever the plan (if any) has more than one pod -- an RWO
+volume can only attach from one node, and the second Pod deadlocks
+`Pending` with `Multi-Attach error` otherwise.
+
+```bash
+MODEL_PVC_SIZE=${MODEL_PVC_SIZE:-100Gi}
+STORAGE_CLASS=${STORAGE_CLASS:-}
+PVC_ACCESS_MODE=ReadWriteOnce   # ReadWriteMany if the plan has >1 pod
+
+if kubectl get pvc "${PVC_NAME}" >/dev/null 2>&1; then
+  echo "PVC ${PVC_NAME} exists; reusing its warm HF_HOME cache for ${MODEL_ID}"
+  echo "NOTE: verify its size fits ${MODEL_ID}, and its accessModes include"
+  echo "      ReadWriteMany if this plan has >1 pod -- an RWO reuse deadlocks the 2nd pod"
+else
+  envsubst '${PVC_NAME} ${MODEL_PVC_SIZE} ${STORAGE_CLASS} ${PVC_ACCESS_MODE}' \
+    < templates/pvc.yaml | grep -v 'storageClassName: *$' | kubectl apply -f -
+fi
+```
+
+If an existing single-pod RWO PVC already holds downloaded weights you want
+to reuse for a multi-pod plan, provision a new RWX PVC and `cp -a` the data
+across (via a short-lived Pod pinned to the node that holds the RWO mount)
+rather than re-downloading.
+
+## 4b. Recipe ConfigMap
+
+Lets the pod run any local recipe -- edited, uncommitted, or new -- without
+rebuilding the image. Run from wherever `RECIPE_SCRIPT_PATH` resolves:
+
+```bash
+kubectl create configmap afd-recipe-script \
+  --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \
+  --dry-run=client -o yaml | kubectl apply -f -
+```
+
+## 4c. fsGroup
+
+Must be inside the namespace's allowed range or the pod is rejected at
+admission. Look it up once, reuse for every pod:
+
+```bash
+FS_GROUP="$(kubectl get ns "$(kubectl config view --minify -o jsonpath='{..namespace}')" \
+  -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' 2>/dev/null \
+  | cut -d/ -f1)"
+FS_GROUP="${FS_GROUP:-1000}"
+```
+
+## 4d. Services
+
+Create every Service that applies -- see [services.md](services.md) for
+which, and why this must happen **before** deploying any Pod in 4e below.
+Everything a Service needs (ports, selectors, which of the four apply)
+comes from the plan parsed in resolve-recipe.md, so none of this waits on a
+Pod existing.
+
+## 4e. Deploy pod(s)
+
+One call per plan entry (or a single call with no overrides for a plain
+recipe) against the one shared `templates/pod.yaml`:
+
+```bash
+deploy_pod() {
+  local pod_name="$1" gpu="$2" attn_role="$3" ffn_role="$4" extra_env_pairs="$5"
+  # extra_env_pairs: space-separated VAR=value tokens straight off this
+  # plan entry's header line (including POD=... itself), empty for a plain
+  # recipe. Build the spliced env-entries block for templates/pod.yaml.
+  # Leading (not trailing) newlines keep the `# ${TEMPLATE_EXTRA_ENV}`
+  # placeholder line a harmless comment when this is empty, and valid
+  # sibling list items when it isn't.
+  local extra_env=""
+  for pair in $extra_env_pairs; do
+    extra_env="$extra_env
+        - {name: ${pair%%=*}, value: \"${pair#*=}\"}"
+  done
+
+  kubectl delete pod "$pod_name" --ignore-not-found
+
+  TEMPLATE_IMAGE="$AFD_PLUGIN_IMAGE" TEMPLATE_MODEL="$MODEL_ID" TEMPLATE_POD="$pod_name" \
+  TEMPLATE_GPU="$gpu" TEMPLATE_PVC="$PVC_NAME" TEMPLATE_FSGROUP="$FS_GROUP" \
+  TEMPLATE_CLIENT_PORT="$CLIENT_PORT" TEMPLATE_ATTN_ROLE="$attn_role" TEMPLATE_FFN_ROLE="$ffn_role" \
+  TEMPLATE_EXTRA_ENV="$extra_env" \
+  envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_GPU} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_ATTN_ROLE} ${TEMPLATE_FFN_ROLE} ${TEMPLATE_EXTRA_ENV}' \
+    < templates/pod.yaml | kubectl apply -f -
+}
+```
+
+Plain recipe, one pod, both roles head, no overrides:
+
+```bash
+deploy_pod vllm-pod "$GPU_COUNT" head head ""
+```
+
+Placement-driven recipe, one call per entry parsed in
+[resolve-recipe.md](resolve-recipe.md), e.g. the qwen3.5 3-pod plan:
+
+```bash
+deploy_pod vllm-pod-attention-0 2 head none "POD=ATTENTION_0"
+deploy_pod vllm-pod-attention-1 2 worker none "POD=ATTENTION_1 ATTENTION_HEADLESS=1 ATTENTION_DP_START_RANK=2"
+deploy_pod vllm-pod-ffn-0 2 none head "POD=FFN_0"
+```
+
+## 4f. Wait for Running, then for each pod's own readiness line
+
+Bounded 15-minute wait per pod for `phase=Running` (unschedulable pods never
+reach `Failed`, so an unbounded loop would spin forever), then poll logs for
+`=== pod <name> READY ===` (budget 60 minutes -- model download and graph
+capture dominate):
+
+```bash
+for pod_name in <every deployed pod_name>; do
+  kubectl wait pod "$pod_name" --for=jsonpath='{.status.phase}'=Running --timeout=15m \
+    || { kubectl describe pod "$pod_name" | tail -40; exit 1; }
+done
+
+for pod_name in <every deployed pod_name>; do
+  deadline=$(( $(date +%s) + 3600 ))
+  while true; do
+    logs="$(kubectl logs "pod/$pod_name" 2>/dev/null || true)"
+    echo "$logs" | grep -q "pod ${pod_name} READY" && break
+    echo "$logs" | grep -q "ERROR:" && { kubectl logs "$pod_name" --tail=200; exit 1; }
+    [ "$(date +%s)" -ge "$deadline" ] && { kubectl logs "$pod_name" --tail=200; exit 1; }
+    sleep 10
+  done
+done
+```
+
+Only report the endpoint ready once every pod in the plan has printed its
+own `READY` line -- one pod becoming ready doesn't imply the others did.
+
+Startup markers are keyed on the fixed log names every colocation recipe
+uses today -- see `templates/pod.yaml`'s `ready_marker()` for the exact
+lines. A recipe using different log names needs that template updated.

.agents/skills/deploy-afd-k8s/references/report-and-teardown.md

@@ -0,0 +1,26 @@
+# 5. Report back
+
+Tell the caller: the endpoint `http://vllm-service:${CLIENT_PORT}`,
+`MODEL_ID` (and whether it's a HF repo id or an in-container path),
+`PVC_NAME`, `GPU_COUNT` (per pod, if more than one), the recipe deployed,
+its `--max-num-batched-tokens`, and the node(s) it landed on:
+
+```bash
+kubectl get pods -l app=afd-recipe -o wide
+```
+
+Every node matters if a follow-up pod needs to attach the same
+`ReadWriteOnce` PVC -- it can only attach from that node.
+
+# 6. Teardown
+
+Left running by design, so weights stay warm -- report what's up and how to
+remove it; do not delete unless asked:
+
+```bash
+kubectl delete pod <pod_name...>
+kubectl delete svc vllm-service vllm-ffn-p2p-service vllm-attn-dp-service vllm-ffn-dp-service --ignore-not-found
+```
+
+The model PVC is intentionally not listed -- deleting it discards the warm
+cache and forces a full re-download.