From 65909fdb56bd8151566721e0241a5301b2d0efea Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Tue, 22 Sep 2026 10:01:22 +0300 Subject: [PATCH 01/15] allow multi-node deployment on kubernetes Signed-off-by: Shlomit Koyfman --- .agents/skills/deploy-afd-k8s/SKILL.md | 891 ++++++++++++++++++ .../skills/deploy-afd-k8s/agents/openai.yaml | 4 + .../4a2f_graph_multinode.sh | 84 ++ 3 files changed, 979 insertions(+) create mode 100644 .agents/skills/deploy-afd-k8s/SKILL.md create mode 100644 .agents/skills/deploy-afd-k8s/agents/openai.yaml create mode 100644 recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph_multinode.sh diff --git a/.agents/skills/deploy-afd-k8s/SKILL.md b/.agents/skills/deploy-afd-k8s/SKILL.md new file mode 100644 index 000000000..bee9ca174 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/SKILL.md @@ -0,0 +1,891 @@ +--- +name: deploy-afd-k8s +description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Supports single-node and (experimental) multi-node placement, the latter via recipes parameterized with ATTENTION_RANKS/FFN_RANKS env vars. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). To drive load against what this deploys, see run-vllm-bench-k8s. +--- + +# Deploy an AFD GPU recipe on Kubernetes + +Stand up a serve pod running a local AFD recipe script on a +Kubernetes/OpenShift cluster and leave a reachable OpenAI-compatible +endpoint behind. + +## Contract + +On success this skill guarantees, in the target namespace: + +| | | +|---|---| +| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`, resolved in step 1) | +| Serving | the model named by `MODEL_ID` | +| Pod / Service | `vllm-pod` / `vllm-service`, labels `app=afd-recipe,role=serve` | +| Model cache | `PVC_NAME`, mounted at `/models`, `HF_HOME=/models/.hf_home` | +| Left running | yes, deliberately -- so weights stay warm for follow-up runs | + +Callers only need `MODEL_ID`, `PVC_NAME`, and `GPU_COUNT` back. + +For a multi-node-capable recipe deployed with `node_mode: "multi"` (two +Pods, `vllm-attn-pod` / `vllm-ffn-pod`, both running the *same* recipe +script with opposite `ATTENTION_RANKS`/`FFN_RANKS` env overrides), the same +contract holds -- `vllm-service:${CLIENT_PORT}` still answers +OpenAI-compatible traffic, backed by the attention Pod. An additional +internal-only `vllm-ffn-p2p-service` also exists solely to carry AFD +rendezvous traffic between the two Pods; callers never talk to it +directly. + +## Scope + +- Recipes under `recipe/gpu/P2pNcclAFDConnector/**` (GPU only, not NPU). +- **`prefill_decode_colocation` only** -- attention + FFN workers, no + prefill split (e.g. `2a2f_graph_dbo_dp2tp1.sh`, `4a4f_eager_dbo_dp2tp2.sh`), + and their `baseline*.sh` counterparts. + `prefill_decode_disaggregation` recipes (e.g. `2p1a1f_graph_dbo.sh`) are + **out of scope**: they need a NIXL-enabled image plus a proxy in front of + the split, and they resolve `$SCRIPT_DIR/../../../../../tools/proxy_server.py` + relative to the script. Step 4b mounts the script flat at + `/recipe/recipe.sh`, so that path would resolve to a nonexistent + `/tools/proxy_server.py`. Colocation recipes reference `SCRIPT_DIR` zero + times, which is what makes the flat mount safe. Ask the user for a + different workflow for disaggregation. +- Recipes are hand-written shell scripts, not JSON configs -- there is no + generator step and no manifest to read. +- A recipe is **multi-node-capable** when it gates its `vllm serve` blocks + on `ATTENTION_RANKS`/`FFN_RANKS` env vars (e.g. + `if [ "$ATTENTION_RANKS" -gt 0 ]`) and embeds the literal string + `AFD_FFN_HOST_PLACEHOLDER` in its AFD `host` field (e.g. + `4a2f_graph_multinode.sh`) -- grep for both before assuming a script only + supports single-node. Deploying such a script with `node_mode: "multi"` + splits attention and FFN across two Pods/nodes instead of one, by running + the *same* script twice with opposite `ATTENTION_RANKS`/`FFN_RANKS` + overrides. This is **experimental**: upstream documents cross-node + `P2pNcclAFDConnector` use as "not established by the current recipes ... + treated as unverified" (`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`). + Deploy it when asked, but don't present its throughput as validated, and + default to `node_mode: "single"` unless the caller specifically wants + cross-node placement. +- Not for `tools/benchmarks/decode_bench_server.sh` (local, non-k8s). + +## Requirements + +- A live, authenticated `kubectl`/`oc` session able to create/delete Pods, + Services, ConfigMaps, and PVCs in the target namespace. +- `envsubst` (from `gettext`) locally. +- A benchmark image, pushed where the cluster can pull it, that satisfies + **all** of the following. There is no ready-made Dockerfile for this in + the repo -- ask the user which image to use, and confirm it meets this + spec before deploying: + + | Requirement | Why | + |---|---| + | vLLM base matching `pyproject.toml` (`vllm==0.26.0`) | the recipe's flags are version-specific | + | An `afd-plugin` install | the recipe loads the plugin | + | Repo sources on disk (conventionally `/opt/afd-plugin`) | recipes and `tools/` are read from the image | + | No BuildKit-only syntax (`COPY --link`, `RUN --mount`) *if* built with buildah/imagebuilder | OpenShift Builds rejects it | + | App dir group-writable (`chgrp -R 0 && chmod -R g=u `) | the restricted SCC runs a random UID in group 0 | + | `HOME` set to a writable path | otherwise `HOME=/` and anything expanding `~` fails | + | `PYTHONDONTWRITEBYTECODE=1` | the random UID cannot write `__pycache__` into the app dir | + + `docker/Dockerfile.ci` is the closest starting point in the repo, but it + does **not** meet this spec as written: it uses `COPY --link`, leaves + `HOME=/`, and does not make the app dir group-writable. Adjust those three + before using it on OpenShift, or supply an image that already conforms. + + ```bash + IMAGE=/: # must satisfy the table above + ``` + + The image supplies only the `afd-plugin` install. The recipe script is + mounted fresh from local disk on every run (step 4b), so editing or adding + a recipe never requires a rebuild. +- An `hf-token-secret` Secret with a `token` key: + ```bash + kubectl create secret generic hf-token-secret --from-literal=token= + ``` +- Nodes with `GPU_COUNT` free GPUs (step 2). + +## Workflow + +### 1. Resolve the recipe and its model + +Take the recipe path from the user, or ask. Read the script and note, per +`vllm serve` block (each backgrounded process ending `> .log 2>&1 &`): + +- `CUDA_VISIBLE_DEVICES` (a literal list on a plain single-node recipe; a + computed variable like `$ATTN_DEVICES`/`$FFN_DEVICES` on a multi-node- + capable one -- see below) +- `--data-parallel-size`, `--tensor-parallel-size` +- the `"afd": {"role": ...}` block in `--additional-config` (attention vs ffn) +- eager (`--enforce-eager`) vs graph (`--max-cudagraph-capture-size` + + `--compilation-config`) +- every other flag (`--enable-expert-parallel`, `--max-num-seqs`, + `--max-num-batched-tokens`, `--max-model-len`, `--trust-remote-code`, + host/port) + +**Check whether the script is multi-node-capable** by grepping for +`ATTENTION_RANKS`/`FFN_RANKS` env-var gating (e.g. +`if [ "$ATTENTION_RANKS" -gt 0 ]`) and the literal string +`AFD_FFN_HOST_PLACEHOLDER`. Both present means the script can run either as +one pod (defaults run the full topology locally) or as two pods (each +overriding one of `ATTENTION_RANKS`/`FFN_RANKS` to `0`). Neither present +means it's a plain fixed single-node script -- skip straight to step 2. +If the script is multi-node-capable, ask the caller whether they want +`node_mode: "single"` (default -- one pod, no env overrides needed) or +`node_mode: "multi"` (two pods); default to `"single"` per Scope unless +they specifically want cross-node placement. + +For a multi-node-capable script, also read its hardcoded topology constants +(`NUM_ATTENTION_RANKS=`, `NUM_FFN_RANKS=`, near the top -- fixed per recipe +and always sent to AFD for rendezvous regardless of which role(s) a given +pod runs locally) and its `AFD_CONNECTOR_PORT` default +(`AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-}`). Only export +`AFD_CONNECTOR_PORT` before deploying if the caller specifically asked for +a non-default rendezvous port. + +Set `CLIENT_PORT` to the `--port` value on the attention (or sole, for +baseline) `vllm serve` block -- every checked-in recipe uses `18305`, but +step 4 must rebind and expose whatever value is actually there, not assume +`18305`. + +Record `--max-num-batched-tokens` explicitly and report it to the caller -- +it caps how much prefill work a single step can absorb and is the usual +reason a load profile saturates a recipe instantly. + +Resolve `MODEL_ID`, the model the serve pod is told to serve. Read the +script's `MODEL_PATH` default plus the model directory name +(`recipe/gpu/P2pNcclAFDConnector//...`), e.g. `deepseek_v2_lite` -> +`deepseek-ai/DeepSeek-V2-Lite`. If the mapping is not obvious for an +unfamiliar family, ask rather than guess. + +`MODEL_ID` may instead be a literal in-container path (e.g. +`/models/Qwen3.5-122B-A10B-FP8`) when weights are staged on the PVC rather +than published on HF Hub -- matching the script's own `MODEL_PATH` default. +The serve pod handles either form transparently. **Tell the caller which +form it is**; consumers that load their own tokenizer need to know. + +### 2. Compute GPU_COUNT + +For a plain single-node script: `GPU_COUNT` = number of distinct GPU +indices across every literal `CUDA_VISIBLE_DEVICES=` in the script (union, +not sum -- each index appears in exactly one block). + +For a multi-node-capable script (step 1 found `ATTENTION_RANKS`/ +`FFN_RANKS` gating), `CUDA_VISIBLE_DEVICES` is computed at runtime rather +than literal, so derive `GPU_COUNT` from the constants read in step 1 +instead: +- `node_mode: "single"`: `GPU_COUNT = NUM_ATTENTION_RANKS + NUM_FFN_RANKS` + (both roles run on the one pod, sharing a contiguous device pool). +- `node_mode: "multi"`: two counts instead of one -- + `GPU_COUNT_ATTN = NUM_ATTENTION_RANKS`, `GPU_COUNT_FFN = NUM_FFN_RANKS`. + +### 3. Confirm before touching the cluster + +State and confirm: target cluster/namespace, `AFD_PLUGIN_IMAGE`, `MODEL_ID`, +`PVC_NAME`, `GPU_COUNT`, `CLIENT_PORT`, `RECIPE_SCRIPT_PATH`. Flag explicitly +that step 4c **deletes any existing `vllm-pod`** (or +`vllm-attn-pod`/`vllm-ffn-pod` in the multi-node variant) -- if one exists +from unrelated work, that is destructive and needs approval first. + +`RECIPE_SCRIPT_PATH` is a *local* path. It need not be committed, exist in +the image, or live at any particular depth -- but per Scope it must be a +colocation recipe, since nothing rewrites `SCRIPT_DIR`-relative lookups. + +```bash +AFD_PLUGIN_IMAGE= +MODEL_ID= +PVC_NAME= +GPU_COUNT= +CLIENT_PORT= # from the script's --port; default 18305 +RECIPE_SCRIPT_PATH= +VLLM_USE_V2_MODEL_RUNNER=${VLLM_USE_V2_MODEL_RUNNER:-0} # pulled from the caller's env, default 0 +``` + +When step 1 identified a `node_mode: "multi"` deploy, confirm two GPU +counts instead of one -- `RECIPE_SCRIPT_PATH` stays a single path, since the +same script deploys to both pods with opposite `ATTENTION_RANKS`/ +`FFN_RANKS`: + +```bash +GPU_COUNT_ATTN= +GPU_COUNT_FFN= +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-} # only export if overriding +``` + +### 4. Deploy + +**4a. Model PVC** (created once; reused so weights stay warm). + +Size it for the model: a DeepSeek-V2-Lite needs far less than a +Qwen3.5-122B-A10B-FP8 checkpoint (~122 GB, so `200Gi` minimum). Set +`MODEL_PVC_SIZE` deliberately -- an existing undersized PVC is *reused +silently* by the branch below and the download then fails with ENOSPC after +the GPUs are already claimed. Set `STORAGE_CLASS` to a class the cluster +actually offers; leaving it unset falls to the default, which is often +RWO-only or absent and leaves the pod `Pending`. + +For `node_mode: "multi"`, the PVC **must** be `ReadWriteMany` (RWX), not +`ReadWriteOnce`. The attention and FFN Pods are forced onto different nodes +by anti-affinity (step 4c-multi), and an RWO volume can only attach on one +node at a time -- the second Pod fails at admission with `Multi-Attach +error for volume ... already used by pod(s) ...` and sits `Pending` +forever. This is a hard deadlock, not something that resolves by waiting +longer. Before deploying multi-node, confirm the storage class actually +supports RWX (a throwaway 1Gi PVC requesting `ReadWriteMany` against the +same `STORAGE_CLASS` either binds or it doesn't -- delete it after). If an +existing single-node PVC is RWO and already holds downloaded weights you +want to reuse for a multi-node deploy, provision a new RWX PVC and copy the +data across rather than re-downloading -- e.g. a short-lived Pod pinned via +`nodeName` to whichever node already holds the RWO mount (same-node +multi-mount of an RWO volume is fine), with both PVCs as volumes, running +`cp -a /old/. /new/.` (a nonzero exit purely from failing to preserve the +top-level directory's mtime is cosmetic; verify with `du -sh` on both sides +instead of trusting the exit code). + +```bash +MODEL_PVC_SIZE=${MODEL_PVC_SIZE:-100Gi} +STORAGE_CLASS=${STORAGE_CLASS:-} # e.g. ocs-storagecluster-cephfs +PVC_ACCESS_MODE=${PVC_ACCESS_MODE:-ReadWriteOnce} # ReadWriteMany for node_mode "multi" + +if kubectl get pvc "${PVC_NAME}" >/dev/null 2>&1; then + have="$(kubectl get pvc "${PVC_NAME}" -o jsonpath='{.spec.resources.requests.storage}')" + have_modes="$(kubectl get pvc "${PVC_NAME}" -o jsonpath='{.spec.accessModes}')" + echo "PVC ${PVC_NAME} exists (${have}, ${have_modes}); serving ${MODEL_ID} from its warm HF_HOME cache" + echo "NOTE: verify ${have} fits ${MODEL_ID} -- an undersized reused PVC fails mid-download" + echo "NOTE: for node_mode multi, ${have_modes} must include ReadWriteMany or the FFN Pod will deadlock on attach" +else + envsubst '${PVC_NAME} ${MODEL_PVC_SIZE} ${STORAGE_CLASS} ${PVC_ACCESS_MODE}' <<'EOF' | \ + grep -v 'storageClassName: *$' | kubectl apply -f - +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ${PVC_NAME} + labels: + app: afd-recipe +spec: + accessModes: + - ${PVC_ACCESS_MODE} + storageClassName: ${STORAGE_CLASS} + resources: + requests: + storage: ${MODEL_PVC_SIZE} +EOF +fi +``` + +**4b. Recipe ConfigMap.** This is what lets the pod run any local recipe -- +edited, uncommitted, or brand new -- without rebuilding the image. Run from +wherever `RECIPE_SCRIPT_PATH` resolves (e.g. the repo root): + +```bash +kubectl create configmap afd-recipe-script \ + --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +**4c. Serve pod.** The container runs the mounted recipe unmodified except +for rebinding the client-facing port off loopback: the recipe binds +`:${CLIENT_PORT}` on `127.0.0.1` (it is normally driven from inside the same +host), but the Service needs `0.0.0.0`. Every internal worker port stays on +loopback. The `uv` shim on `PATH` strips `uv run` -- the image's base +environment is already correct. + +`fsGroup` must be inside the namespace's allowed range or the pod is +rejected at admission (`fsGroup: Invalid value: ... is not an allowed +group`). Look it up rather than copying a number: + +```bash +FS_GROUP="$(kubectl get ns "$(kubectl config view --minify -o jsonpath='{..namespace}')" \ + -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' 2>/dev/null \ + | cut -d/ -f1)" +FS_GROUP="${FS_GROUP:-1000}" +echo "using fsGroup=${FS_GROUP}" + +TEMPLATE_IMAGE="$AFD_PLUGIN_IMAGE" +TEMPLATE_MODEL="$MODEL_ID" +TEMPLATE_CLIENT_PORT="$CLIENT_PORT" + +kubectl delete pod vllm-pod --ignore-not-found + +envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${GPU_COUNT} ${PVC_NAME} ${FS_GROUP} ${TEMPLATE_CLIENT_PORT} ${VLLM_USE_V2_MODEL_RUNNER}' <<'EOF' | kubectl apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: vllm-pod + labels: + app: afd-recipe + role: serve +spec: + restartPolicy: Never + securityContext: + fsGroup: ${FS_GROUP} + volumes: + - name: model-storage + persistentVolumeClaim: + claimName: ${PVC_NAME} + - name: dshm + emptyDir: + medium: Memory + sizeLimit: 16Gi + - name: work + emptyDir: {} + - name: recipe-script + configMap: + name: afd-recipe-script + containers: + - name: afd + image: ${TEMPLATE_IMAGE} + imagePullPolicy: Always + command: + - /bin/bash + - -c + - | + set -euo pipefail + RECIPE_SCRIPT=/recipe/recipe.sh + mkdir -p /work/bin + # Resolve the real uv BEFORE putting the shim on PATH (afterwards + # `command -v uv` would find the shim and recurse). `|| true` keeps a + # missing uv from killing the container on line 1: the shim exists + # only to strip `uv run`, which needs no uv at all. + REAL_UV="$(command -v uv || true)" + printf '#!/bin/bash\nif [ "$1" = "run" ]; then\n shift\n exec "$@"\nfi\nREAL_UV="%s"\n[ -n "$REAL_UV" ] || { echo "uv not found in image" >&2; exit 127; }\nexec "$REAL_UV" "$@"\n' "$REAL_UV" > /work/bin/uv + chmod +x /work/bin/uv + export PATH="/work/bin:$PATH" + + echo "=== launching $(basename "$RECIPE_SCRIPT") recipe ===" + + RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" + PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" + awk ' + { line[NR] = $0 } + END { + for (i = 1; i <= NR; i++) { + l = line[i] + if (l ~ /--host 127\.0\.0\.1/ && i < NR && line[i+1] ~ /--port ${TEMPLATE_CLIENT_PORT}/) { + gsub(/127\.0\.0\.1/, "0.0.0.0", l) + } + print l + } + }' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" + + cd /work + bash "$PATCHED_SCRIPT" & + RECIPE_PID=$! + + LOGS="$(grep -oE '>[[:space:]]*[A-Za-z0-9_]+\.log' "$RECIPE_SCRIPT" \ + | grep -oE '[A-Za-z0-9_]+\.log' | sort -u | tr '\n' ' ')" + + fail() { + echo "ERROR: $*" + for log in $LOGS; do + echo "----- tail $log -----" + tail -n 100 "/work/$log" 2>/dev/null || echo "(missing)" + done + sleep infinity + } + + if [ -z "$LOGS" ]; then + fail "no '> .log' redirects in $RECIPE_SCRIPT; cannot tell which processes to wait for" + fi + + echo "=== wait for every worker to report startup ===" + ready_marker() { + case "$1" in + ffn.log) echo "AFD FFN EngineCore started" ;; + *) echo "Application startup complete" ;; + esac + } + for log in $LOGS; do + marker="$(ready_marker "$log")" + echo "--- waiting on $log (marker: $marker) ---" + ready=0 + for i in $(seq 1 240); do + if [ -f "/work/$log" ] && grep -q "$marker" "/work/$log"; then + echo "$log: ready" + ready=1 + break + fi + if ! kill -0 "$RECIPE_PID" 2>/dev/null; then + fail "recipe process exited before $log came up" + fi + sleep 10 + done + [ "$ready" = "1" ] || fail "timed out waiting for $log" + done + + echo "--- waiting on attention server /health (127.0.0.1:${TEMPLATE_CLIENT_PORT}) ---" + endpoint_ready=0 + for i in $(seq 1 60); do + if curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1; then + echo "attention server: ready" + endpoint_ready=1 + break + fi + sleep 5 + done + [ "$endpoint_ready" = "1" ] || fail "attention server did not answer /health" + + echo "=== stack READY; drive load at 127.0.0.1:${TEMPLATE_CLIENT_PORT} ===" + sleep infinity + env: + - name: USER + value: "vllm" + - name: HOME + value: "/work/home" + - name: MODEL_PATH + value: "${TEMPLATE_MODEL}" + - name: HF_HOME + value: "/models/.hf_home" + - name: XDG_CACHE_HOME + value: "/work/xdg" + - name: TORCHINDUCTOR_CACHE_DIR + value: "/work/inductor" + - name: TRITON_CACHE_DIR + value: "/work/triton" + - name: VLLM_CACHE_ROOT + value: "/work/vllm" + - name: UV_CACHE_DIR + value: "/work/uv" + - name: VLLM_LOGGING_LEVEL + value: "INFO" + - name: VLLM_USE_V2_MODEL_RUNNER + value: "${VLLM_USE_V2_MODEL_RUNNER}" + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: token + resources: + requests: + nvidia.com/gpu: "${GPU_COUNT}" + cpu: "16" + memory: 128Gi + limits: + nvidia.com/gpu: "${GPU_COUNT}" + cpu: "32" + memory: 200Gi + volumeMounts: + - name: model-storage + mountPath: /models + - name: dshm + mountPath: /dev/shm + - name: work + mountPath: /work + - name: recipe-script + mountPath: /recipe + readOnly: true +EOF +``` + +**4d. Service** (idempotent): + +```bash +kubectl apply -f - </dev/null)" = "Running" ]; do + phase="$(kubectl get pod vllm-pod -o jsonpath='{.status.phase}' 2>/dev/null || true)" + if [ "$phase" = "Failed" ]; then + echo "pod Failed"; kubectl logs vllm-pod --tail=50; exit 1 + fi + if [ "$(date +%s)" -ge "$deadline" ]; then + echo "timed out after 15m waiting for Running (phase=${phase:-})" + kubectl describe pod vllm-pod | tail -40 + exit 1 + fi + sleep 10 +done +``` + +**4f. Wait for the stack to report ready.** Model download and graph capture +dominate here; a cold PVC on a large model can legitimately take a while, so +this budget is generous: + +```bash +deadline=$(( $(date +%s) + 3600 )) +while true; do + pod_logs="$(kubectl logs pod/vllm-pod 2>/dev/null || true)" + echo "$pod_logs" | grep -q "stack READY" && break + if echo "$pod_logs" | grep -q "ERROR:"; then + echo "serve pod hit an error during startup"; kubectl logs vllm-pod --tail=200; exit 1 + fi + if [ "$(kubectl get pod vllm-pod -o jsonpath='{.status.phase}' 2>/dev/null)" = "Failed" ]; then + echo "pod Failed"; kubectl logs vllm-pod --tail=200; exit 1 + fi + if [ "$(date +%s)" -ge "$deadline" ]; then + echo "timed out after 60m waiting for 'stack READY'" + kubectl logs vllm-pod --tail=200; exit 1 + fi + sleep 10 +done +echo "=== serve pod ready at http://vllm-service:${CLIENT_PORT} ===" +``` + +### 4-multi. Multi-node variant + +Only applies when step 1 identified a multi-node-capable script and the +caller wants `node_mode: "multi"`. Step 4a (the model PVC) is shared -- both +Pods mount the same PVC -- but per step 4a's note it must be provisioned +`ReadWriteMany`, not the single-node default `ReadWriteOnce`, or the FFN +Pod deadlocks trying to attach on its own node. Step 4b (the recipe +ConfigMap) is also shared as-is: it's the *same* script for both pods now, +so there is only one ConfigMap, not one per role. Everything below +replaces steps 4c-4f. + +**Resolve the FFN host placeholder.** The recipe script contains the +literal string `AFD_FFN_HOST_PLACEHOLDER` wherever `host` appears inside +`--additional-config`. Per `docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`, +`host` must resolve to FFN's first rank, and every participating rank +(including attention) must agree on the same value. Since both pods mount +the *same* ConfigMap, substitute once, before creating it: + +```bash +FFN_HOST="vllm-ffn-p2p-service" # short form resolves within the namespace +sed -i "s/AFD_FFN_HOST_PLACEHOLDER/${FFN_HOST}/g" "$RECIPE_SCRIPT_PATH" +``` + +**Recipe ConfigMap (shared, same as step 4b):** + +```bash +kubectl create configmap afd-recipe-script \ + --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +**4c-multi. Two Pods, forced onto different nodes.** Same container script +body, env, and `uv` shim as step 4c, both mounting the one shared +ConfigMap. Differences from step 4c: each Pod passes `ATTENTION_RANKS`/ +`FFN_RANKS` env vars so the script's own gating +(`if [ "$ATTENTION_RANKS" -gt 0 ]` / `if [ "$FFN_RANKS" -gt 0 ]`) runs only +its role locally -- the attention Pod sets `ATTENTION_RANKS=$GPU_COUNT_ATTN +FFN_RANKS=0`, the FFN Pod sets `ATTENTION_RANKS=0 FFN_RANKS=$GPU_COUNT_FFN`; +each Pod requests only its own role's GPU count (`GPU_COUNT_ATTN` / +`GPU_COUNT_FFN`, not the union); and the two Pods share an +`afd-multinode-group` label so a `podAntiAffinity` can force them onto +different nodes -- matching GPU requests alone don't guarantee that on a +cluster with multiple GPU nodes. Only the attention Pod's startup wait hits +`/health`; the FFN Pod only waits on its own log marker (`ffn.log` -> `AFD +FFN EngineCore started`) and never opens an HTTP port, so a `/health` check +there would hang forever. + +Because the shared script always contains both `> attn.log` and +`> ffn.log` redirects (one inside each role's `if` block), grepping the +script text for log redirects -- as step 4c does for plain scripts -- +would wrongly expect a log that this pod's disabled role never produces. +Derive the expected log(s) instead from this pod's own +`ATTENTION_RANKS`/`FFN_RANKS` values: + +```bash +kubectl delete pod vllm-attn-pod vllm-ffn-pod --ignore-not-found + +deploy_role_pod() { + local role="$1" pod_name="$2" attn_ranks="$3" ffn_ranks="$4" + local gpu_n=$((attn_ranks + ffn_ranks)) + TEMPLATE_IMAGE="$AFD_PLUGIN_IMAGE" TEMPLATE_MODEL="$MODEL_ID" TEMPLATE_POD="$pod_name" \ + TEMPLATE_ROLE="$role" TEMPLATE_ATTN_RANKS="$attn_ranks" TEMPLATE_FFN_RANKS="$ffn_ranks" \ + TEMPLATE_GPU="$gpu_n" TEMPLATE_GROUP="afd-multinode" TEMPLATE_PVC="$PVC_NAME" \ + TEMPLATE_FSGROUP="$FS_GROUP" TEMPLATE_CLIENT_PORT="$CLIENT_PORT" \ + TEMPLATE_AFD_PORT="${AFD_CONNECTOR_PORT:-}" \ + envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_ROLE} ${TEMPLATE_ATTN_RANKS} ${TEMPLATE_FFN_RANKS} ${TEMPLATE_GPU} ${TEMPLATE_GROUP} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_AFD_PORT} ${VLLM_USE_V2_MODEL_RUNNER}' <<'EOF' | kubectl apply -f - +apiVersion: v1 +kind: Pod +metadata: + name: ${TEMPLATE_POD} + labels: + app: afd-recipe + role: serve + afd-role: ${TEMPLATE_ROLE} + afd-multinode-group: ${TEMPLATE_GROUP} +spec: + restartPolicy: Never + securityContext: + fsGroup: ${TEMPLATE_FSGROUP} + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - {key: afd-multinode-group, operator: In, values: ["${TEMPLATE_GROUP}"]} + topologyKey: kubernetes.io/hostname + volumes: + - name: model-storage + persistentVolumeClaim: + claimName: ${TEMPLATE_PVC} + - name: dshm + emptyDir: {medium: Memory, sizeLimit: 16Gi} + - name: work + emptyDir: {} + - name: recipe-script + configMap: + name: afd-recipe-script + containers: + - name: afd + image: ${TEMPLATE_IMAGE} + imagePullPolicy: Always + command: + - /bin/bash + - -c + - | + set -euo pipefail + RECIPE_SCRIPT=/recipe/recipe.sh + mkdir -p /work/bin + REAL_UV="$(command -v uv || true)" + printf '#!/bin/bash\nif [ "$1" = "run" ]; then\n shift\n exec "$@"\nfi\nREAL_UV="%s"\n[ -n "$REAL_UV" ] || { echo "uv not found in image" >&2; exit 127; }\nexec "$REAL_UV" "$@"\n' "$REAL_UV" > /work/bin/uv + chmod +x /work/bin/uv + export PATH="/work/bin:$PATH" + + echo "=== launching $(basename "$RECIPE_SCRIPT") recipe (role: ${TEMPLATE_ROLE}) ===" + + RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" + PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" + awk ' + { line[NR] = $0 } + END { + for (i = 1; i <= NR; i++) { + l = line[i] + if (l ~ /--host 127\.0\.0\.1/ && i < NR && line[i+1] ~ /--port ${TEMPLATE_CLIENT_PORT}/) { + gsub(/127\.0\.0\.1/, "0.0.0.0", l) + } + print l + } + }' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" + + cd /work + bash "$PATCHED_SCRIPT" & + RECIPE_PID=$! + + LOGS="" + [ "${TEMPLATE_ATTN_RANKS}" -gt 0 ] && LOGS="$LOGS attn.log" + [ "${TEMPLATE_FFN_RANKS}" -gt 0 ] && LOGS="$LOGS ffn.log" + + fail() { + echo "ERROR: $*" + for log in $LOGS; do + echo "----- tail $log -----" + tail -n 100 "/work/$log" 2>/dev/null || echo "(missing)" + done + sleep infinity + } + + [ -n "$LOGS" ] || fail "this pod's ATTENTION_RANKS/FFN_RANKS are both 0 -- nothing to run" + + ready_marker() { + case "$1" in + ffn.log) echo "AFD FFN EngineCore started" ;; + *) echo "Application startup complete" ;; + esac + } + for log in $LOGS; do + marker="$(ready_marker "$log")" + echo "--- waiting on $log (marker: $marker) ---" + ready=0 + for i in $(seq 1 240); do + if [ -f "/work/$log" ] && grep -q "$marker" "/work/$log"; then + ready=1; break + fi + kill -0 "$RECIPE_PID" 2>/dev/null || fail "recipe process exited before $log came up" + sleep 10 + done + [ "$ready" = "1" ] || fail "timed out waiting for $log" + done + + if [ "${TEMPLATE_ATTN_RANKS}" -gt 0 ]; then + echo "--- waiting on attention server /health (127.0.0.1:${TEMPLATE_CLIENT_PORT}) ---" + endpoint_ready=0 + for i in $(seq 1 60); do + curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1 && { endpoint_ready=1; break; } + sleep 5 + done + [ "$endpoint_ready" = "1" ] || fail "attention server did not answer /health" + fi + + echo "=== ${TEMPLATE_ROLE} role READY ===" + sleep infinity + env: + - {name: USER, value: "vllm"} + - {name: HOME, value: "/work/home"} + - {name: MODEL_PATH, value: "${TEMPLATE_MODEL}"} + - {name: HF_HOME, value: "/models/.hf_home"} + - {name: XDG_CACHE_HOME, value: "/work/xdg"} + - {name: TORCHINDUCTOR_CACHE_DIR, value: "/work/inductor"} + - {name: TRITON_CACHE_DIR, value: "/work/triton"} + - {name: VLLM_CACHE_ROOT, value: "/work/vllm"} + - {name: UV_CACHE_DIR, value: "/work/uv"} + - {name: VLLM_LOGGING_LEVEL, value: "INFO"} + - {name: VLLM_USE_V2_MODEL_RUNNER, value: "${VLLM_USE_V2_MODEL_RUNNER}"} + - {name: ATTENTION_RANKS, value: "${TEMPLATE_ATTN_RANKS}"} + - {name: FFN_RANKS, value: "${TEMPLATE_FFN_RANKS}"} + - {name: AFD_CONNECTOR_PORT, value: "${TEMPLATE_AFD_PORT}"} + - name: HF_TOKEN + valueFrom: + secretKeyRef: {name: hf-token-secret, key: token} + resources: + requests: {nvidia.com/gpu: "${TEMPLATE_GPU}", cpu: "16", memory: 128Gi} + limits: {nvidia.com/gpu: "${TEMPLATE_GPU}", cpu: "32", memory: 200Gi} + volumeMounts: + - {name: model-storage, mountPath: /models} + - {name: dshm, mountPath: /dev/shm} + - {name: work, mountPath: /work} + - {name: recipe-script, mountPath: /recipe, readOnly: true} +EOF +} + +deploy_role_pod attention vllm-attn-pod "$GPU_COUNT_ATTN" 0 +deploy_role_pod ffn vllm-ffn-pod 0 "$GPU_COUNT_FFN" +``` + +`AFD_CONNECTOR_PORT` above defaults to an empty string when the caller +didn't override it -- an empty env value still falls through to the +script's own `${AFD_CONNECTOR_PORT:-}` default, so it's safe to +always pass the container env entry. + +**4d-multi. Two Services.** `vllm-service` keeps its existing shape and +contract (client-facing, port `${CLIENT_PORT}`) but now selects `afd-role: attention` +instead of the generic `role: serve`: + +```bash +kubectl apply -f - < `AFD FFN + EngineCore started`, everything else -> `Application startup complete`). A + recipe using different log names gets the generic marker. +- Multi-node placement requires a recipe written in the parameterized + style (`ATTENTION_RANKS`/`FFN_RANKS` gating + `AFD_FFN_HOST_PLACEHOLDER`, + e.g. `4a2f_graph_multinode.sh`). Older fixed two-block recipes (both + roles backgrounded unconditionally, literal `CUDA_VISIBLE_DEVICES`) only + support `node_mode: "single"` -- don't attempt to split one of those + across two Pods. +- The multi-node variant is experimental: it deploys correctly (two Pods + forced onto different nodes, correct port exposure per + `docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`), but upstream has not + validated `P2pNcclAFDConnector` correctness or performance across nodes. + NCCL runs over whatever the cluster's pod network provides -- no RDMA is + assumed. Treat throughput numbers from a multi-node deployment as + exploratory, and say so when reporting results. diff --git a/.agents/skills/deploy-afd-k8s/agents/openai.yaml b/.agents/skills/deploy-afd-k8s/agents/openai.yaml new file mode 100644 index 000000000..e2f15ea63 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Deploy AFD on Kubernetes" + short_description: "Deploy an AFD GPU recipe on k8s and expose a ready endpoint" + default_prompt: "Use $deploy-afd-k8s to deploy an AFD GPU colocation recipe on Kubernetes and wait until the endpoint is ready." diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph_multinode.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph_multinode.sh new file mode 100644 index 000000000..2e6b364d1 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph_multinode.sh @@ -0,0 +1,84 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3.5-122B-A10B-FP8} +export VLLM_USE_V2_MODEL_RUNNER=0 + +# Fixed topology for this recipe -- always sent to AFD for rendezvous, +# regardless of which role(s) this node launches locally. +NUM_ATTENTION_RANKS=4 +NUM_FFN_RANKS=2 + +# How many ranks of each role to launch on THIS node. 0 skips that role. +# Default runs the full topology on one node (single-node colocation); +# a multi-node deploy overrides one of these to 0 per pod. +ATTENTION_RANKS=${ATTENTION_RANKS:-$NUM_ATTENTION_RANKS} +FFN_RANKS=${FFN_RANKS:-$NUM_FFN_RANKS} + +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-1239} + +if [ "$ATTENTION_RANKS" -gt 0 ]; then + ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_RANKS - 1))) + CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$ATTENTION_RANKS" \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --dtype bfloat16 \ + --language-model-only \ + --max-model-len 114688 \ + --mamba-cache-mode align \ + --all2all-backend allgather_reducescatter \ + --seed 0 \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "vllm-ffn-p2p-service", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 32 \ + --max-num-batched-tokens 8192 \ + --max-cudagraph-capture-size 32 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] + }' \ + --host 127.0.0.1 \ + --port 18305 > attn.log 2>&1 & +fi + +if [ "$FFN_RANKS" -gt 0 ]; then + FFN_DEVICES=$(seq -s, "$ATTENTION_RANKS" $((ATTENTION_RANKS + FFN_RANKS - 1))) + CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$FFN_RANKS" \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --dtype bfloat16 \ + --language-model-only \ + --max-model-len 114688 \ + --mamba-cache-mode align \ + --all2all-backend allgather_reducescatter \ + --seed 0 \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "vllm-ffn-p2p-service", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 32 \ + --max-num-batched-tokens 8192 \ + --max-cudagraph-capture-size 32 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] + }' \ + --host 127.0.0.1 \ + --port 18305 > ffn.log 2>&1 & +fi + +wait From 4bcbd52c2e036eccd756a170b446c2b108beaa29 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Wed, 23 Sep 2026 16:04:12 +0300 Subject: [PATCH 02/15] generalizing the skill Signed-off-by: Shlomit Koyfman --- .agents/skills/deploy-afd-k8s/SKILL.md | 687 ++++++++++++------ .../2a2f_eager_dbo_dp1tp2.sh | 144 ++-- .../2a2f_eager_dbo_dp2tp1.sh | 144 ++-- .../2a2f_graph_dbo_dp1tp2.sh | 156 ++-- .../2a2f_graph_dbo_dp2tp1.sh | 155 ++-- .../4a4f_eager_dbo_dp2tp2.sh | 144 ++-- .../4a4f_graph_dbo_dp2tp2.sh | 156 ++-- ...{4a2f_graph_multinode.sh => 4a2f_graph.sh} | 58 +- 8 files changed, 1101 insertions(+), 543 deletions(-) rename recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/{4a2f_graph_multinode.sh => 4a2f_graph.sh} (51%) diff --git a/.agents/skills/deploy-afd-k8s/SKILL.md b/.agents/skills/deploy-afd-k8s/SKILL.md index bee9ca174..30d2c0dfa 100644 --- a/.agents/skills/deploy-afd-k8s/SKILL.md +++ b/.agents/skills/deploy-afd-k8s/SKILL.md @@ -1,6 +1,6 @@ --- name: deploy-afd-k8s -description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Supports single-node and (experimental) multi-node placement, the latter via recipes parameterized with ATTENTION_RANKS/FFN_RANKS env vars. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). To drive load against what this deploys, see run-vllm-bench-k8s. +description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. For multi-node-capable recipes (parameterized with ATTENTION_DP_RANKS/FFN_DP_RANKS env vars), ask the caller for an explicit placement plan -- how many attention/FFN ranks go in each Pod, and how many Pods -- rather than assuming a shape; a plan can range from one Pod holding every rank to many Pods with arbitrary, possibly mixed, per-Pod shares (experimental beyond the single-Pod case). Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). To drive load against what this deploys, see run-vllm-bench-k8s. --- # Deploy an AFD GPU recipe on Kubernetes @@ -17,20 +17,22 @@ On success this skill guarantees, in the target namespace: |---|---| | Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`, resolved in step 1) | | Serving | the model named by `MODEL_ID` | -| Pod / Service | `vllm-pod` / `vllm-service`, labels `app=afd-recipe,role=serve` | +| Pod(s) / Service | one or more Pods, named per the caller's placement plan (step 1b) for a multi-node-capable recipe, or the single `vllm-pod` for a plain fixed recipe; Service `vllm-service`, labels `app=afd-recipe,role=serve` | | Model cache | `PVC_NAME`, mounted at `/models`, `HF_HOME=/models/.hf_home` | | Left running | yes, deliberately -- so weights stay warm for follow-up runs | -Callers only need `MODEL_ID`, `PVC_NAME`, and `GPU_COUNT` back. +Callers only need `MODEL_ID`, `PVC_NAME`, and `GPU_COUNT` back for a plain +fixed single-node recipe. -For a multi-node-capable recipe deployed with `node_mode: "multi"` (two -Pods, `vllm-attn-pod` / `vllm-ffn-pod`, both running the *same* recipe -script with opposite `ATTENTION_RANKS`/`FFN_RANKS` env overrides), the same -contract holds -- `vllm-service:${CLIENT_PORT}` still answers -OpenAI-compatible traffic, backed by the attention Pod. An additional -internal-only `vllm-ffn-p2p-service` also exists solely to carry AFD -rendezvous traffic between the two Pods; callers never talk to it -directly. +For a multi-node-capable recipe, the caller's placement plan (step 1b) +decides how many Pods exist and what each is named. The same contract still +holds -- `vllm-service:${CLIENT_PORT}` answers OpenAI-compatible traffic, +backed by whichever Pod holds attention rank 0. Whenever the plan spans more +than one Pod, an additional internal-only `vllm-ffn-p2p-service` also exists +solely to carry AFD rendezvous traffic to whichever Pod holds FFN rank 0; +callers never talk to it directly. If either role is itself split across +more than one Pod, a `vllm-attn-dp-service` and/or `vllm-ffn-dp-service` also +exist to carry that role's internal DP-RPC coordination traffic. ## Scope @@ -49,19 +51,25 @@ directly. - Recipes are hand-written shell scripts, not JSON configs -- there is no generator step and no manifest to read. - A recipe is **multi-node-capable** when it gates its `vllm serve` blocks - on `ATTENTION_RANKS`/`FFN_RANKS` env vars (e.g. - `if [ "$ATTENTION_RANKS" -gt 0 ]`) and embeds the literal string - `AFD_FFN_HOST_PLACEHOLDER` in its AFD `host` field (e.g. - `4a2f_graph_multinode.sh`) -- grep for both before assuming a script only - supports single-node. Deploying such a script with `node_mode: "multi"` - splits attention and FFN across two Pods/nodes instead of one, by running - the *same* script twice with opposite `ATTENTION_RANKS`/`FFN_RANKS` - overrides. This is **experimental**: upstream documents cross-node + on `ATTENTION_DP_RANKS`/`FFN_DP_RANKS` env vars (e.g. + `if [ "$ATTENTION_DP_RANKS" -gt 0 ]`) and reads its AFD `host` field from an + `AFD_CONNECTOR_HOST` env var (e.g. + `AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1}`, used as + `"host": "'"${AFD_CONNECTOR_HOST}"'"` inside `--additional-config`; see + e.g. the `deepseek_v2_lite/prefill_decode_colocation/*.sh` recipes) -- + grep for both before assuming a script only supports single-node. + Deploying such a script means asking the caller for an explicit + **placement plan** (step 1b): a list of Pods, each carrying its own share + of attention/FFN ranks (summing to the recipe's fixed `NUM_ATTENTION_RANKS`/ + `NUM_FFN_RANKS` totals) -- anywhere from one Pod holding every rank, to a + dedicated Pod per role, to many Pods each holding an arbitrary, possibly + mixed, share of either role. **Never assume a shape on the caller's + behalf** -- ask, every time. Placements beyond a single Pod holding + everything are **experimental**: upstream documents cross-node `P2pNcclAFDConnector` use as "not established by the current recipes ... treated as unverified" (`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`). - Deploy it when asked, but don't present its throughput as validated, and - default to `node_mode: "single"` unless the caller specifically wants - cross-node placement. + Deploy whatever the caller asks for, but don't present throughput as + validated once the plan spans more than one Pod. - Not for `tools/benchmarks/decode_bench_server.sh` (local, non-k8s). ## Requirements @@ -69,28 +77,14 @@ directly. - A live, authenticated `kubectl`/`oc` session able to create/delete Pods, Services, ConfigMaps, and PVCs in the target namespace. - `envsubst` (from `gettext`) locally. -- A benchmark image, pushed where the cluster can pull it, that satisfies - **all** of the following. There is no ready-made Dockerfile for this in - the repo -- ask the user which image to use, and confirm it meets this - spec before deploying: - - | Requirement | Why | - |---|---| - | vLLM base matching `pyproject.toml` (`vllm==0.26.0`) | the recipe's flags are version-specific | - | An `afd-plugin` install | the recipe loads the plugin | - | Repo sources on disk (conventionally `/opt/afd-plugin`) | recipes and `tools/` are read from the image | - | No BuildKit-only syntax (`COPY --link`, `RUN --mount`) *if* built with buildah/imagebuilder | OpenShift Builds rejects it | - | App dir group-writable (`chgrp -R 0 && chmod -R g=u `) | the restricted SCC runs a random UID in group 0 | - | `HOME` set to a writable path | otherwise `HOME=/` and anything expanding `~` fails | - | `PYTHONDONTWRITEBYTECODE=1` | the random UID cannot write `__pycache__` into the app dir | - - `docker/Dockerfile.ci` is the closest starting point in the repo, but it - does **not** meet this spec as written: it uses `COPY --link`, leaves - `HOME=/`, and does not make the app dir group-writable. Adjust those three - before using it on OpenShift, or supply an image that already conforms. +- An image, pushed where the cluster can pull it. Ask the user whether to + use an image they already have, or to build one from `docker/Dockerfile.ci` + and push it to a registry they provide: ```bash - IMAGE=/: # must satisfy the table above + IMAGE=/: + docker build -t "$IMAGE" -f docker/Dockerfile.ci . + docker push "$IMAGE" ``` The image supplies only the `afd-plugin` install. The recipe script is @@ -121,21 +115,17 @@ Take the recipe path from the user, or ask. Read the script and note, per host/port) **Check whether the script is multi-node-capable** by grepping for -`ATTENTION_RANKS`/`FFN_RANKS` env-var gating (e.g. -`if [ "$ATTENTION_RANKS" -gt 0 ]`) and the literal string -`AFD_FFN_HOST_PLACEHOLDER`. Both present means the script can run either as -one pod (defaults run the full topology locally) or as two pods (each -overriding one of `ATTENTION_RANKS`/`FFN_RANKS` to `0`). Neither present -means it's a plain fixed single-node script -- skip straight to step 2. -If the script is multi-node-capable, ask the caller whether they want -`node_mode: "single"` (default -- one pod, no env overrides needed) or -`node_mode: "multi"` (two pods); default to `"single"` per Scope unless -they specifically want cross-node placement. +`ATTENTION_DP_RANKS`/`FFN_DP_RANKS` env-var gating (e.g. +`if [ "$ATTENTION_DP_RANKS" -gt 0 ]`) and an `AFD_CONNECTOR_HOST` env-var +default (e.g. `AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1}`). +Neither present means it's a plain fixed single-node script -- skip straight +to step 2. Both present means the script's fixed rank totals (below) can be +placed across Pods however the caller wants -- continue to step 1b. For a multi-node-capable script, also read its hardcoded topology constants (`NUM_ATTENTION_RANKS=`, `NUM_FFN_RANKS=`, near the top -- fixed per recipe -and always sent to AFD for rendezvous regardless of which role(s) a given -pod runs locally) and its `AFD_CONNECTOR_PORT` default +and always sent to AFD for rendezvous regardless of how the caller places +them) and its `AFD_CONNECTOR_PORT` default (`AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-}`). Only export `AFD_CONNECTOR_PORT` before deploying if the caller specifically asked for a non-default rendezvous port. @@ -161,28 +151,94 @@ than published on HF Hub -- matching the script's own `MODEL_PATH` default. The serve pod handles either form transparently. **Tell the caller which form it is**; consumers that load their own tokenizer need to know. +### 1b. Ask for a placement plan (multi-node-capable scripts only) + +Skip this step entirely for a plain fixed single-node script (step 1 already +sent it straight to step 2). + +Ask the caller how they want the recipe's fixed `NUM_ATTENTION_RANKS` +attention ranks and `NUM_FFN_RANKS` FFN ranks placed across Pods. **Do not +default to a particular shape** -- these are all equally valid answers to +the same question: + +- one Pod holding every rank (`attn_ranks=NUM_ATTENTION_RANKS, + ffn_ranks=NUM_FFN_RANKS` on a single Pod) -- the simplest shape, and the + only one that isn't experimental; +- a dedicated Pod per role (one Pod with all attention ranks, another with + all FFN ranks); +- either role split across several Pods (e.g. attention's ranks spread + 2-and-2 across two Pods, FFN whole in a third); +- Pods that mix a partial share of both roles (e.g. two Pods of `2 attention + + 1 ffn` each for a 4-attention/2-ffn recipe). + +Record the plan as a list of `(pod_name, attn_ranks, ffn_ranks)` tuples, in +the order the caller wants them evaluated. Validate before proceeding: + +- every `pod_name` is unique; +- `sum(attn_ranks) == NUM_ATTENTION_RANKS` and `sum(ffn_ranks) == + NUM_FFN_RANKS` across the whole plan -- AFD's rendezvous always expects + the recipe's fixed totals regardless of how they're distributed, so an + under- or over-count silently breaks rendezvous rather than failing + loudly; +- every Pod has `attn_ranks + ffn_ranks >= 1` (no empty Pods). + +**Derive per-role rank ordering**, needed for `_START_RANK`/`_HEADLESS`/ +`_DP_ADDRESS` in step 4-placed: within each role, walk the plan in the given +order and set that Pod's `start_rank` for the role to the running sum of the +role's local counts from every earlier Pod carrying it. The first Pod +carrying a nonzero share of a role is that role's **head** +(`start_rank=0`, `headless=0` -- it runs attention's API server, or binds +FFN's rendezvous rank 0); every later Pod carrying that role is a **worker** +(`headless=1`). A role that is whole in one Pod trivially has only a head, +no workers. + +Only a role that is genuinely split across more than one Pod needs +`_START_RANK`/`_HEADLESS`/`_DP_ADDRESS` to vary at all -- and only if the +recipe exposes `ATTENTION_DP_START_RANK`/`FFN_DP_START_RANK`, +`ATTENTION_HEADLESS`/`FFN_HEADLESS`, `ATTENTION_DP_ADDRESS`/`FFN_DP_ADDRESS` +env vars (grep for `_START_RANK`/`_HEADLESS`; a script only gated on +`ATTENTION_DP_RANKS`/`FFN_DP_RANKS`/`AFD_CONNECTOR_HOST` can place each role +freely relative to the other, but can't split a single role across more +than one Pod). Flag this to the caller before proposing a plan that would +require it. + +**Resolve `AFD_CONNECTOR_HOST`** from the plan's Pod count, not from a fixed +mode: exactly one Pod in the plan means the recipe's loopback default +(`127.0.0.1`) already works -- no Service needed, don't override it. More +than one Pod means every Pod's container env must set `AFD_CONNECTOR_HOST` +to `vllm-ffn-p2p-service` (created in step 4-placed), including Pods that +carry no FFN ranks themselves -- per +`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`, every participating rank must +agree on the same host string, and the Pod holding FFN rank 0 both binds and +is reached through that name. + +**Node placement is left to the scheduler by default.** Do not add +`podAntiAffinity` forcing Pods onto separate nodes unless the caller +explicitly asks to exercise cross-node placement -- an unforced multi-Pod +plan may legitimately land every Pod on the same node, and that's fine. +Only add the shared-label `podAntiAffinity` block in step 4-placed if asked. + ### 2. Compute GPU_COUNT -For a plain single-node script: `GPU_COUNT` = number of distinct GPU -indices across every literal `CUDA_VISIBLE_DEVICES=` in the script (union, -not sum -- each index appears in exactly one block). +For a plain single-node script (step 1 found no `ATTENTION_DP_RANKS`/ +`FFN_DP_RANKS` gating): `GPU_COUNT` = number of distinct GPU indices across +every literal `CUDA_VISIBLE_DEVICES=` in the script (union, not sum -- each +index appears in exactly one block). -For a multi-node-capable script (step 1 found `ATTENTION_RANKS`/ -`FFN_RANKS` gating), `CUDA_VISIBLE_DEVICES` is computed at runtime rather -than literal, so derive `GPU_COUNT` from the constants read in step 1 -instead: -- `node_mode: "single"`: `GPU_COUNT = NUM_ATTENTION_RANKS + NUM_FFN_RANKS` - (both roles run on the one pod, sharing a contiguous device pool). -- `node_mode: "multi"`: two counts instead of one -- - `GPU_COUNT_ATTN = NUM_ATTENTION_RANKS`, `GPU_COUNT_FFN = NUM_FFN_RANKS`. +For a multi-node-capable script, `GPU_COUNT` is per-Pod, read straight off +the placement plan from step 1b: for each Pod, `GPU_COUNT = attn_ranks + +ffn_ranks` assigned to it there. ### 3. Confirm before touching the cluster State and confirm: target cluster/namespace, `AFD_PLUGIN_IMAGE`, `MODEL_ID`, -`PVC_NAME`, `GPU_COUNT`, `CLIENT_PORT`, `RECIPE_SCRIPT_PATH`. Flag explicitly -that step 4c **deletes any existing `vllm-pod`** (or -`vllm-attn-pod`/`vllm-ffn-pod` in the multi-node variant) -- if one exists -from unrelated work, that is destructive and needs approval first. +`PVC_NAME`, `CLIENT_PORT`, `RECIPE_SCRIPT_PATH`, and either `GPU_COUNT` +(plain single-node script) or the full placement plan as a table of +`pod_name | attn_ranks | ffn_ranks | GPU_COUNT` (multi-node-capable script). +Flag explicitly that step 4 **deletes any existing Pod(s) with the same +name(s)** as what's about to be deployed (`vllm-pod`, or every `pod_name` in +the placement plan) -- if one exists from unrelated work, that is +destructive and needs approval first. `RECIPE_SCRIPT_PATH` is a *local* path. It need not be committed, exist in the image, or live at any particular depth -- but per Scope it must be a @@ -192,21 +248,36 @@ colocation recipe, since nothing rewrites `SCRIPT_DIR`-relative lookups. AFD_PLUGIN_IMAGE= MODEL_ID= PVC_NAME= -GPU_COUNT= CLIENT_PORT= # from the script's --port; default 18305 RECIPE_SCRIPT_PATH= VLLM_USE_V2_MODEL_RUNNER=${VLLM_USE_V2_MODEL_RUNNER:-0} # pulled from the caller's env, default 0 ``` -When step 1 identified a `node_mode: "multi"` deploy, confirm two GPU -counts instead of one -- `RECIPE_SCRIPT_PATH` stays a single path, since the -same script deploys to both pods with opposite `ATTENTION_RANKS`/ -`FFN_RANKS`: +For a plain single-node script: + +```bash +GPU_COUNT= +``` + +For a multi-node-capable script, confirm the full placement plan instead, +e.g. for a 4-attention/2-ffn recipe split as two attention Pods of 2 ranks +each plus a dedicated FFN Pod: ```bash -GPU_COUNT_ATTN= -GPU_COUNT_FFN= +# pod_name attn_ranks ffn_ranks gpu_count +# vllm-attn-pod-0 2 0 2 +# vllm-attn-pod-1 2 0 2 +# vllm-ffn-pod 0 2 2 + AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-} # only export if overriding +AFD_CONNECTOR_HOST=vllm-ffn-p2p-service # only when the plan has >1 pod (step 1b); leave unset/loopback for a single-pod plan +``` + +or, for the same recipe placed as one Pod holding every rank: + +```bash +# pod_name attn_ranks ffn_ranks gpu_count +# vllm-pod-0 4 2 6 ``` ### 4. Deploy @@ -221,35 +292,36 @@ the GPUs are already claimed. Set `STORAGE_CLASS` to a class the cluster actually offers; leaving it unset falls to the default, which is often RWO-only or absent and leaves the pod `Pending`. -For `node_mode: "multi"`, the PVC **must** be `ReadWriteMany` (RWX), not -`ReadWriteOnce`. The attention and FFN Pods are forced onto different nodes -by anti-affinity (step 4c-multi), and an RWO volume can only attach on one -node at a time -- the second Pod fails at admission with `Multi-Attach -error for volume ... already used by pod(s) ...` and sits `Pending` -forever. This is a hard deadlock, not something that resolves by waiting -longer. Before deploying multi-node, confirm the storage class actually -supports RWX (a throwaway 1Gi PVC requesting `ReadWriteMany` against the -same `STORAGE_CLASS` either binds or it doesn't -- delete it after). If an -existing single-node PVC is RWO and already holds downloaded weights you -want to reuse for a multi-node deploy, provision a new RWX PVC and copy the -data across rather than re-downloading -- e.g. a short-lived Pod pinned via -`nodeName` to whichever node already holds the RWO mount (same-node -multi-mount of an RWO volume is fine), with both PVCs as volumes, running -`cp -a /old/. /new/.` (a nonzero exit purely from failing to preserve the -top-level directory's mtime is cosmetic; verify with `du -sh` on both sides -instead of trusting the exit code). +Whenever the placement plan (step 1b) has more than one Pod, the PVC +**must** be `ReadWriteMany` (RWX), not `ReadWriteOnce` -- even without +forced anti-affinity (step 1b defaults to leaving node placement to the +scheduler), nothing guarantees the Pods land on the same node, and an RWO +volume can only attach on one node at a time. The second Pod to need it +fails at admission with `Multi-Attach error for volume ... already used by +pod(s) ...` and sits `Pending` forever. This is a hard deadlock, not +something that resolves by waiting longer. Before deploying a multi-Pod +plan, confirm the storage class actually supports RWX (a throwaway 1Gi PVC +requesting `ReadWriteMany` against the same `STORAGE_CLASS` either binds or +it doesn't -- delete it after). If an existing single-Pod PVC is RWO and +already holds downloaded weights you want to reuse for a multi-Pod plan, +provision a new RWX PVC and copy the data across rather than re-downloading +-- e.g. a short-lived Pod pinned via `nodeName` to whichever node already +holds the RWO mount (same-node multi-mount of an RWO volume is fine), with +both PVCs as volumes, running `cp -a /old/. /new/.` (a nonzero exit purely +from failing to preserve the top-level directory's mtime is cosmetic; +verify with `du -sh` on both sides instead of trusting the exit code). ```bash MODEL_PVC_SIZE=${MODEL_PVC_SIZE:-100Gi} STORAGE_CLASS=${STORAGE_CLASS:-} # e.g. ocs-storagecluster-cephfs -PVC_ACCESS_MODE=${PVC_ACCESS_MODE:-ReadWriteOnce} # ReadWriteMany for node_mode "multi" +PVC_ACCESS_MODE=${PVC_ACCESS_MODE:-ReadWriteOnce} # ReadWriteMany whenever the placement plan has more than one Pod if kubectl get pvc "${PVC_NAME}" >/dev/null 2>&1; then have="$(kubectl get pvc "${PVC_NAME}" -o jsonpath='{.spec.resources.requests.storage}')" have_modes="$(kubectl get pvc "${PVC_NAME}" -o jsonpath='{.spec.accessModes}')" echo "PVC ${PVC_NAME} exists (${have}, ${have_modes}); serving ${MODEL_ID} from its warm HF_HOME cache" echo "NOTE: verify ${have} fits ${MODEL_ID} -- an undersized reused PVC fails mid-download" - echo "NOTE: for node_mode multi, ${have_modes} must include ReadWriteMany or the FFN Pod will deadlock on attach" + echo "NOTE: for a multi-Pod placement plan, ${have_modes} must include ReadWriteMany or the second Pod will deadlock on attach" else envsubst '${PVC_NAME} ${MODEL_PVC_SIZE} ${STORAGE_CLASS} ${PVC_ACCESS_MODE}' <<'EOF' | \ grep -v 'storageClassName: *$' | kubectl apply -f - @@ -271,8 +343,10 @@ fi ``` **4b. Recipe ConfigMap.** This is what lets the pod run any local recipe -- -edited, uncommitted, or brand new -- without rebuilding the image. Run from -wherever `RECIPE_SCRIPT_PATH` resolves (e.g. the repo root): +edited, uncommitted, or brand new -- without rebuilding the image. Shared as- +is regardless of how many Pods the placement plan has (it's the *same* +script mounted into every Pod). Run from wherever `RECIPE_SCRIPT_PATH` +resolves (e.g. the repo root): ```bash kubectl create configmap afd-recipe-script \ @@ -280,12 +354,13 @@ kubectl create configmap afd-recipe-script \ --dry-run=client -o yaml | kubectl apply -f - ``` -**4c. Serve pod.** The container runs the mounted recipe unmodified except -for rebinding the client-facing port off loopback: the recipe binds -`:${CLIENT_PORT}` on `127.0.0.1` (it is normally driven from inside the same -host), but the Service needs `0.0.0.0`. Every internal worker port stays on -loopback. The `uv` shim on `PATH` strips `uv run` -- the image's base -environment is already correct. +**4c. Serve pod** (plain fixed single-node script only -- for a multi-node- +capable script, skip to **4-placed** below instead). The container runs the +mounted recipe unmodified except for rebinding the client-facing port off +loopback: the recipe binds `:${CLIENT_PORT}` on `127.0.0.1` (it is normally +driven from inside the same host), but the Service needs `0.0.0.0`. Every +internal worker port stays on loopback. The `uv` shim on `PATH` strips +`uv run` -- the image's base environment is already correct. `fsGroup` must be inside the namespace's allowed range or the pod is rejected at admission (`fsGroup: Invalid value: ... is not an allowed @@ -541,72 +616,75 @@ done echo "=== serve pod ready at http://vllm-service:${CLIENT_PORT} ===" ``` -### 4-multi. Multi-node variant +### 4-placed. Placement-plan-driven variant -Only applies when step 1 identified a multi-node-capable script and the -caller wants `node_mode: "multi"`. Step 4a (the model PVC) is shared -- both -Pods mount the same PVC -- but per step 4a's note it must be provisioned -`ReadWriteMany`, not the single-node default `ReadWriteOnce`, or the FFN -Pod deadlocks trying to attach on its own node. Step 4b (the recipe -ConfigMap) is also shared as-is: it's the *same* script for both pods now, -so there is only one ConfigMap, not one per role. Everything below -replaces steps 4c-4f. +Only applies when step 1 identified a multi-node-capable script -- use the +placement plan and per-role rank ordering worked out in step 1b. Step 4a +(the model PVC) is shared -- every Pod in the plan mounts the same PVC -- +but per step 4a's note it must be provisioned `ReadWriteMany` whenever the +plan has more than one Pod. Step 4b (the recipe ConfigMap) is also shared +as-is: the same script is mounted into every Pod regardless of how many the +plan has, since it already reads its rank counts and AFD host from env +vars -- no mutation needed. Everything below replaces steps 4c-4f. -**Resolve the FFN host placeholder.** The recipe script contains the -literal string `AFD_FFN_HOST_PLACEHOLDER` wherever `host` appears inside -`--additional-config`. Per `docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`, -`host` must resolve to FFN's first rank, and every participating rank -(including attention) must agree on the same value. Since both pods mount -the *same* ConfigMap, substitute once, before creating it: +**Look up `fsGroup`**, same as step 4c: ```bash -FFN_HOST="vllm-ffn-p2p-service" # short form resolves within the namespace -sed -i "s/AFD_FFN_HOST_PLACEHOLDER/${FFN_HOST}/g" "$RECIPE_SCRIPT_PATH" +FS_GROUP="$(kubectl get ns "$(kubectl config view --minify -o jsonpath='{..namespace}')" \ + -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' 2>/dev/null \ + | cut -d/ -f1)" +FS_GROUP="${FS_GROUP:-1000}" +echo "using fsGroup=${FS_GROUP}" ``` -**Recipe ConfigMap (shared, same as step 4b):** +**Resolve the FFN host value** (skip if the plan has only one Pod -- step 1b +already said to leave `AFD_CONNECTOR_HOST` at its loopback default). Per +`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`, `host` must resolve to FFN's +first rank, and every participating rank (including attention) must agree +on the same value: ```bash -kubectl create configmap afd-recipe-script \ - --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \ - --dry-run=client -o yaml | kubectl apply -f - +AFD_CONNECTOR_HOST="vllm-ffn-p2p-service" # short form resolves within the namespace; short-circuit to 127.0.0.1 for a single-Pod plan ``` -**4c-multi. Two Pods, forced onto different nodes.** Same container script -body, env, and `uv` shim as step 4c, both mounting the one shared -ConfigMap. Differences from step 4c: each Pod passes `ATTENTION_RANKS`/ -`FFN_RANKS` env vars so the script's own gating -(`if [ "$ATTENTION_RANKS" -gt 0 ]` / `if [ "$FFN_RANKS" -gt 0 ]`) runs only -its role locally -- the attention Pod sets `ATTENTION_RANKS=$GPU_COUNT_ATTN -FFN_RANKS=0`, the FFN Pod sets `ATTENTION_RANKS=0 FFN_RANKS=$GPU_COUNT_FFN`; -each Pod requests only its own role's GPU count (`GPU_COUNT_ATTN` / -`GPU_COUNT_FFN`, not the union); and the two Pods share an -`afd-multinode-group` label so a `podAntiAffinity` can force them onto -different nodes -- matching GPU requests alone don't guarantee that on a -cluster with multiple GPU nodes. Only the attention Pod's startup wait hits -`/health`; the FFN Pod only waits on its own log marker (`ffn.log` -> `AFD -FFN EngineCore started`) and never opens an HTTP port, so a `/health` check -there would hang forever. - -Because the shared script always contains both `> attn.log` and -`> ffn.log` redirects (one inside each role's `if` block), grepping the -script text for log redirects -- as step 4c does for plain scripts -- -would wrongly expect a log that this pod's disabled role never produces. -Derive the expected log(s) instead from this pod's own -`ATTENTION_RANKS`/`FFN_RANKS` values: +This is passed as the `AFD_CONNECTOR_HOST` container env var to *every* Pod +in the plan below, including Pods carrying no FFN ranks themselves -- no +script mutation needed, since the recipe already reads `host` from that env +var. -```bash -kubectl delete pod vllm-attn-pod vllm-ffn-pod --ignore-not-found +**One Pod per plan entry, each requesting only its own share of GPUs.** Same +container script body, env, and `uv` shim as step 4c, but driven by two +independent role positions (`attn_*` / `ffn_*`) per Pod instead of one +combined role, since a single Pod can now carry a share of either role, +both, or (in a one-Pod-holds-everything plan) all of both: -deploy_role_pod() { - local role="$1" pod_name="$2" attn_ranks="$3" ffn_ranks="$4" +```bash +# delete every Pod name that appears in the placement plan before redeploying +kubectl delete pod ... --ignore-not-found + +# attn_start_rank/attn_headless/attn_dp_address and their ffn_ counterparts +# describe this Pod's position within each role's DP group -- defaults +# (0, 0, 127.0.0.1) mean "this Pod is the whole role" or "this Pod is that +# role's head". Only a role actually split across more than one Pod (step +# 1b) needs non-default values for that role. +deploy_pod() { + local pod_name="$1" attn_ranks="$2" ffn_ranks="$3" \ + attn_start_rank="${4:-0}" attn_headless="${5:-0}" attn_dp_address="${6:-127.0.0.1}" \ + ffn_start_rank="${7:-0}" ffn_headless="${8:-0}" ffn_dp_address="${9:-127.0.0.1}" local gpu_n=$((attn_ranks + ffn_ranks)) + local attn_node_role="none" ffn_node_role="none" + [ "$attn_ranks" -gt 0 ] && attn_node_role=$([ "$attn_headless" = "1" ] && echo worker || echo head) + [ "$ffn_ranks" -gt 0 ] && ffn_node_role=$([ "$ffn_headless" = "1" ] && echo worker || echo head) + TEMPLATE_IMAGE="$AFD_PLUGIN_IMAGE" TEMPLATE_MODEL="$MODEL_ID" TEMPLATE_POD="$pod_name" \ - TEMPLATE_ROLE="$role" TEMPLATE_ATTN_RANKS="$attn_ranks" TEMPLATE_FFN_RANKS="$ffn_ranks" \ - TEMPLATE_GPU="$gpu_n" TEMPLATE_GROUP="afd-multinode" TEMPLATE_PVC="$PVC_NAME" \ + TEMPLATE_ATTN_RANKS="$attn_ranks" TEMPLATE_FFN_RANKS="$ffn_ranks" \ + TEMPLATE_ATTN_START_RANK="$attn_start_rank" TEMPLATE_ATTN_HEADLESS="$attn_headless" TEMPLATE_ATTN_DP_ADDRESS="$attn_dp_address" \ + TEMPLATE_FFN_START_RANK="$ffn_start_rank" TEMPLATE_FFN_HEADLESS="$ffn_headless" TEMPLATE_FFN_DP_ADDRESS="$ffn_dp_address" \ + TEMPLATE_ATTN_NODE_ROLE="$attn_node_role" TEMPLATE_FFN_NODE_ROLE="$ffn_node_role" \ + TEMPLATE_GPU="$gpu_n" TEMPLATE_PVC="$PVC_NAME" \ TEMPLATE_FSGROUP="$FS_GROUP" TEMPLATE_CLIENT_PORT="$CLIENT_PORT" \ - TEMPLATE_AFD_PORT="${AFD_CONNECTOR_PORT:-}" \ - envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_ROLE} ${TEMPLATE_ATTN_RANKS} ${TEMPLATE_FFN_RANKS} ${TEMPLATE_GPU} ${TEMPLATE_GROUP} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_AFD_PORT} ${VLLM_USE_V2_MODEL_RUNNER}' <<'EOF' | kubectl apply -f - + TEMPLATE_AFD_PORT="${AFD_CONNECTOR_PORT:-}" TEMPLATE_AFD_HOST="${AFD_CONNECTOR_HOST:-127.0.0.1}" \ + envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_ATTN_RANKS} ${TEMPLATE_FFN_RANKS} ${TEMPLATE_ATTN_START_RANK} ${TEMPLATE_ATTN_HEADLESS} ${TEMPLATE_ATTN_DP_ADDRESS} ${TEMPLATE_FFN_START_RANK} ${TEMPLATE_FFN_HEADLESS} ${TEMPLATE_FFN_DP_ADDRESS} ${TEMPLATE_ATTN_NODE_ROLE} ${TEMPLATE_FFN_NODE_ROLE} ${TEMPLATE_GPU} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_AFD_PORT} ${TEMPLATE_AFD_HOST} ${VLLM_USE_V2_MODEL_RUNNER}' <<'EOF' | kubectl apply -f - apiVersion: v1 kind: Pod metadata: @@ -614,19 +692,12 @@ metadata: labels: app: afd-recipe role: serve - afd-role: ${TEMPLATE_ROLE} - afd-multinode-group: ${TEMPLATE_GROUP} + afd-attn-node-role: ${TEMPLATE_ATTN_NODE_ROLE} + afd-ffn-node-role: ${TEMPLATE_FFN_NODE_ROLE} spec: restartPolicy: Never securityContext: fsGroup: ${TEMPLATE_FSGROUP} - affinity: - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - {key: afd-multinode-group, operator: In, values: ["${TEMPLATE_GROUP}"]} - topologyKey: kubernetes.io/hostname volumes: - name: model-storage persistentVolumeClaim: @@ -654,7 +725,7 @@ spec: chmod +x /work/bin/uv export PATH="/work/bin:$PATH" - echo "=== launching $(basename "$RECIPE_SCRIPT") recipe (role: ${TEMPLATE_ROLE}) ===" + echo "=== launching $(basename "$RECIPE_SCRIPT") recipe (attn_ranks=${TEMPLATE_ATTN_RANKS} ffn_ranks=${TEMPLATE_FFN_RANKS}) ===" RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" @@ -687,12 +758,24 @@ spec: sleep infinity } - [ -n "$LOGS" ] || fail "this pod's ATTENTION_RANKS/FFN_RANKS are both 0 -- nothing to run" + [ -n "$LOGS" ] || fail "this pod's attn_ranks/ffn_ranks are both 0 -- nothing to run" + # A headless pod (worker shard of a role split across multiple + # Pods, per --headless) never starts an API server, so it never + # logs "Application startup complete" -- fall back to the + # EngineCore's own init-complete line, which fires regardless of + # headless status (unverified marker; not live-validated). ready_marker() { case "$1" in ffn.log) echo "AFD FFN EngineCore started" ;; - *) echo "Application startup complete" ;; + attn.log) + if [ "${TEMPLATE_ATTN_HEADLESS}" = "1" ]; then + echo "init engine (profile, create kv cache, warmup model) took" + else + echo "Application startup complete" + fi + ;; + *) echo "Application startup complete" ;; esac } for log in $LOGS; do @@ -709,7 +792,7 @@ spec: [ "$ready" = "1" ] || fail "timed out waiting for $log" done - if [ "${TEMPLATE_ATTN_RANKS}" -gt 0 ]; then + if [ "${TEMPLATE_ATTN_RANKS}" -gt 0 ] && [ "${TEMPLATE_ATTN_HEADLESS}" != "1" ]; then echo "--- waiting on attention server /health (127.0.0.1:${TEMPLATE_CLIENT_PORT}) ---" endpoint_ready=0 for i in $(seq 1 60); do @@ -719,7 +802,7 @@ spec: [ "$endpoint_ready" = "1" ] || fail "attention server did not answer /health" fi - echo "=== ${TEMPLATE_ROLE} role READY ===" + echo "=== pod ${TEMPLATE_POD} READY (attn_ranks=${TEMPLATE_ATTN_RANKS} ffn_ranks=${TEMPLATE_FFN_RANKS}) ===" sleep infinity env: - {name: USER, value: "vllm"} @@ -733,9 +816,16 @@ spec: - {name: UV_CACHE_DIR, value: "/work/uv"} - {name: VLLM_LOGGING_LEVEL, value: "INFO"} - {name: VLLM_USE_V2_MODEL_RUNNER, value: "${VLLM_USE_V2_MODEL_RUNNER}"} - - {name: ATTENTION_RANKS, value: "${TEMPLATE_ATTN_RANKS}"} - - {name: FFN_RANKS, value: "${TEMPLATE_FFN_RANKS}"} + - {name: ATTENTION_DP_RANKS, value: "${TEMPLATE_ATTN_RANKS}"} + - {name: FFN_DP_RANKS, value: "${TEMPLATE_FFN_RANKS}"} + - {name: ATTENTION_DP_START_RANK, value: "${TEMPLATE_ATTN_START_RANK}"} + - {name: FFN_DP_START_RANK, value: "${TEMPLATE_FFN_START_RANK}"} + - {name: ATTENTION_HEADLESS, value: "${TEMPLATE_ATTN_HEADLESS}"} + - {name: FFN_HEADLESS, value: "${TEMPLATE_FFN_HEADLESS}"} + - {name: ATTENTION_DP_ADDRESS, value: "${TEMPLATE_ATTN_DP_ADDRESS}"} + - {name: FFN_DP_ADDRESS, value: "${TEMPLATE_FFN_DP_ADDRESS}"} - {name: AFD_CONNECTOR_PORT, value: "${TEMPLATE_AFD_PORT}"} + - {name: AFD_CONNECTOR_HOST, value: "${TEMPLATE_AFD_HOST}"} - name: HF_TOKEN valueFrom: secretKeyRef: {name: hf-token-secret, key: token} @@ -749,19 +839,108 @@ spec: - {name: recipe-script, mountPath: /recipe, readOnly: true} EOF } +``` + +Call `deploy_pod` once per entry in the placement plan. One Pod holding +every rank of a 4-attention/2-ffn recipe: + +```bash +deploy_pod vllm-pod-0 4 2 0 0 127.0.0.1 0 0 127.0.0.1 +``` -deploy_role_pod attention vllm-attn-pod "$GPU_COUNT_ATTN" 0 -deploy_role_pod ffn vllm-ffn-pod 0 "$GPU_COUNT_FFN" +A dedicated Pod per role, attention split 2-and-2 across two Pods, FFN whole +in a third: + +```bash +deploy_pod vllm-attn-pod-0 2 0 0 0 vllm-attn-dp-service 0 0 127.0.0.1 +deploy_pod vllm-attn-pod-1 2 0 2 1 vllm-attn-dp-service 0 0 127.0.0.1 +deploy_pod vllm-ffn-pod 0 2 0 0 127.0.0.1 0 0 127.0.0.1 +``` + +`vllm-attn-pod-1`'s `attn_start_rank=2` because `vllm-attn-pod-0` already +claimed local ranks `[0, 2)`; `vllm-attn-dp-service` is the DP-coordination +Service (below) exposing `vllm-attn-pod-0`'s DP-RPC port. + +`vllm-attn-pod-0`'s own `_DP_ADDRESS` matters here, and it must be the +**same headless Service name** the worker connects through (`vllm-attn-dp-service`), +not `127.0.0.1` and not `0.0.0.0`. Two separate mechanisms consume this one +value on the head, with conflicting requirements: + +- vLLM's `DPCoordinator` uses `--data-parallel-address` as the literal ZMQ + **bind** address. `127.0.0.1` binds loopback-only and the worker's + connection hangs until it times out -- this needs `0.0.0.0` or a real, + non-loopback address. +- Separately, when the recipe passes `--enable-expert-parallel`, every + rank -- including the worker's, on the other Pod -- calls + `ParallelConfig.stateless_init_dp_group()` (`vllm/config/parallel.py`), + which reuses this same address as the **connect target** for a second, + independent rendezvous. `0.0.0.0` satisfies the bind case but is not a + dialable address, so the worker's ranks hang trying to connect to it and + the whole DP group fails after that rendezvous's timeout. + +Because a headless Service with a selector matching exactly one Pod +resolves to that Pod's real IP from anywhere in the cluster -- including +from the Pod itself -- pointing the head's own `_DP_ADDRESS` at +`vllm-attn-dp-service` satisfies both: it binds to the head's real +interface (not loopback) and is dialable by the worker (not the wildcard). +This only applies to a head Pod whose role is split (`_headless=0` but +another Pod shares the same role) -- a Pod that is whole in one Pod (like +`vllm-ffn-pod` above, or the single-Pod example before it) keeps +`127.0.0.1` since nothing external ever needs to reach it, and there's no +second Pod for `stateless_init_dp_group` to hang against. Two Pods mixing a +partial share of both roles (e.g. `2 attention + 1 ffn` each, for a +4-attention/2-ffn recipe): + +```bash +deploy_pod vllm-mixed-pod-0 2 1 0 0 vllm-attn-dp-service 0 0 vllm-ffn-dp-service +deploy_pod vllm-mixed-pod-1 2 1 2 1 vllm-attn-dp-service 1 1 vllm-ffn-dp-service ``` +Both of `vllm-mixed-pod-0`'s `_DP_ADDRESS` values point at the matching +headless Service (its own head address for each split role) rather than +`0.0.0.0` -- same bind-and-connect reasoning as above, applied to each role +independently. + +Splitting a role across Pods this way additionally requires the recipe to +expose the `_START_RANK`/`_HEADLESS`/`_DP_ADDRESS` env vars noted in step 1b +-- confirm that before proposing such a plan. + `AFD_CONNECTOR_PORT` above defaults to an empty string when the caller didn't override it -- an empty env value still falls through to the -script's own `${AFD_CONNECTOR_PORT:-}` default, so it's safe to -always pass the container env entry. +script's own `${AFD_CONNECTOR_PORT:-}` default, so it's safe to always +pass the container env entry. -**4d-multi. Two Services.** `vllm-service` keeps its existing shape and -contract (client-facing, port `${CLIENT_PORT}`) but now selects `afd-role: attention` -instead of the generic `role: serve`: +**Forced cross-node placement, only if the caller explicitly asked for it** +(step 1b defaults to leaving node placement to the scheduler). Add this +label to every Pod's `metadata.labels` in the manifest above, and this +`affinity` block to every Pod's `spec`: + +```yaml + labels: + afd-multinode-group: afd-multinode + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - {key: afd-multinode-group, operator: In, values: ["afd-multinode"]} + topologyKey: kubernetes.io/hostname +``` + +This is the same experimental caveat already noted in Scope for cross-Pod +`P2pNcclAFDConnector` use: any plan beyond a single Pod deploys correctly by +construction (correct AFD/DP flags, correct port exposure per +`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`), but has not been validated +end-to-end on a live cluster in this repo. Treat it as exploratory and say +so when reporting results. + +**Services.** `vllm-service` keeps its existing shape and contract (client- +facing, port `${CLIENT_PORT}`) but selects on `afd-attn-node-role: head` +instead of the generic `role: serve` -- this resolves to whichever Pod holds +attention rank 0, whether that Pod is dedicated to attention or also carries +FFN ranks (including the single-Pod-holds-everything plan, where the one +Pod is trivially both roles' head): ```bash kubectl apply -f - < ... -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}' ``` -Both matter now: either Pod may have been the one to trigger the PVC's -first `ReadWriteOnce` bind, and a follow-up pod can only attach from that -Pod's node. +Every node matters now: whichever Pod triggered the PVC's first +`ReadWriteMany`/`ReadWriteOnce` bind constrains where a follow-up pod can +attach from. ### 6. Teardown @@ -853,16 +1060,21 @@ The pod(s) and Service(s) are **left running by design**, so weights stay warm. Report what is still up and how to remove it; do not delete unless asked: +For a plain single-node script: + ```bash kubectl delete pod vllm-pod kubectl delete svc vllm-service ``` -For the multi-node variant, list all four resources instead: +For a multi-node-capable script deployed via a placement plan, list every +Pod name from the plan plus whichever Services were actually created +(`vllm-service` always; `vllm-ffn-p2p-service` if the plan had more than one +Pod; `vllm-attn-dp-service`/`vllm-ffn-dp-service` if that role was split): ```bash -kubectl delete pod vllm-attn-pod vllm-ffn-pod -kubectl delete svc vllm-service vllm-ffn-p2p-service +kubectl delete pod ... +kubectl delete svc vllm-service vllm-ffn-p2p-service vllm-attn-dp-service vllm-ffn-dp-service ``` The model PVC is intentionally not listed -- deleting it discards the warm @@ -870,22 +1082,41 @@ cache and forces a full re-download. ## Caveats -- Redeploying a different recipe means re-running steps 4b-4f (or their - multi-node equivalents); the pod(s) are replaced, not reconfigured. Two - recipes cannot share a pod or its GPUs. +- Redeploying a different recipe means re-running steps 4b-4f (or the + placement-plan-driven equivalent); the pod(s) are replaced, not + reconfigured. Two recipes cannot share a pod or its GPUs. - Startup markers are keyed on log filename (`ffn.log` -> `AFD FFN - EngineCore started`, everything else -> `Application startup complete`). A - recipe using different log names gets the generic marker. -- Multi-node placement requires a recipe written in the parameterized - style (`ATTENTION_RANKS`/`FFN_RANKS` gating + `AFD_FFN_HOST_PLACEHOLDER`, - e.g. `4a2f_graph_multinode.sh`). Older fixed two-block recipes (both - roles backgrounded unconditionally, literal `CUDA_VISIBLE_DEVICES`) only - support `node_mode: "single"` -- don't attempt to split one of those - across two Pods. -- The multi-node variant is experimental: it deploys correctly (two Pods - forced onto different nodes, correct port exposure per - `docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`), but upstream has not - validated `P2pNcclAFDConnector` correctness or performance across nodes. - NCCL runs over whatever the cluster's pod network provides -- no RDMA is - assumed. Treat throughput numbers from a multi-node deployment as + EngineCore started`, everything else -> `Application startup complete`, + or -- for a headless attention Pod sharding a split role -- `init engine + (profile, create kv cache, warmup model) took`). A recipe using different + log names gets the generic marker. +- A placement plan requires a recipe written in the parameterized style + (`ATTENTION_DP_RANKS`/`FFN_DP_RANKS` gating + an `AFD_CONNECTOR_HOST` + env-var default for the AFD `host` field, e.g. the + `deepseek_v2_lite/prefill_decode_colocation/*.sh` recipes). Older fixed + two-block recipes (both roles backgrounded unconditionally, literal + `CUDA_VISIBLE_DEVICES`, hardcoded `host`) aren't multi-node-capable at all + -- they only support the plain step 4c/4d/4e/4f path, one Pod, no + placement question. Don't attempt to place one of those across more than + one Pod. +- Any placement plan beyond a single Pod holding every rank is + **experimental**: it deploys correctly (correct port exposure per + `docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`, correct AFD/DP flags per + Pod), but upstream has not validated `P2pNcclAFDConnector` correctness or + performance across Pods -- whether or not those Pods land on the same + node. NCCL runs over whatever the cluster's pod network provides -- no + RDMA is assumed. Treat throughput numbers from any multi-Pod deployment as exploratory, and say so when reporting results. +- Splitting a single role across more than one Pod additionally requires + the recipe to expose `_START_RANK`/`_HEADLESS`/`_DP_ADDRESS` env vars + (grep for `_START_RANK`/`_HEADLESS`; a script only gated on + `ATTENTION_DP_RANKS`/`FFN_DP_RANKS`/`AFD_CONNECTOR_HOST` supports placing + each role in its own Pod, or colocating both roles in one Pod, but not + finer sharding of a single role). This is source-backed against vLLM's + native multi-node DP flags (`--data-parallel-size-local`, + `--data-parallel-start-rank`, `--data-parallel-address`, + `--data-parallel-rpc-port`, `--headless` -- confirmed present in + `vllm/entrypoints/openai/cli_args.py` and `vllm/engine/arg_utils.py`) but + has not been live-validated on a cluster -- in particular the + headless-attention readiness marker above is an assumption, not a + confirmed observation. Flag this explicitly if asked to deploy it. diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh index ba5db48c2..85e061bf6 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh @@ -1,52 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & +# Fixed topology for this recipe -- always sent to AFD for rendezvous, +# regardless of which role(s) this node launches locally. +ATTENTION_DP_SIZE=1 +ATTENTION_TP_SIZE=2 +FFN_DP_SIZE=1 +FFN_TP_SIZE=2 +NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) +NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) + +# How many DP ranks of each role to launch on THIS node. 0 skips that role. +# Default runs the full topology on one node (single-node colocation); +# a multi-node deploy overrides one of these to 0 per pod. +ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} +FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} + +# DP sharding: lets a single role's DP group be split across multiple pods. +# Defaults reproduce today's single-pod-per-role behavior exactly (local +# rank count == full DP size, start rank 0, loopback DP-RPC address). +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +FFN_HEADLESS=${FFN_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} +FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} +FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} + +if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then + ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$ATTENTION_DP_SIZE" \ + --data-parallel-size-local "$ATTENTION_DP_RANKS" \ + --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size "$ATTENTION_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --enforce-eager \ + $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & +fi + +if [ "$FFN_DP_RANKS" -gt 0 ]; then + FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) + FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$FFN_DP_SIZE" \ + --data-parallel-size-local "$FFN_DP_RANKS" \ + --data-parallel-start-rank "$FFN_DP_START_RANK" \ + --data-parallel-address "$FFN_DP_ADDRESS" \ + --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ + --tensor-parallel-size "$FFN_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --enforce-eager \ + $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & +fi wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh index a51058278..a2f0e27ff 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh @@ -1,52 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & +# Fixed topology for this recipe -- always sent to AFD for rendezvous, +# regardless of which role(s) this node launches locally. +ATTENTION_DP_SIZE=2 +ATTENTION_TP_SIZE=1 +FFN_DP_SIZE=2 +FFN_TP_SIZE=1 +NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) +NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) + +# How many DP ranks of each role to launch on THIS node. 0 skips that role. +# Default runs the full topology on one node (single-node colocation); +# a multi-node deploy overrides one of these to 0 per pod. +ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} +FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} + +# DP sharding: lets a single role's DP group be split across multiple pods. +# Defaults reproduce today's single-pod-per-role behavior exactly (local +# rank count == full DP size, start rank 0, loopback DP-RPC address). +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +FFN_HEADLESS=${FFN_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} +FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} +FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} + +if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then + ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$ATTENTION_DP_SIZE" \ + --data-parallel-size-local "$ATTENTION_DP_RANKS" \ + --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size "$ATTENTION_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --enforce-eager \ + $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & +fi + +if [ "$FFN_DP_RANKS" -gt 0 ]; then + FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) + FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$FFN_DP_SIZE" \ + --data-parallel-size-local "$FFN_DP_RANKS" \ + --data-parallel-start-rank "$FFN_DP_START_RANK" \ + --data-parallel-address "$FFN_DP_ADDRESS" \ + --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ + --tensor-parallel-size "$FFN_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --enforce-eager \ + $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & +fi wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh index 2beabcad1..9b58ca5dd 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh @@ -1,58 +1,108 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 1 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & +# Fixed topology for this recipe -- always sent to AFD for rendezvous, +# regardless of which role(s) this node launches locally. +ATTENTION_DP_SIZE=1 +ATTENTION_TP_SIZE=2 +FFN_DP_SIZE=1 +FFN_TP_SIZE=2 +NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) +NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) + +# How many DP ranks of each role to launch on THIS node. 0 skips that role. +# Default runs the full topology on one node (single-node colocation); +# a multi-node deploy overrides one of these to 0 per pod. +ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} +FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} + +# DP sharding: lets a single role's DP group be split across multiple pods. +# Defaults reproduce today's single-pod-per-role behavior exactly (local +# rank count == full DP size, start rank 0, loopback DP-RPC address). +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +FFN_HEADLESS=${FFN_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} +FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} +FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} + +if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then + ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$ATTENTION_DP_SIZE" \ + --data-parallel-size-local "$ATTENTION_DP_RANKS" \ + --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size "$ATTENTION_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & +fi + +if [ "$FFN_DP_RANKS" -gt 0 ]; then + FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) + FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$FFN_DP_SIZE" \ + --data-parallel-size-local "$FFN_DP_RANKS" \ + --data-parallel-start-rank "$FFN_DP_START_RANK" \ + --data-parallel-address "$FFN_DP_ADDRESS" \ + --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ + --tensor-parallel-size "$FFN_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & +fi wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh index 72db1a705..eec093521 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh @@ -1,58 +1,107 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 1 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 2, - "num_ffn_ranks": 2 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & +# Fixed topology for this recipe -- always sent to AFD for rendezvous, +# regardless of which role(s) this node launches locally. +ATTENTION_DP_SIZE=2 +ATTENTION_TP_SIZE=1 +FFN_DP_SIZE=2 +FFN_TP_SIZE=1 +NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) +NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) + +# How many DP ranks of each role to launch on THIS node. 0 skips that role. +# Default runs the full topology on one node (single-node colocation); +# a multi-node deploy overrides one of these to 0 per pod. +ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} +FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} + +# DP sharding: lets a single role's DP group be split across multiple pods. +# Defaults reproduce today's single-pod-per-role behavior exactly (local +# rank count == full DP size, start rank 0, loopback DP-RPC address). +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +FFN_HEADLESS=${FFN_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} +FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} +FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} + +if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then + ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$ATTENTION_DP_SIZE" \ + --data-parallel-size-local "$ATTENTION_DP_RANKS" \ + --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size "$ATTENTION_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & +fi + +if [ "$FFN_DP_RANKS" -gt 0 ]; then + FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) + FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$FFN_DP_SIZE" \ + --data-parallel-size-local "$FFN_DP_RANKS" \ + --data-parallel-start-rank "$FFN_DP_START_RANK" \ + --data-parallel-address "$FFN_DP_ADDRESS" \ + --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ + --tensor-parallel-size "$FFN_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & +fi wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh index 7b7e9291e..b4cd292c6 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh @@ -1,52 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -CUDA_VISIBLE_DEVICES=0,1,2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 4, - "num_ffn_ranks": 4 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=4,5,6,7 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 4, - "num_ffn_ranks": 4 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & +# Fixed topology for this recipe -- always sent to AFD for rendezvous, +# regardless of which role(s) this node launches locally. +ATTENTION_DP_SIZE=2 +ATTENTION_TP_SIZE=2 +FFN_DP_SIZE=2 +FFN_TP_SIZE=2 +NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) +NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) + +# How many DP ranks of each role to launch on THIS node. 0 skips that role. +# Default runs the full topology on one node (single-node colocation); +# a multi-node deploy overrides one of these to 0 per pod. +ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} +FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} + +# DP sharding: lets a single role's DP group be split across multiple pods. +# Defaults reproduce today's single-pod-per-role behavior exactly (local +# rank count == full DP size, start rank 0, loopback DP-RPC address). +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +FFN_HEADLESS=${FFN_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} +FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} +FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} + +if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then + ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$ATTENTION_DP_SIZE" \ + --data-parallel-size-local "$ATTENTION_DP_RANKS" \ + --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size "$ATTENTION_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --enforce-eager \ + $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & +fi + +if [ "$FFN_DP_RANKS" -gt 0 ]; then + FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) + FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$FFN_DP_SIZE" \ + --data-parallel-size-local "$FFN_DP_RANKS" \ + --data-parallel-start-rank "$FFN_DP_START_RANK" \ + --data-parallel-address "$FFN_DP_ADDRESS" \ + --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ + --tensor-parallel-size "$FFN_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --enforce-eager \ + $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & +fi wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh index baa00323c..95fc82fd5 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh @@ -1,58 +1,108 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -CUDA_VISIBLE_DEVICES=0,1,2,3 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 4, - "num_ffn_ranks": 4 - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & - -CUDA_VISIBLE_DEVICES=4,5,6,7 uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size 2 \ - --tensor-parallel-size 2 \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "127.0.0.1", - "port": 6269, - "num_attention_ranks": 4, - "num_ffn_ranks": 4 - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & +# Fixed topology for this recipe -- always sent to AFD for rendezvous, +# regardless of which role(s) this node launches locally. +ATTENTION_DP_SIZE=2 +ATTENTION_TP_SIZE=2 +FFN_DP_SIZE=2 +FFN_TP_SIZE=2 +NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) +NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) + +# How many DP ranks of each role to launch on THIS node. 0 skips that role. +# Default runs the full topology on one node (single-node colocation); +# a multi-node deploy overrides one of these to 0 per pod. +ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} +FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} + +# DP sharding: lets a single role's DP group be split across multiple pods. +# Defaults reproduce today's single-pod-per-role behavior exactly (local +# rank count == full DP size, start rank 0, loopback DP-RPC address). +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +FFN_HEADLESS=${FFN_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} +FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} +FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} + +if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then + ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$ATTENTION_DP_SIZE" \ + --data-parallel-size-local "$ATTENTION_DP_RANKS" \ + --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size "$ATTENTION_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & +fi + +if [ "$FFN_DP_RANKS" -gt 0 ]; then + FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) + FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size "$FFN_DP_SIZE" \ + --data-parallel-size-local "$FFN_DP_RANKS" \ + --data-parallel-start-rank "$FFN_DP_START_RANK" \ + --data-parallel-address "$FFN_DP_ADDRESS" \ + --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ + --tensor-parallel-size "$FFN_TP_SIZE" \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', + "num_ffn_ranks": '"${NUM_FFN_RANKS}"' + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & +fi wait diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph_multinode.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh similarity index 51% rename from recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph_multinode.sh rename to recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh index 2e6b364d1..4affd40bd 100644 --- a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph_multinode.sh +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh @@ -6,22 +6,43 @@ export VLLM_USE_V2_MODEL_RUNNER=0 # Fixed topology for this recipe -- always sent to AFD for rendezvous, # regardless of which role(s) this node launches locally. -NUM_ATTENTION_RANKS=4 -NUM_FFN_RANKS=2 +ATTENTION_DP_SIZE=4 +ATTENTION_TP_SIZE=1 +FFN_DP_SIZE=2 +FFN_TP_SIZE=1 +NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) +NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) -# How many ranks of each role to launch on THIS node. 0 skips that role. +# How many DP ranks of each role to launch on THIS node. 0 skips that role. # Default runs the full topology on one node (single-node colocation); # a multi-node deploy overrides one of these to 0 per pod. -ATTENTION_RANKS=${ATTENTION_RANKS:-$NUM_ATTENTION_RANKS} -FFN_RANKS=${FFN_RANKS:-$NUM_FFN_RANKS} +ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} +FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-1239} -if [ "$ATTENTION_RANKS" -gt 0 ]; then - ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_RANKS - 1))) +# DP sharding: lets a single role's DP group be split across multiple pods. +# Defaults reproduce today's single-pod-per-role behavior exactly (local +# rank count == full DP size, start rank 0, loopback DP-RPC address). +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +FFN_HEADLESS=${FFN_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} +FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} +FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} + +if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then + ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$ATTENTION_RANKS" \ - --tensor-parallel-size 1 \ + --data-parallel-size "$ATTENTION_DP_SIZE" \ + --data-parallel-size-local "$ATTENTION_DP_RANKS" \ + --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size "$ATTENTION_TP_SIZE" \ --enable-expert-parallel \ --dtype bfloat16 \ --language-model-only \ @@ -33,7 +54,7 @@ if [ "$ATTENTION_RANKS" -gt 0 ]; then "afd": { "role": "attention", "connector": "P2pNcclAFDConnector", - "host": "vllm-ffn-p2p-service", + "host": "'"${AFD_CONNECTOR_HOST}"'", "port": '"${AFD_CONNECTOR_PORT}"', "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', "num_ffn_ranks": '"${NUM_FFN_RANKS}"' @@ -45,15 +66,21 @@ if [ "$ATTENTION_RANKS" -gt 0 ]; then --compilation-config '{ "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] }' \ + $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ --host 127.0.0.1 \ --port 18305 > attn.log 2>&1 & fi -if [ "$FFN_RANKS" -gt 0 ]; then - FFN_DEVICES=$(seq -s, "$ATTENTION_RANKS" $((ATTENTION_RANKS + FFN_RANKS - 1))) +if [ "$FFN_DP_RANKS" -gt 0 ]; then + FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) + FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$FFN_RANKS" \ - --tensor-parallel-size 1 \ + --data-parallel-size "$FFN_DP_SIZE" \ + --data-parallel-size-local "$FFN_DP_RANKS" \ + --data-parallel-start-rank "$FFN_DP_START_RANK" \ + --data-parallel-address "$FFN_DP_ADDRESS" \ + --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ + --tensor-parallel-size "$FFN_TP_SIZE" \ --enable-expert-parallel \ --dtype bfloat16 \ --language-model-only \ @@ -65,7 +92,7 @@ if [ "$FFN_RANKS" -gt 0 ]; then "afd": { "role": "ffn", "connector": "P2pNcclAFDConnector", - "host": "vllm-ffn-p2p-service", + "host": "'"${AFD_CONNECTOR_HOST}"'", "port": '"${AFD_CONNECTOR_PORT}"', "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', "num_ffn_ranks": '"${NUM_FFN_RANKS}"' @@ -77,6 +104,7 @@ if [ "$FFN_RANKS" -gt 0 ]; then --compilation-config '{ "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] }' \ + $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ --host 127.0.0.1 \ --port 18305 > ffn.log 2>&1 & fi From ad11df45607e748fee28f804b51dab908b203475 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Wed, 23 Sep 2026 16:35:45 +0300 Subject: [PATCH 03/15] fix pre-commit errors Signed-off-by: Shlomit Koyfman --- .../prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh | 2 ++ .../prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh | 2 ++ .../prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh | 2 ++ .../prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh | 2 ++ .../prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh | 2 ++ .../prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh | 2 ++ 6 files changed, 12 insertions(+) diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh index 85e061bf6..f6ecc29d1 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh @@ -36,6 +36,7 @@ FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$ATTENTION_DP_SIZE" \ --data-parallel-size-local "$ATTENTION_DP_RANKS" \ @@ -69,6 +70,7 @@ fi if [ "$FFN_DP_RANKS" -gt 0 ]; then FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$FFN_DP_SIZE" \ --data-parallel-size-local "$FFN_DP_RANKS" \ diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh index a2f0e27ff..b85c28512 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh @@ -36,6 +36,7 @@ FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$ATTENTION_DP_SIZE" \ --data-parallel-size-local "$ATTENTION_DP_RANKS" \ @@ -69,6 +70,7 @@ fi if [ "$FFN_DP_RANKS" -gt 0 ]; then FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$FFN_DP_SIZE" \ --data-parallel-size-local "$FFN_DP_RANKS" \ diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh index 9b58ca5dd..a2b119716 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh @@ -36,6 +36,7 @@ FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$ATTENTION_DP_SIZE" \ --data-parallel-size-local "$ATTENTION_DP_RANKS" \ @@ -72,6 +73,7 @@ fi if [ "$FFN_DP_RANKS" -gt 0 ]; then FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$FFN_DP_SIZE" \ --data-parallel-size-local "$FFN_DP_RANKS" \ diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh index eec093521..54fcef80a 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh @@ -36,6 +36,7 @@ FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$ATTENTION_DP_SIZE" \ --data-parallel-size-local "$ATTENTION_DP_RANKS" \ @@ -72,6 +73,7 @@ fi if [ "$FFN_DP_RANKS" -gt 0 ]; then FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$FFN_DP_SIZE" \ --data-parallel-size-local "$FFN_DP_RANKS" \ diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh index b4cd292c6..0eb987ef9 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh @@ -36,6 +36,7 @@ FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$ATTENTION_DP_SIZE" \ --data-parallel-size-local "$ATTENTION_DP_RANKS" \ @@ -69,6 +70,7 @@ fi if [ "$FFN_DP_RANKS" -gt 0 ]; then FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$FFN_DP_SIZE" \ --data-parallel-size-local "$FFN_DP_RANKS" \ diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh index 95fc82fd5..2b87d9624 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh @@ -36,6 +36,7 @@ FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$ATTENTION_DP_SIZE" \ --data-parallel-size-local "$ATTENTION_DP_RANKS" \ @@ -72,6 +73,7 @@ fi if [ "$FFN_DP_RANKS" -gt 0 ]; then FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$FFN_DP_SIZE" \ --data-parallel-size-local "$FFN_DP_RANKS" \ From 8085ddbe36efc9ce97fa3d87dc043e131d19ede2 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Wed, 23 Sep 2026 16:39:55 +0300 Subject: [PATCH 04/15] pre-commit errors Signed-off-by: Shlomit Koyfman --- .../prefill_decode_colocation/4a2f_graph.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh index 4affd40bd..ac7206c6b 100644 --- a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh @@ -36,6 +36,7 @@ FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$ATTENTION_DP_SIZE" \ --data-parallel-size-local "$ATTENTION_DP_RANKS" \ @@ -74,6 +75,7 @@ fi if [ "$FFN_DP_RANKS" -gt 0 ]; then FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) + # shellcheck disable=SC2046 CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ --data-parallel-size "$FFN_DP_SIZE" \ --data-parallel-size-local "$FFN_DP_RANKS" \ From d5516c410d9d2c96a3b001e80cf368ebd3cf2e20 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Wed, 23 Sep 2026 17:19:39 +0300 Subject: [PATCH 05/15] add README for deploying recipes on a local host or in a k8s cluster Signed-off-by: Shlomit Koyfman --- recipe/gpu/P2pNcclAFDConnector/README.md | 88 ++++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 recipe/gpu/P2pNcclAFDConnector/README.md diff --git a/recipe/gpu/P2pNcclAFDConnector/README.md b/recipe/gpu/P2pNcclAFDConnector/README.md new file mode 100644 index 000000000..7abf26660 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/README.md @@ -0,0 +1,88 @@ +# P2pNcclAFDConnector Recipes + +Launch scripts for running Attention-FFN Disaggregation (AFD) with +`P2pNcclAFDConnector`, a GPU point-to-point connector built on vLLM's +`PyNcclCommunicator`. See +[`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`](../../../docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md) +for the connector's rank mapping and configuration contract. + +## Directory layout + +```text +. +├── deepseek_v2_lite/ +│ ├── README.md +│ ├── prefill_decode_colocation/ # 2A2F, DP/TP variants, eager/graph +│ └── prefill_decode_disaggregation/ # 2P1A1F +└── qwen3_5_122b_a10b_fp8/ + └── prefill_decode_colocation/ # 4A2F, graph +``` + +Each model directory holds its own recipe scripts; open its `README.md` +(where present) before running anything in it -- it documents prerequisites, +topology, ports, and known limitations for that model. + +Recipes under `prefill_decode_colocation/` background an attention worker and +an FFN worker (no prefill split). Recipes under `prefill_decode_disaggregation/` +add a separate prefill stage and a proxy in front of it -- see the caveat in +"Deploying on Kubernetes" below. + +## Running on a local host + +Run any script directly from the repository root, e.g.: + +```bash +export MODEL_PATH=/path/model_weights/DeepSeek-V2-Lite +bash recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh +``` + +Each script backgrounds its workers and writes per-worker logs (`attn.log`, +`ffn.log`, and for disaggregation `afd_prefill*.log`) into the current +directory. Wait for `Application startup complete` in each log before +sending traffic. See the model README for exact prerequisites (GPU count, +weights, ports) and the benchmark command for that model. + +## Deploying on Kubernetes + +Use the **`deploy-afd-k8s`** skill (`.agents/skills/deploy-afd-k8s/SKILL.md`) +to stand up a recipe from this folder on a Kubernetes/OpenShift cluster. It +provisions the model PVC, mounts the chosen recipe script into a serve pod +via a ConfigMap (no image rebuild needed to try an edited or new recipe), +and waits until an OpenAI-compatible endpoint answers. + +``` +/deploy-afd-k8s +``` + +then give it the recipe path (e.g. +`recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh`) +plus the image, model id, and PVC name it needs. It derives `GPU_COUNT` +itself for a plain single-node recipe; for a multi-node-capable recipe it +instead confirms the recipe's attention/FFN rank totals and asks you for a +placement plan (see below). + +**Scope.** The skill only covers `prefill_decode_colocation` recipes (and +their `baseline*` counterparts) under `recipe/gpu/P2pNcclAFDConnector/**`. +`prefill_decode_disaggregation` recipes are out of scope for it -- they need +a NIXL-enabled image and a proxy, and rely on a `SCRIPT_DIR`-relative path +that breaks once the script is mounted flat into the pod. Run those locally +instead, or ask for a different workflow. + +**Single-node.** Any recipe deploys as one pod holding every rank -- this is +the simple, non-experimental path. The skill computes `GPU_COUNT` itself from +the script's `CUDA_VISIBLE_DEVICES` blocks and deploys a single `vllm-pod` / +`vllm-service` pair. + +**Multi-node.** Only recipes that gate their `vllm serve` blocks on +`ATTENTION_DP_RANKS`/`FFN_DP_RANKS` and read the AFD `host` field from an +`AFD_CONNECTOR_HOST` env var support spreading attention/FFN ranks across +more than one pod (e.g. `deepseek_v2_lite/prefill_decode_colocation/*.sh`). +For these, the skill reads the recipe's fixed attention/FFN rank totals and +asks you for an explicit **placement plan** -- how many of those ranks go in +each pod, and how many pods -- rather than assuming a shape. It then creates +one pod per plan entry, an internal FFN rendezvous Service when the plan +spans more than one pod, and DP-coordination Services for any role that is +itself split across pods. + +Pods and Services are left running after deployment so weights stay warm for +follow-up runs; see the skill's teardown step to tear them down explicitly. From aa4bfc41408df0c86ffaf73ed63fe8e1c7a4d527 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Wed, 23 Sep 2026 17:21:24 +0300 Subject: [PATCH 06/15] add README for deploying recipes on a local host or in a k8s cluster Signed-off-by: Shlomit Koyfman --- recipe/gpu/P2pNcclAFDConnector/README.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/recipe/gpu/P2pNcclAFDConnector/README.md b/recipe/gpu/P2pNcclAFDConnector/README.md index 7abf26660..59484e71b 100644 --- a/recipe/gpu/P2pNcclAFDConnector/README.md +++ b/recipe/gpu/P2pNcclAFDConnector/README.md @@ -50,11 +50,7 @@ provisions the model PVC, mounts the chosen recipe script into a serve pod via a ConfigMap (no image rebuild needed to try an edited or new recipe), and waits until an OpenAI-compatible endpoint answers. -``` -/deploy-afd-k8s -``` - -then give it the recipe path (e.g. +The skill requires the recipe path (e.g. `recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh`) plus the image, model id, and PVC name it needs. It derives `GPU_COUNT` itself for a plain single-node recipe; for a multi-node-capable recipe it From ae414fe7899115fc87db93ec3ed3bd267b1949bc Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Sun, 4 Oct 2026 13:57:41 +0300 Subject: [PATCH 07/15] PR review fixes Signed-off-by: Shlomit Koyfman --- .agents/skills/deploy-afd-k8s/SKILL.md | 1107 +---------------- .../deploy-afd-k8s/references/deploy.md | 211 ++++ .../references/report-and-teardown.md | 31 + .../references/resolve-recipe.md | 79 ++ .../deploy-afd-k8s/references/services.md | 64 + .../skills/deploy-afd-k8s/templates/pod.yaml | 190 +++ .../skills/deploy-afd-k8s/templates/pvc.yaml | 13 + .../templates/service-client.yaml | 15 + .../templates/service-dp-coord.yaml | 23 + .../templates/service-ffn-p2p.yaml | 20 + recipe/gpu/P2pNcclAFDConnector/README.md | 30 +- .../deepseek_v2_lite/README.md | 19 +- .../2a2f_eager_dbo_dp1tp2.sh | 146 +-- .../2a2f_eager_dbo_dp2tp1.sh | 146 +-- .../2a2f_graph_dbo_dp1tp2.sh | 158 +-- .../2a2f_graph_dbo_dp2tp1.sh | 157 +-- .../4a4f_eager_dbo_dp2tp2.sh | 146 +-- .../4a4f_graph_dbo_dp2tp2.sh | 158 +-- .../multipod_2a_2f_graph_dbo_dp1tp2.sh | 73 ++ .../prefill_decode_colocation/4a2f_graph.sh | 114 -- .../multipod_2a_2a_2f_graph.sh | 88 ++ 21 files changed, 1177 insertions(+), 1811 deletions(-) create mode 100644 .agents/skills/deploy-afd-k8s/references/deploy.md create mode 100644 .agents/skills/deploy-afd-k8s/references/report-and-teardown.md create mode 100644 .agents/skills/deploy-afd-k8s/references/resolve-recipe.md create mode 100644 .agents/skills/deploy-afd-k8s/references/services.md create mode 100644 .agents/skills/deploy-afd-k8s/templates/pod.yaml create mode 100644 .agents/skills/deploy-afd-k8s/templates/pvc.yaml create mode 100644 .agents/skills/deploy-afd-k8s/templates/service-client.yaml create mode 100644 .agents/skills/deploy-afd-k8s/templates/service-dp-coord.yaml create mode 100644 .agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml create mode 100644 recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh delete mode 100644 recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh create mode 100644 recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh diff --git a/.agents/skills/deploy-afd-k8s/SKILL.md b/.agents/skills/deploy-afd-k8s/SKILL.md index 30d2c0dfa..bcbfe1b9d 100644 --- a/.agents/skills/deploy-afd-k8s/SKILL.md +++ b/.agents/skills/deploy-afd-k8s/SKILL.md @@ -1,11 +1,11 @@ --- name: deploy-afd-k8s -description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. For multi-node-capable recipes (parameterized with ATTENTION_DP_RANKS/FFN_DP_RANKS env vars), ask the caller for an explicit placement plan -- how many attention/FFN ranks go in each Pod, and how many Pods -- rather than assuming a shape; a plan can range from one Pod holding every rank to many Pods with arbitrary, possibly mixed, per-Pod shares (experimental beyond the single-Pod case). Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). To drive load against what this deploys, see run-vllm-bench-k8s. +description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is read directly off the recipe's own top-of-file comment block (however that block describes it, strict or prose): a recipe whose comment describes a per-pod placement deploys exactly that many pods with exactly that per-pod setup; a recipe without one deploys as a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). To drive load against what this deploys, see run-vllm-bench-k8s. --- # Deploy an AFD GPU recipe on Kubernetes -Stand up a serve pod running a local AFD recipe script on a +Stand up a serve pod (or pods) running a local AFD recipe script on a Kubernetes/OpenShift cluster and leave a reachable OpenAI-compatible endpoint behind. @@ -15,61 +15,35 @@ On success this skill guarantees, in the target namespace: | | | |---|---| -| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`, resolved in step 1) | +| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`) | | Serving | the model named by `MODEL_ID` | -| Pod(s) / Service | one or more Pods, named per the caller's placement plan (step 1b) for a multi-node-capable recipe, or the single `vllm-pod` for a plain fixed recipe; Service `vllm-service`, labels `app=afd-recipe,role=serve` | +| Pod(s) / Service | `vllm-pod`, or one pod per pod the recipe's header comment describes (named `vllm-pod-`); Service `vllm-service`, labels `app=afd-recipe,role=serve` | | Model cache | `PVC_NAME`, mounted at `/models`, `HF_HOME=/models/.hf_home` | | Left running | yes, deliberately -- so weights stay warm for follow-up runs | -Callers only need `MODEL_ID`, `PVC_NAME`, and `GPU_COUNT` back for a plain -fixed single-node recipe. - -For a multi-node-capable recipe, the caller's placement plan (step 1b) -decides how many Pods exist and what each is named. The same contract still -holds -- `vllm-service:${CLIENT_PORT}` answers OpenAI-compatible traffic, -backed by whichever Pod holds attention rank 0. Whenever the plan spans more -than one Pod, an additional internal-only `vllm-ffn-p2p-service` also exists -solely to carry AFD rendezvous traffic to whichever Pod holds FFN rank 0; -callers never talk to it directly. If either role is itself split across -more than one Pod, a `vllm-attn-dp-service` and/or `vllm-ffn-dp-service` also -exist to carry that role's internal DP-RPC coordination traffic. +Whenever a recipe's header spans more than one pod, two more internal-only +Services may also exist (`vllm-ffn-p2p-service` always; `vllm-attn-dp-service` +and/or `vllm-ffn-dp-service` only if that role is itself split across more +than one pod) -- callers never talk to these directly. See +[services.md](references/services.md). ## Scope -- Recipes under `recipe/gpu/P2pNcclAFDConnector/**` (GPU only, not NPU). -- **`prefill_decode_colocation` only** -- attention + FFN workers, no - prefill split (e.g. `2a2f_graph_dbo_dp2tp1.sh`, `4a4f_eager_dbo_dp2tp2.sh`), - and their `baseline*.sh` counterparts. - `prefill_decode_disaggregation` recipes (e.g. `2p1a1f_graph_dbo.sh`) are - **out of scope**: they need a NIXL-enabled image plus a proxy in front of - the split, and they resolve `$SCRIPT_DIR/../../../../../tools/proxy_server.py` - relative to the script. Step 4b mounts the script flat at - `/recipe/recipe.sh`, so that path would resolve to a nonexistent - `/tools/proxy_server.py`. Colocation recipes reference `SCRIPT_DIR` zero - times, which is what makes the flat mount safe. Ask the user for a +- Recipes under `recipe/gpu/P2pNcclAFDConnector/**` (GPU only, not NPU), + **`prefill_decode_colocation` only** (attention + FFN workers, no prefill + split) and their `baseline*.sh` counterparts. + `prefill_decode_disaggregation` recipes are **out of scope**: they need a + NIXL-enabled image plus a proxy, and resolve a `SCRIPT_DIR`-relative path + that this skill's flat script mount (step 4b) breaks. Ask the user for a different workflow for disaggregation. - Recipes are hand-written shell scripts, not JSON configs -- there is no generator step and no manifest to read. -- A recipe is **multi-node-capable** when it gates its `vllm serve` blocks - on `ATTENTION_DP_RANKS`/`FFN_DP_RANKS` env vars (e.g. - `if [ "$ATTENTION_DP_RANKS" -gt 0 ]`) and reads its AFD `host` field from an - `AFD_CONNECTOR_HOST` env var (e.g. - `AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1}`, used as - `"host": "'"${AFD_CONNECTOR_HOST}"'"` inside `--additional-config`; see - e.g. the `deepseek_v2_lite/prefill_decode_colocation/*.sh` recipes) -- - grep for both before assuming a script only supports single-node. - Deploying such a script means asking the caller for an explicit - **placement plan** (step 1b): a list of Pods, each carrying its own share - of attention/FFN ranks (summing to the recipe's fixed `NUM_ATTENTION_RANKS`/ - `NUM_FFN_RANKS` totals) -- anywhere from one Pod holding every rank, to a - dedicated Pod per role, to many Pods each holding an arbitrary, possibly - mixed, share of either role. **Never assume a shape on the caller's - behalf** -- ask, every time. Placements beyond a single Pod holding - everything are **experimental**: upstream documents cross-node - `P2pNcclAFDConnector` use as "not established by the current recipes ... - treated as unverified" (`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`). - Deploy whatever the caller asks for, but don't present throughput as - validated once the plan spans more than one Pod. +- A recipe is **placement-driven** (more than one pod) exactly when its + top-of-file comment block describes a per-pod placement -- read that + block, don't infer shape from flags. Today's recipes spell it as one + `# For pod N: POD=...` line per pod, but the comment isn't guaranteed to + use that exact grammar going forward. See + [resolve-recipe.md](references/resolve-recipe.md). - Not for `tools/benchmarks/decode_bench_server.sh` (local, non-k8s). ## Requirements @@ -94,1029 +68,24 @@ exist to carry that role's internal DP-RPC coordination traffic. ```bash kubectl create secret generic hf-token-secret --from-literal=token= ``` -- Nodes with `GPU_COUNT` free GPUs (step 2). +- Nodes with enough free GPUs for `GPU_COUNT` (per pod). ## Workflow -### 1. Resolve the recipe and its model - -Take the recipe path from the user, or ask. Read the script and note, per -`vllm serve` block (each backgrounded process ending `> .log 2>&1 &`): - -- `CUDA_VISIBLE_DEVICES` (a literal list on a plain single-node recipe; a - computed variable like `$ATTN_DEVICES`/`$FFN_DEVICES` on a multi-node- - capable one -- see below) -- `--data-parallel-size`, `--tensor-parallel-size` -- the `"afd": {"role": ...}` block in `--additional-config` (attention vs ffn) -- eager (`--enforce-eager`) vs graph (`--max-cudagraph-capture-size` + - `--compilation-config`) -- every other flag (`--enable-expert-parallel`, `--max-num-seqs`, - `--max-num-batched-tokens`, `--max-model-len`, `--trust-remote-code`, - host/port) - -**Check whether the script is multi-node-capable** by grepping for -`ATTENTION_DP_RANKS`/`FFN_DP_RANKS` env-var gating (e.g. -`if [ "$ATTENTION_DP_RANKS" -gt 0 ]`) and an `AFD_CONNECTOR_HOST` env-var -default (e.g. `AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1}`). -Neither present means it's a plain fixed single-node script -- skip straight -to step 2. Both present means the script's fixed rank totals (below) can be -placed across Pods however the caller wants -- continue to step 1b. - -For a multi-node-capable script, also read its hardcoded topology constants -(`NUM_ATTENTION_RANKS=`, `NUM_FFN_RANKS=`, near the top -- fixed per recipe -and always sent to AFD for rendezvous regardless of how the caller places -them) and its `AFD_CONNECTOR_PORT` default -(`AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-}`). Only export -`AFD_CONNECTOR_PORT` before deploying if the caller specifically asked for -a non-default rendezvous port. - -Set `CLIENT_PORT` to the `--port` value on the attention (or sole, for -baseline) `vllm serve` block -- every checked-in recipe uses `18305`, but -step 4 must rebind and expose whatever value is actually there, not assume -`18305`. - -Record `--max-num-batched-tokens` explicitly and report it to the caller -- -it caps how much prefill work a single step can absorb and is the usual -reason a load profile saturates a recipe instantly. - -Resolve `MODEL_ID`, the model the serve pod is told to serve. Read the -script's `MODEL_PATH` default plus the model directory name -(`recipe/gpu/P2pNcclAFDConnector//...`), e.g. `deepseek_v2_lite` -> -`deepseek-ai/DeepSeek-V2-Lite`. If the mapping is not obvious for an -unfamiliar family, ask rather than guess. - -`MODEL_ID` may instead be a literal in-container path (e.g. -`/models/Qwen3.5-122B-A10B-FP8`) when weights are staged on the PVC rather -than published on HF Hub -- matching the script's own `MODEL_PATH` default. -The serve pod handles either form transparently. **Tell the caller which -form it is**; consumers that load their own tokenizer need to know. - -### 1b. Ask for a placement plan (multi-node-capable scripts only) - -Skip this step entirely for a plain fixed single-node script (step 1 already -sent it straight to step 2). - -Ask the caller how they want the recipe's fixed `NUM_ATTENTION_RANKS` -attention ranks and `NUM_FFN_RANKS` FFN ranks placed across Pods. **Do not -default to a particular shape** -- these are all equally valid answers to -the same question: - -- one Pod holding every rank (`attn_ranks=NUM_ATTENTION_RANKS, - ffn_ranks=NUM_FFN_RANKS` on a single Pod) -- the simplest shape, and the - only one that isn't experimental; -- a dedicated Pod per role (one Pod with all attention ranks, another with - all FFN ranks); -- either role split across several Pods (e.g. attention's ranks spread - 2-and-2 across two Pods, FFN whole in a third); -- Pods that mix a partial share of both roles (e.g. two Pods of `2 attention - + 1 ffn` each for a 4-attention/2-ffn recipe). - -Record the plan as a list of `(pod_name, attn_ranks, ffn_ranks)` tuples, in -the order the caller wants them evaluated. Validate before proceeding: - -- every `pod_name` is unique; -- `sum(attn_ranks) == NUM_ATTENTION_RANKS` and `sum(ffn_ranks) == - NUM_FFN_RANKS` across the whole plan -- AFD's rendezvous always expects - the recipe's fixed totals regardless of how they're distributed, so an - under- or over-count silently breaks rendezvous rather than failing - loudly; -- every Pod has `attn_ranks + ffn_ranks >= 1` (no empty Pods). - -**Derive per-role rank ordering**, needed for `_START_RANK`/`_HEADLESS`/ -`_DP_ADDRESS` in step 4-placed: within each role, walk the plan in the given -order and set that Pod's `start_rank` for the role to the running sum of the -role's local counts from every earlier Pod carrying it. The first Pod -carrying a nonzero share of a role is that role's **head** -(`start_rank=0`, `headless=0` -- it runs attention's API server, or binds -FFN's rendezvous rank 0); every later Pod carrying that role is a **worker** -(`headless=1`). A role that is whole in one Pod trivially has only a head, -no workers. - -Only a role that is genuinely split across more than one Pod needs -`_START_RANK`/`_HEADLESS`/`_DP_ADDRESS` to vary at all -- and only if the -recipe exposes `ATTENTION_DP_START_RANK`/`FFN_DP_START_RANK`, -`ATTENTION_HEADLESS`/`FFN_HEADLESS`, `ATTENTION_DP_ADDRESS`/`FFN_DP_ADDRESS` -env vars (grep for `_START_RANK`/`_HEADLESS`; a script only gated on -`ATTENTION_DP_RANKS`/`FFN_DP_RANKS`/`AFD_CONNECTOR_HOST` can place each role -freely relative to the other, but can't split a single role across more -than one Pod). Flag this to the caller before proposing a plan that would -require it. - -**Resolve `AFD_CONNECTOR_HOST`** from the plan's Pod count, not from a fixed -mode: exactly one Pod in the plan means the recipe's loopback default -(`127.0.0.1`) already works -- no Service needed, don't override it. More -than one Pod means every Pod's container env must set `AFD_CONNECTOR_HOST` -to `vllm-ffn-p2p-service` (created in step 4-placed), including Pods that -carry no FFN ranks themselves -- per -`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`, every participating rank must -agree on the same host string, and the Pod holding FFN rank 0 both binds and -is reached through that name. - -**Node placement is left to the scheduler by default.** Do not add -`podAntiAffinity` forcing Pods onto separate nodes unless the caller -explicitly asks to exercise cross-node placement -- an unforced multi-Pod -plan may legitimately land every Pod on the same node, and that's fine. -Only add the shared-label `podAntiAffinity` block in step 4-placed if asked. - -### 2. Compute GPU_COUNT - -For a plain single-node script (step 1 found no `ATTENTION_DP_RANKS`/ -`FFN_DP_RANKS` gating): `GPU_COUNT` = number of distinct GPU indices across -every literal `CUDA_VISIBLE_DEVICES=` in the script (union, not sum -- each -index appears in exactly one block). - -For a multi-node-capable script, `GPU_COUNT` is per-Pod, read straight off -the placement plan from step 1b: for each Pod, `GPU_COUNT = attn_ranks + -ffn_ranks` assigned to it there. - -### 3. Confirm before touching the cluster - -State and confirm: target cluster/namespace, `AFD_PLUGIN_IMAGE`, `MODEL_ID`, -`PVC_NAME`, `CLIENT_PORT`, `RECIPE_SCRIPT_PATH`, and either `GPU_COUNT` -(plain single-node script) or the full placement plan as a table of -`pod_name | attn_ranks | ffn_ranks | GPU_COUNT` (multi-node-capable script). -Flag explicitly that step 4 **deletes any existing Pod(s) with the same -name(s)** as what's about to be deployed (`vllm-pod`, or every `pod_name` in -the placement plan) -- if one exists from unrelated work, that is -destructive and needs approval first. - -`RECIPE_SCRIPT_PATH` is a *local* path. It need not be committed, exist in -the image, or live at any particular depth -- but per Scope it must be a -colocation recipe, since nothing rewrites `SCRIPT_DIR`-relative lookups. - -```bash -AFD_PLUGIN_IMAGE= -MODEL_ID= -PVC_NAME= -CLIENT_PORT= # from the script's --port; default 18305 -RECIPE_SCRIPT_PATH= -VLLM_USE_V2_MODEL_RUNNER=${VLLM_USE_V2_MODEL_RUNNER:-0} # pulled from the caller's env, default 0 -``` - -For a plain single-node script: - -```bash -GPU_COUNT= -``` - -For a multi-node-capable script, confirm the full placement plan instead, -e.g. for a 4-attention/2-ffn recipe split as two attention Pods of 2 ranks -each plus a dedicated FFN Pod: - -```bash -# pod_name attn_ranks ffn_ranks gpu_count -# vllm-attn-pod-0 2 0 2 -# vllm-attn-pod-1 2 0 2 -# vllm-ffn-pod 0 2 2 - -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-} # only export if overriding -AFD_CONNECTOR_HOST=vllm-ffn-p2p-service # only when the plan has >1 pod (step 1b); leave unset/loopback for a single-pod plan -``` - -or, for the same recipe placed as one Pod holding every rank: - -```bash -# pod_name attn_ranks ffn_ranks gpu_count -# vllm-pod-0 4 2 6 -``` - -### 4. Deploy - -**4a. Model PVC** (created once; reused so weights stay warm). - -Size it for the model: a DeepSeek-V2-Lite needs far less than a -Qwen3.5-122B-A10B-FP8 checkpoint (~122 GB, so `200Gi` minimum). Set -`MODEL_PVC_SIZE` deliberately -- an existing undersized PVC is *reused -silently* by the branch below and the download then fails with ENOSPC after -the GPUs are already claimed. Set `STORAGE_CLASS` to a class the cluster -actually offers; leaving it unset falls to the default, which is often -RWO-only or absent and leaves the pod `Pending`. - -Whenever the placement plan (step 1b) has more than one Pod, the PVC -**must** be `ReadWriteMany` (RWX), not `ReadWriteOnce` -- even without -forced anti-affinity (step 1b defaults to leaving node placement to the -scheduler), nothing guarantees the Pods land on the same node, and an RWO -volume can only attach on one node at a time. The second Pod to need it -fails at admission with `Multi-Attach error for volume ... already used by -pod(s) ...` and sits `Pending` forever. This is a hard deadlock, not -something that resolves by waiting longer. Before deploying a multi-Pod -plan, confirm the storage class actually supports RWX (a throwaway 1Gi PVC -requesting `ReadWriteMany` against the same `STORAGE_CLASS` either binds or -it doesn't -- delete it after). If an existing single-Pod PVC is RWO and -already holds downloaded weights you want to reuse for a multi-Pod plan, -provision a new RWX PVC and copy the data across rather than re-downloading --- e.g. a short-lived Pod pinned via `nodeName` to whichever node already -holds the RWO mount (same-node multi-mount of an RWO volume is fine), with -both PVCs as volumes, running `cp -a /old/. /new/.` (a nonzero exit purely -from failing to preserve the top-level directory's mtime is cosmetic; -verify with `du -sh` on both sides instead of trusting the exit code). - -```bash -MODEL_PVC_SIZE=${MODEL_PVC_SIZE:-100Gi} -STORAGE_CLASS=${STORAGE_CLASS:-} # e.g. ocs-storagecluster-cephfs -PVC_ACCESS_MODE=${PVC_ACCESS_MODE:-ReadWriteOnce} # ReadWriteMany whenever the placement plan has more than one Pod - -if kubectl get pvc "${PVC_NAME}" >/dev/null 2>&1; then - have="$(kubectl get pvc "${PVC_NAME}" -o jsonpath='{.spec.resources.requests.storage}')" - have_modes="$(kubectl get pvc "${PVC_NAME}" -o jsonpath='{.spec.accessModes}')" - echo "PVC ${PVC_NAME} exists (${have}, ${have_modes}); serving ${MODEL_ID} from its warm HF_HOME cache" - echo "NOTE: verify ${have} fits ${MODEL_ID} -- an undersized reused PVC fails mid-download" - echo "NOTE: for a multi-Pod placement plan, ${have_modes} must include ReadWriteMany or the second Pod will deadlock on attach" -else - envsubst '${PVC_NAME} ${MODEL_PVC_SIZE} ${STORAGE_CLASS} ${PVC_ACCESS_MODE}' <<'EOF' | \ - grep -v 'storageClassName: *$' | kubectl apply -f - -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: ${PVC_NAME} - labels: - app: afd-recipe -spec: - accessModes: - - ${PVC_ACCESS_MODE} - storageClassName: ${STORAGE_CLASS} - resources: - requests: - storage: ${MODEL_PVC_SIZE} -EOF -fi -``` - -**4b. Recipe ConfigMap.** This is what lets the pod run any local recipe -- -edited, uncommitted, or brand new -- without rebuilding the image. Shared as- -is regardless of how many Pods the placement plan has (it's the *same* -script mounted into every Pod). Run from wherever `RECIPE_SCRIPT_PATH` -resolves (e.g. the repo root): - -```bash -kubectl create configmap afd-recipe-script \ - --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \ - --dry-run=client -o yaml | kubectl apply -f - -``` - -**4c. Serve pod** (plain fixed single-node script only -- for a multi-node- -capable script, skip to **4-placed** below instead). The container runs the -mounted recipe unmodified except for rebinding the client-facing port off -loopback: the recipe binds `:${CLIENT_PORT}` on `127.0.0.1` (it is normally -driven from inside the same host), but the Service needs `0.0.0.0`. Every -internal worker port stays on loopback. The `uv` shim on `PATH` strips -`uv run` -- the image's base environment is already correct. - -`fsGroup` must be inside the namespace's allowed range or the pod is -rejected at admission (`fsGroup: Invalid value: ... is not an allowed -group`). Look it up rather than copying a number: - -```bash -FS_GROUP="$(kubectl get ns "$(kubectl config view --minify -o jsonpath='{..namespace}')" \ - -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' 2>/dev/null \ - | cut -d/ -f1)" -FS_GROUP="${FS_GROUP:-1000}" -echo "using fsGroup=${FS_GROUP}" - -TEMPLATE_IMAGE="$AFD_PLUGIN_IMAGE" -TEMPLATE_MODEL="$MODEL_ID" -TEMPLATE_CLIENT_PORT="$CLIENT_PORT" - -kubectl delete pod vllm-pod --ignore-not-found - -envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${GPU_COUNT} ${PVC_NAME} ${FS_GROUP} ${TEMPLATE_CLIENT_PORT} ${VLLM_USE_V2_MODEL_RUNNER}' <<'EOF' | kubectl apply -f - -apiVersion: v1 -kind: Pod -metadata: - name: vllm-pod - labels: - app: afd-recipe - role: serve -spec: - restartPolicy: Never - securityContext: - fsGroup: ${FS_GROUP} - volumes: - - name: model-storage - persistentVolumeClaim: - claimName: ${PVC_NAME} - - name: dshm - emptyDir: - medium: Memory - sizeLimit: 16Gi - - name: work - emptyDir: {} - - name: recipe-script - configMap: - name: afd-recipe-script - containers: - - name: afd - image: ${TEMPLATE_IMAGE} - imagePullPolicy: Always - command: - - /bin/bash - - -c - - | - set -euo pipefail - RECIPE_SCRIPT=/recipe/recipe.sh - mkdir -p /work/bin - # Resolve the real uv BEFORE putting the shim on PATH (afterwards - # `command -v uv` would find the shim and recurse). `|| true` keeps a - # missing uv from killing the container on line 1: the shim exists - # only to strip `uv run`, which needs no uv at all. - REAL_UV="$(command -v uv || true)" - printf '#!/bin/bash\nif [ "$1" = "run" ]; then\n shift\n exec "$@"\nfi\nREAL_UV="%s"\n[ -n "$REAL_UV" ] || { echo "uv not found in image" >&2; exit 127; }\nexec "$REAL_UV" "$@"\n' "$REAL_UV" > /work/bin/uv - chmod +x /work/bin/uv - export PATH="/work/bin:$PATH" - - echo "=== launching $(basename "$RECIPE_SCRIPT") recipe ===" - - RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" - PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" - awk ' - { line[NR] = $0 } - END { - for (i = 1; i <= NR; i++) { - l = line[i] - if (l ~ /--host 127\.0\.0\.1/ && i < NR && line[i+1] ~ /--port ${TEMPLATE_CLIENT_PORT}/) { - gsub(/127\.0\.0\.1/, "0.0.0.0", l) - } - print l - } - }' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" - - cd /work - bash "$PATCHED_SCRIPT" & - RECIPE_PID=$! - - LOGS="$(grep -oE '>[[:space:]]*[A-Za-z0-9_]+\.log' "$RECIPE_SCRIPT" \ - | grep -oE '[A-Za-z0-9_]+\.log' | sort -u | tr '\n' ' ')" - - fail() { - echo "ERROR: $*" - for log in $LOGS; do - echo "----- tail $log -----" - tail -n 100 "/work/$log" 2>/dev/null || echo "(missing)" - done - sleep infinity - } - - if [ -z "$LOGS" ]; then - fail "no '> .log' redirects in $RECIPE_SCRIPT; cannot tell which processes to wait for" - fi - - echo "=== wait for every worker to report startup ===" - ready_marker() { - case "$1" in - ffn.log) echo "AFD FFN EngineCore started" ;; - *) echo "Application startup complete" ;; - esac - } - for log in $LOGS; do - marker="$(ready_marker "$log")" - echo "--- waiting on $log (marker: $marker) ---" - ready=0 - for i in $(seq 1 240); do - if [ -f "/work/$log" ] && grep -q "$marker" "/work/$log"; then - echo "$log: ready" - ready=1 - break - fi - if ! kill -0 "$RECIPE_PID" 2>/dev/null; then - fail "recipe process exited before $log came up" - fi - sleep 10 - done - [ "$ready" = "1" ] || fail "timed out waiting for $log" - done - - echo "--- waiting on attention server /health (127.0.0.1:${TEMPLATE_CLIENT_PORT}) ---" - endpoint_ready=0 - for i in $(seq 1 60); do - if curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1; then - echo "attention server: ready" - endpoint_ready=1 - break - fi - sleep 5 - done - [ "$endpoint_ready" = "1" ] || fail "attention server did not answer /health" - - echo "=== stack READY; drive load at 127.0.0.1:${TEMPLATE_CLIENT_PORT} ===" - sleep infinity - env: - - name: USER - value: "vllm" - - name: HOME - value: "/work/home" - - name: MODEL_PATH - value: "${TEMPLATE_MODEL}" - - name: HF_HOME - value: "/models/.hf_home" - - name: XDG_CACHE_HOME - value: "/work/xdg" - - name: TORCHINDUCTOR_CACHE_DIR - value: "/work/inductor" - - name: TRITON_CACHE_DIR - value: "/work/triton" - - name: VLLM_CACHE_ROOT - value: "/work/vllm" - - name: UV_CACHE_DIR - value: "/work/uv" - - name: VLLM_LOGGING_LEVEL - value: "INFO" - - name: VLLM_USE_V2_MODEL_RUNNER - value: "${VLLM_USE_V2_MODEL_RUNNER}" - - name: HF_TOKEN - valueFrom: - secretKeyRef: - name: hf-token-secret - key: token - resources: - requests: - nvidia.com/gpu: "${GPU_COUNT}" - cpu: "16" - memory: 128Gi - limits: - nvidia.com/gpu: "${GPU_COUNT}" - cpu: "32" - memory: 200Gi - volumeMounts: - - name: model-storage - mountPath: /models - - name: dshm - mountPath: /dev/shm - - name: work - mountPath: /work - - name: recipe-script - mountPath: /recipe - readOnly: true -EOF -``` - -**4d. Service** (idempotent): - -```bash -kubectl apply -f - </dev/null)" = "Running" ]; do - phase="$(kubectl get pod vllm-pod -o jsonpath='{.status.phase}' 2>/dev/null || true)" - if [ "$phase" = "Failed" ]; then - echo "pod Failed"; kubectl logs vllm-pod --tail=50; exit 1 - fi - if [ "$(date +%s)" -ge "$deadline" ]; then - echo "timed out after 15m waiting for Running (phase=${phase:-})" - kubectl describe pod vllm-pod | tail -40 - exit 1 - fi - sleep 10 -done -``` - -**4f. Wait for the stack to report ready.** Model download and graph capture -dominate here; a cold PVC on a large model can legitimately take a while, so -this budget is generous: - -```bash -deadline=$(( $(date +%s) + 3600 )) -while true; do - pod_logs="$(kubectl logs pod/vllm-pod 2>/dev/null || true)" - echo "$pod_logs" | grep -q "stack READY" && break - if echo "$pod_logs" | grep -q "ERROR:"; then - echo "serve pod hit an error during startup"; kubectl logs vllm-pod --tail=200; exit 1 - fi - if [ "$(kubectl get pod vllm-pod -o jsonpath='{.status.phase}' 2>/dev/null)" = "Failed" ]; then - echo "pod Failed"; kubectl logs vllm-pod --tail=200; exit 1 - fi - if [ "$(date +%s)" -ge "$deadline" ]; then - echo "timed out after 60m waiting for 'stack READY'" - kubectl logs vllm-pod --tail=200; exit 1 - fi - sleep 10 -done -echo "=== serve pod ready at http://vllm-service:${CLIENT_PORT} ===" -``` +Load these in order; each covers one phase and nothing else. `deploy.md` +hands off to `services.md` partway through (its own step 4d) and back -- +Services must exist **before** any Pod is deployed (step 4e), since a +role's head Pod binds its own rendezvous/DP-RPC server to its Service's +name within seconds of starting, and `restartPolicy: Never` means a bind +that fails because the Service doesn't exist yet never retries: -### 4-placed. Placement-plan-driven variant - -Only applies when step 1 identified a multi-node-capable script -- use the -placement plan and per-role rank ordering worked out in step 1b. Step 4a -(the model PVC) is shared -- every Pod in the plan mounts the same PVC -- -but per step 4a's note it must be provisioned `ReadWriteMany` whenever the -plan has more than one Pod. Step 4b (the recipe ConfigMap) is also shared -as-is: the same script is mounted into every Pod regardless of how many the -plan has, since it already reads its rank counts and AFD host from env -vars -- no mutation needed. Everything below replaces steps 4c-4f. - -**Look up `fsGroup`**, same as step 4c: - -```bash -FS_GROUP="$(kubectl get ns "$(kubectl config view --minify -o jsonpath='{..namespace}')" \ - -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' 2>/dev/null \ - | cut -d/ -f1)" -FS_GROUP="${FS_GROUP:-1000}" -echo "using fsGroup=${FS_GROUP}" -``` - -**Resolve the FFN host value** (skip if the plan has only one Pod -- step 1b -already said to leave `AFD_CONNECTOR_HOST` at its loopback default). Per -`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`, `host` must resolve to FFN's -first rank, and every participating rank (including attention) must agree -on the same value: - -```bash -AFD_CONNECTOR_HOST="vllm-ffn-p2p-service" # short form resolves within the namespace; short-circuit to 127.0.0.1 for a single-Pod plan -``` - -This is passed as the `AFD_CONNECTOR_HOST` container env var to *every* Pod -in the plan below, including Pods carrying no FFN ranks themselves -- no -script mutation needed, since the recipe already reads `host` from that env -var. - -**One Pod per plan entry, each requesting only its own share of GPUs.** Same -container script body, env, and `uv` shim as step 4c, but driven by two -independent role positions (`attn_*` / `ffn_*`) per Pod instead of one -combined role, since a single Pod can now carry a share of either role, -both, or (in a one-Pod-holds-everything plan) all of both: - -```bash -# delete every Pod name that appears in the placement plan before redeploying -kubectl delete pod ... --ignore-not-found - -# attn_start_rank/attn_headless/attn_dp_address and their ffn_ counterparts -# describe this Pod's position within each role's DP group -- defaults -# (0, 0, 127.0.0.1) mean "this Pod is the whole role" or "this Pod is that -# role's head". Only a role actually split across more than one Pod (step -# 1b) needs non-default values for that role. -deploy_pod() { - local pod_name="$1" attn_ranks="$2" ffn_ranks="$3" \ - attn_start_rank="${4:-0}" attn_headless="${5:-0}" attn_dp_address="${6:-127.0.0.1}" \ - ffn_start_rank="${7:-0}" ffn_headless="${8:-0}" ffn_dp_address="${9:-127.0.0.1}" - local gpu_n=$((attn_ranks + ffn_ranks)) - local attn_node_role="none" ffn_node_role="none" - [ "$attn_ranks" -gt 0 ] && attn_node_role=$([ "$attn_headless" = "1" ] && echo worker || echo head) - [ "$ffn_ranks" -gt 0 ] && ffn_node_role=$([ "$ffn_headless" = "1" ] && echo worker || echo head) - - TEMPLATE_IMAGE="$AFD_PLUGIN_IMAGE" TEMPLATE_MODEL="$MODEL_ID" TEMPLATE_POD="$pod_name" \ - TEMPLATE_ATTN_RANKS="$attn_ranks" TEMPLATE_FFN_RANKS="$ffn_ranks" \ - TEMPLATE_ATTN_START_RANK="$attn_start_rank" TEMPLATE_ATTN_HEADLESS="$attn_headless" TEMPLATE_ATTN_DP_ADDRESS="$attn_dp_address" \ - TEMPLATE_FFN_START_RANK="$ffn_start_rank" TEMPLATE_FFN_HEADLESS="$ffn_headless" TEMPLATE_FFN_DP_ADDRESS="$ffn_dp_address" \ - TEMPLATE_ATTN_NODE_ROLE="$attn_node_role" TEMPLATE_FFN_NODE_ROLE="$ffn_node_role" \ - TEMPLATE_GPU="$gpu_n" TEMPLATE_PVC="$PVC_NAME" \ - TEMPLATE_FSGROUP="$FS_GROUP" TEMPLATE_CLIENT_PORT="$CLIENT_PORT" \ - TEMPLATE_AFD_PORT="${AFD_CONNECTOR_PORT:-}" TEMPLATE_AFD_HOST="${AFD_CONNECTOR_HOST:-127.0.0.1}" \ - envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_ATTN_RANKS} ${TEMPLATE_FFN_RANKS} ${TEMPLATE_ATTN_START_RANK} ${TEMPLATE_ATTN_HEADLESS} ${TEMPLATE_ATTN_DP_ADDRESS} ${TEMPLATE_FFN_START_RANK} ${TEMPLATE_FFN_HEADLESS} ${TEMPLATE_FFN_DP_ADDRESS} ${TEMPLATE_ATTN_NODE_ROLE} ${TEMPLATE_FFN_NODE_ROLE} ${TEMPLATE_GPU} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_AFD_PORT} ${TEMPLATE_AFD_HOST} ${VLLM_USE_V2_MODEL_RUNNER}' <<'EOF' | kubectl apply -f - -apiVersion: v1 -kind: Pod -metadata: - name: ${TEMPLATE_POD} - labels: - app: afd-recipe - role: serve - afd-attn-node-role: ${TEMPLATE_ATTN_NODE_ROLE} - afd-ffn-node-role: ${TEMPLATE_FFN_NODE_ROLE} -spec: - restartPolicy: Never - securityContext: - fsGroup: ${TEMPLATE_FSGROUP} - volumes: - - name: model-storage - persistentVolumeClaim: - claimName: ${TEMPLATE_PVC} - - name: dshm - emptyDir: {medium: Memory, sizeLimit: 16Gi} - - name: work - emptyDir: {} - - name: recipe-script - configMap: - name: afd-recipe-script - containers: - - name: afd - image: ${TEMPLATE_IMAGE} - imagePullPolicy: Always - command: - - /bin/bash - - -c - - | - set -euo pipefail - RECIPE_SCRIPT=/recipe/recipe.sh - mkdir -p /work/bin - REAL_UV="$(command -v uv || true)" - printf '#!/bin/bash\nif [ "$1" = "run" ]; then\n shift\n exec "$@"\nfi\nREAL_UV="%s"\n[ -n "$REAL_UV" ] || { echo "uv not found in image" >&2; exit 127; }\nexec "$REAL_UV" "$@"\n' "$REAL_UV" > /work/bin/uv - chmod +x /work/bin/uv - export PATH="/work/bin:$PATH" - - echo "=== launching $(basename "$RECIPE_SCRIPT") recipe (attn_ranks=${TEMPLATE_ATTN_RANKS} ffn_ranks=${TEMPLATE_FFN_RANKS}) ===" - - RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" - PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" - awk ' - { line[NR] = $0 } - END { - for (i = 1; i <= NR; i++) { - l = line[i] - if (l ~ /--host 127\.0\.0\.1/ && i < NR && line[i+1] ~ /--port ${TEMPLATE_CLIENT_PORT}/) { - gsub(/127\.0\.0\.1/, "0.0.0.0", l) - } - print l - } - }' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" - - cd /work - bash "$PATCHED_SCRIPT" & - RECIPE_PID=$! - - LOGS="" - [ "${TEMPLATE_ATTN_RANKS}" -gt 0 ] && LOGS="$LOGS attn.log" - [ "${TEMPLATE_FFN_RANKS}" -gt 0 ] && LOGS="$LOGS ffn.log" - - fail() { - echo "ERROR: $*" - for log in $LOGS; do - echo "----- tail $log -----" - tail -n 100 "/work/$log" 2>/dev/null || echo "(missing)" - done - sleep infinity - } - - [ -n "$LOGS" ] || fail "this pod's attn_ranks/ffn_ranks are both 0 -- nothing to run" - - # A headless pod (worker shard of a role split across multiple - # Pods, per --headless) never starts an API server, so it never - # logs "Application startup complete" -- fall back to the - # EngineCore's own init-complete line, which fires regardless of - # headless status (unverified marker; not live-validated). - ready_marker() { - case "$1" in - ffn.log) echo "AFD FFN EngineCore started" ;; - attn.log) - if [ "${TEMPLATE_ATTN_HEADLESS}" = "1" ]; then - echo "init engine (profile, create kv cache, warmup model) took" - else - echo "Application startup complete" - fi - ;; - *) echo "Application startup complete" ;; - esac - } - for log in $LOGS; do - marker="$(ready_marker "$log")" - echo "--- waiting on $log (marker: $marker) ---" - ready=0 - for i in $(seq 1 240); do - if [ -f "/work/$log" ] && grep -q "$marker" "/work/$log"; then - ready=1; break - fi - kill -0 "$RECIPE_PID" 2>/dev/null || fail "recipe process exited before $log came up" - sleep 10 - done - [ "$ready" = "1" ] || fail "timed out waiting for $log" - done - - if [ "${TEMPLATE_ATTN_RANKS}" -gt 0 ] && [ "${TEMPLATE_ATTN_HEADLESS}" != "1" ]; then - echo "--- waiting on attention server /health (127.0.0.1:${TEMPLATE_CLIENT_PORT}) ---" - endpoint_ready=0 - for i in $(seq 1 60); do - curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1 && { endpoint_ready=1; break; } - sleep 5 - done - [ "$endpoint_ready" = "1" ] || fail "attention server did not answer /health" - fi - - echo "=== pod ${TEMPLATE_POD} READY (attn_ranks=${TEMPLATE_ATTN_RANKS} ffn_ranks=${TEMPLATE_FFN_RANKS}) ===" - sleep infinity - env: - - {name: USER, value: "vllm"} - - {name: HOME, value: "/work/home"} - - {name: MODEL_PATH, value: "${TEMPLATE_MODEL}"} - - {name: HF_HOME, value: "/models/.hf_home"} - - {name: XDG_CACHE_HOME, value: "/work/xdg"} - - {name: TORCHINDUCTOR_CACHE_DIR, value: "/work/inductor"} - - {name: TRITON_CACHE_DIR, value: "/work/triton"} - - {name: VLLM_CACHE_ROOT, value: "/work/vllm"} - - {name: UV_CACHE_DIR, value: "/work/uv"} - - {name: VLLM_LOGGING_LEVEL, value: "INFO"} - - {name: VLLM_USE_V2_MODEL_RUNNER, value: "${VLLM_USE_V2_MODEL_RUNNER}"} - - {name: ATTENTION_DP_RANKS, value: "${TEMPLATE_ATTN_RANKS}"} - - {name: FFN_DP_RANKS, value: "${TEMPLATE_FFN_RANKS}"} - - {name: ATTENTION_DP_START_RANK, value: "${TEMPLATE_ATTN_START_RANK}"} - - {name: FFN_DP_START_RANK, value: "${TEMPLATE_FFN_START_RANK}"} - - {name: ATTENTION_HEADLESS, value: "${TEMPLATE_ATTN_HEADLESS}"} - - {name: FFN_HEADLESS, value: "${TEMPLATE_FFN_HEADLESS}"} - - {name: ATTENTION_DP_ADDRESS, value: "${TEMPLATE_ATTN_DP_ADDRESS}"} - - {name: FFN_DP_ADDRESS, value: "${TEMPLATE_FFN_DP_ADDRESS}"} - - {name: AFD_CONNECTOR_PORT, value: "${TEMPLATE_AFD_PORT}"} - - {name: AFD_CONNECTOR_HOST, value: "${TEMPLATE_AFD_HOST}"} - - name: HF_TOKEN - valueFrom: - secretKeyRef: {name: hf-token-secret, key: token} - resources: - requests: {nvidia.com/gpu: "${TEMPLATE_GPU}", cpu: "16", memory: 128Gi} - limits: {nvidia.com/gpu: "${TEMPLATE_GPU}", cpu: "32", memory: 200Gi} - volumeMounts: - - {name: model-storage, mountPath: /models} - - {name: dshm, mountPath: /dev/shm} - - {name: work, mountPath: /work} - - {name: recipe-script, mountPath: /recipe, readOnly: true} -EOF -} -``` - -Call `deploy_pod` once per entry in the placement plan. One Pod holding -every rank of a 4-attention/2-ffn recipe: - -```bash -deploy_pod vllm-pod-0 4 2 0 0 127.0.0.1 0 0 127.0.0.1 -``` - -A dedicated Pod per role, attention split 2-and-2 across two Pods, FFN whole -in a third: - -```bash -deploy_pod vllm-attn-pod-0 2 0 0 0 vllm-attn-dp-service 0 0 127.0.0.1 -deploy_pod vllm-attn-pod-1 2 0 2 1 vllm-attn-dp-service 0 0 127.0.0.1 -deploy_pod vllm-ffn-pod 0 2 0 0 127.0.0.1 0 0 127.0.0.1 -``` - -`vllm-attn-pod-1`'s `attn_start_rank=2` because `vllm-attn-pod-0` already -claimed local ranks `[0, 2)`; `vllm-attn-dp-service` is the DP-coordination -Service (below) exposing `vllm-attn-pod-0`'s DP-RPC port. - -`vllm-attn-pod-0`'s own `_DP_ADDRESS` matters here, and it must be the -**same headless Service name** the worker connects through (`vllm-attn-dp-service`), -not `127.0.0.1` and not `0.0.0.0`. Two separate mechanisms consume this one -value on the head, with conflicting requirements: - -- vLLM's `DPCoordinator` uses `--data-parallel-address` as the literal ZMQ - **bind** address. `127.0.0.1` binds loopback-only and the worker's - connection hangs until it times out -- this needs `0.0.0.0` or a real, - non-loopback address. -- Separately, when the recipe passes `--enable-expert-parallel`, every - rank -- including the worker's, on the other Pod -- calls - `ParallelConfig.stateless_init_dp_group()` (`vllm/config/parallel.py`), - which reuses this same address as the **connect target** for a second, - independent rendezvous. `0.0.0.0` satisfies the bind case but is not a - dialable address, so the worker's ranks hang trying to connect to it and - the whole DP group fails after that rendezvous's timeout. - -Because a headless Service with a selector matching exactly one Pod -resolves to that Pod's real IP from anywhere in the cluster -- including -from the Pod itself -- pointing the head's own `_DP_ADDRESS` at -`vllm-attn-dp-service` satisfies both: it binds to the head's real -interface (not loopback) and is dialable by the worker (not the wildcard). -This only applies to a head Pod whose role is split (`_headless=0` but -another Pod shares the same role) -- a Pod that is whole in one Pod (like -`vllm-ffn-pod` above, or the single-Pod example before it) keeps -`127.0.0.1` since nothing external ever needs to reach it, and there's no -second Pod for `stateless_init_dp_group` to hang against. Two Pods mixing a -partial share of both roles (e.g. `2 attention + 1 ffn` each, for a -4-attention/2-ffn recipe): - -```bash -deploy_pod vllm-mixed-pod-0 2 1 0 0 vllm-attn-dp-service 0 0 vllm-ffn-dp-service -deploy_pod vllm-mixed-pod-1 2 1 2 1 vllm-attn-dp-service 1 1 vllm-ffn-dp-service -``` - -Both of `vllm-mixed-pod-0`'s `_DP_ADDRESS` values point at the matching -headless Service (its own head address for each split role) rather than -`0.0.0.0` -- same bind-and-connect reasoning as above, applied to each role -independently. - -Splitting a role across Pods this way additionally requires the recipe to -expose the `_START_RANK`/`_HEADLESS`/`_DP_ADDRESS` env vars noted in step 1b --- confirm that before proposing such a plan. - -`AFD_CONNECTOR_PORT` above defaults to an empty string when the caller -didn't override it -- an empty env value still falls through to the -script's own `${AFD_CONNECTOR_PORT:-}` default, so it's safe to always -pass the container env entry. - -**Forced cross-node placement, only if the caller explicitly asked for it** -(step 1b defaults to leaving node placement to the scheduler). Add this -label to every Pod's `metadata.labels` in the manifest above, and this -`affinity` block to every Pod's `spec`: - -```yaml - labels: - afd-multinode-group: afd-multinode - spec: - affinity: - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - {key: afd-multinode-group, operator: In, values: ["afd-multinode"]} - topologyKey: kubernetes.io/hostname -``` - -This is the same experimental caveat already noted in Scope for cross-Pod -`P2pNcclAFDConnector` use: any plan beyond a single Pod deploys correctly by -construction (correct AFD/DP flags, correct port exposure per -`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`), but has not been validated -end-to-end on a live cluster in this repo. Treat it as exploratory and say -so when reporting results. - -**Services.** `vllm-service` keeps its existing shape and contract (client- -facing, port `${CLIENT_PORT}`) but selects on `afd-attn-node-role: head` -instead of the generic `role: serve` -- this resolves to whichever Pod holds -attention rank 0, whether that Pod is dedicated to attention or also carries -FFN ranks (including the single-Pod-holds-everything plan, where the one -Pod is trivially both roles' head): - -```bash -kubectl apply -f - < ... -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}' -``` - -Every node matters now: whichever Pod triggered the PVC's first -`ReadWriteMany`/`ReadWriteOnce` bind constrains where a follow-up pod can -attach from. - -### 6. Teardown - -The pod(s) and Service(s) are **left running by design**, so weights stay -warm. Report what is still up and how to remove it; do not delete unless -asked: - -For a plain single-node script: - -```bash -kubectl delete pod vllm-pod -kubectl delete svc vllm-service -``` - -For a multi-node-capable script deployed via a placement plan, list every -Pod name from the plan plus whichever Services were actually created -(`vllm-service` always; `vllm-ffn-p2p-service` if the plan had more than one -Pod; `vllm-attn-dp-service`/`vllm-ffn-dp-service` if that role was split): - -```bash -kubectl delete pod ... -kubectl delete svc vllm-service vllm-ffn-p2p-service vllm-attn-dp-service vllm-ffn-dp-service -``` - -The model PVC is intentionally not listed -- deleting it discards the warm -cache and forces a full re-download. - -## Caveats - -- Redeploying a different recipe means re-running steps 4b-4f (or the - placement-plan-driven equivalent); the pod(s) are replaced, not - reconfigured. Two recipes cannot share a pod or its GPUs. -- Startup markers are keyed on log filename (`ffn.log` -> `AFD FFN - EngineCore started`, everything else -> `Application startup complete`, - or -- for a headless attention Pod sharding a split role -- `init engine - (profile, create kv cache, warmup model) took`). A recipe using different - log names gets the generic marker. -- A placement plan requires a recipe written in the parameterized style - (`ATTENTION_DP_RANKS`/`FFN_DP_RANKS` gating + an `AFD_CONNECTOR_HOST` - env-var default for the AFD `host` field, e.g. the - `deepseek_v2_lite/prefill_decode_colocation/*.sh` recipes). Older fixed - two-block recipes (both roles backgrounded unconditionally, literal - `CUDA_VISIBLE_DEVICES`, hardcoded `host`) aren't multi-node-capable at all - -- they only support the plain step 4c/4d/4e/4f path, one Pod, no - placement question. Don't attempt to place one of those across more than - one Pod. -- Any placement plan beyond a single Pod holding every rank is - **experimental**: it deploys correctly (correct port exposure per - `docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`, correct AFD/DP flags per - Pod), but upstream has not validated `P2pNcclAFDConnector` correctness or - performance across Pods -- whether or not those Pods land on the same - node. NCCL runs over whatever the cluster's pod network provides -- no - RDMA is assumed. Treat throughput numbers from any multi-Pod deployment as - exploratory, and say so when reporting results. -- Splitting a single role across more than one Pod additionally requires - the recipe to expose `_START_RANK`/`_HEADLESS`/`_DP_ADDRESS` env vars - (grep for `_START_RANK`/`_HEADLESS`; a script only gated on - `ATTENTION_DP_RANKS`/`FFN_DP_RANKS`/`AFD_CONNECTOR_HOST` supports placing - each role in its own Pod, or colocating both roles in one Pod, but not - finer sharding of a single role). This is source-backed against vLLM's - native multi-node DP flags (`--data-parallel-size-local`, - `--data-parallel-start-rank`, `--data-parallel-address`, - `--data-parallel-rpc-port`, `--headless` -- confirmed present in - `vllm/entrypoints/openai/cli_args.py` and `vllm/engine/arg_utils.py`) but - has not been live-validated on a cluster -- in particular the - headless-attention readiness marker above is an assumption, not a - confirmed observation. Flag this explicitly if asked to deploy it. +| Reference | Covers | +|---|---| +| [resolve-recipe.md](references/resolve-recipe.md) | Step 1: read the recipe's serve-block fields and model id; read its placement (or lack of one) from its header comment -- never ask. | +| [deploy.md](references/deploy.md) | Steps 2-4c: GPU_COUNT, confirm, PVC, ConfigMap, fsGroup. | +| [services.md](references/services.md) | Step 4d: which of the four Services to create, derived from the parsed plan -- before any Pod. | +| [deploy.md](references/deploy.md) (resume) | Steps 4e-4f: pod(s) via `templates/pod.yaml`, wait for Running and for each pod's own readiness line. | +| [report-and-teardown.md](references/report-and-teardown.md) | Step 5: report the endpoint/model/nodes. Step 6: teardown (pods/Services only -- never the PVC unless asked). | + +All k8s manifests live in `templates/*.yaml`, applied via `envsubst | kubectl +apply -f -`; no manifest is inlined in these docs. diff --git a/.agents/skills/deploy-afd-k8s/references/deploy.md b/.agents/skills/deploy-afd-k8s/references/deploy.md new file mode 100644 index 000000000..3a27501e7 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/references/deploy.md @@ -0,0 +1,211 @@ +# 2-4. Compute GPU_COUNT, confirm, and deploy + +## 2. GPU_COUNT + +Plain single-pod recipe (no plan from [resolve-recipe.md](resolve-recipe.md)): +union of every literal `CUDA_VISIBLE_DEVICES=` in the script. + +```bash +GPU_COUNT="$(grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' "$RECIPE_SCRIPT_PATH" \ + | cut -d= -f2 | tr ',' '\n' | sort -u | wc -l)" +``` + +Per plan entry: count the devices in the literal `CUDA_VISIBLE_DEVICES=` inside +the recipe's `if` block that tests this pod's role (`ATTENTION` or `FFN` -- +whichever prefix `POD` starts with). Every pod sharing a role shares that +literal, so this is a plain text extraction, not a per-pod computation: + +```bash +role_block() { # $1=script $2=ATTENTION|FFN + awk -v role="$2" ' + /if.*\$POD/ && $0 ~ role { inblock=1; next } + inblock && /^fi/ { inblock=0 } + inblock { print } + ' "$1" +} +devices="$(role_block "$RECIPE_SCRIPT_PATH" "$ROLE" \ + | grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' | head -1 | cut -d= -f2)" +GPU_COUNT=$(( $(grep -o ',' <<<"$devices" | wc -l) + 1 )) +``` + +## 3. Confirm before touching the cluster + +State and confirm: target cluster/namespace, `AFD_PLUGIN_IMAGE`, `MODEL_ID`, +`PVC_NAME`, `CLIENT_PORT`, `RECIPE_SCRIPT_PATH`, and either `GPU_COUNT` +(plain recipe) or the plan as a table of `pod_name | role | GPU_COUNT` +(placement-driven recipe, straight from resolve-recipe.md -- no shape to +negotiate, just restate what was parsed). Flag that step 4 **deletes any +existing Pod(s)** with the same name(s) about to be deployed. + +```bash +AFD_PLUGIN_IMAGE= +MODEL_ID= +PVC_NAME= +CLIENT_PORT= # from the script's --port; default 18305 +RECIPE_SCRIPT_PATH= +VLLM_USE_V2_MODEL_RUNNER=${VLLM_USE_V2_MODEL_RUNNER:-0} +``` + +**Node placement is left to the scheduler by default.** Only add +`podAntiAffinity` forcing Pods onto separate nodes if the caller explicitly +asks to exercise cross-node placement -- add this to every Pod's +`metadata.labels` and `spec` in `templates/pod.yaml` before applying: + +```yaml + labels: + afd-multinode-group: afd-multinode + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - {key: afd-multinode-group, operator: In, values: ["afd-multinode"]} + topologyKey: kubernetes.io/hostname +``` + +## 4a. Model PVC + +Size for the model (e.g. Qwen3.5-122B-A10B-FP8 needs `200Gi` minimum). An +existing undersized PVC is reused silently and fails mid-download, so set +`MODEL_PVC_SIZE` deliberately. `PVC_ACCESS_MODE` is derived, not asked: +`ReadWriteMany` whenever the plan (if any) has more than one pod -- an RWO +volume can only attach from one node, and the second Pod deadlocks +`Pending` with `Multi-Attach error` otherwise. + +```bash +MODEL_PVC_SIZE=${MODEL_PVC_SIZE:-100Gi} +STORAGE_CLASS=${STORAGE_CLASS:-} +PVC_ACCESS_MODE=ReadWriteOnce # ReadWriteMany if the plan has >1 pod + +if kubectl get pvc "${PVC_NAME}" >/dev/null 2>&1; then + echo "PVC ${PVC_NAME} exists; reusing its warm HF_HOME cache for ${MODEL_ID}" + echo "NOTE: verify its size fits ${MODEL_ID}, and its accessModes include" + echo " ReadWriteMany if this plan has >1 pod -- an RWO reuse deadlocks the 2nd pod" +else + envsubst '${PVC_NAME} ${MODEL_PVC_SIZE} ${STORAGE_CLASS} ${PVC_ACCESS_MODE}' \ + < templates/pvc.yaml | grep -v 'storageClassName: *$' | kubectl apply -f - +fi +``` + +If an existing single-pod RWO PVC already holds downloaded weights you want +to reuse for a multi-pod plan, provision a new RWX PVC and `cp -a` the data +across (via a short-lived Pod pinned to the node that holds the RWO mount) +rather than re-downloading. + +## 4b. Recipe ConfigMap + +Lets the pod run any local recipe -- edited, uncommitted, or new -- without +rebuilding the image. Run from wherever `RECIPE_SCRIPT_PATH` resolves: + +```bash +kubectl create configmap afd-recipe-script \ + --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +## 4c. fsGroup + +Must be inside the namespace's allowed range or the pod is rejected at +admission. Look it up once, reuse for every pod: + +```bash +FS_GROUP="$(kubectl get ns "$(kubectl config view --minify -o jsonpath='{..namespace}')" \ + -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' 2>/dev/null \ + | cut -d/ -f1)" +FS_GROUP="${FS_GROUP:-1000}" +``` + +## 4d. Services + +Create every Service that applies -- see [services.md](services.md) -- +**before** deploying any Pod in 4e below. Everything a Service needs +(ports, selectors, which of the four apply) comes from the plan parsed in +resolve-recipe.md, so none of this waits on a Pod existing. Creating +Services late is the actual failure mode this guards against: a role's +head Pod binds its own rendezvous/DP-RPC server to its Service's DNS name +within seconds of starting, and with `restartPolicy: Never` a bind that +fails because the Service doesn't exist yet never gets retried. + +## 4e. Deploy pod(s) + +One call per plan entry (or a single call with no overrides for a plain +recipe) against the one shared `templates/pod.yaml`: + +```bash +deploy_pod() { + local pod_name="$1" gpu="$2" attn_role="$3" ffn_role="$4" extra_env_pairs="$5" + # extra_env_pairs: space-separated VAR=value tokens straight off this + # plan entry's header line (including POD=... itself), empty for a plain + # recipe. Build the spliced env-entries block for templates/pod.yaml. + local extra_env="" + for pair in $extra_env_pairs; do + extra_env="$extra_env - {name: ${pair%%=*}, value: \"${pair#*=}\"} +" + done + + kubectl delete pod "$pod_name" --ignore-not-found + + TEMPLATE_IMAGE="$AFD_PLUGIN_IMAGE" TEMPLATE_MODEL="$MODEL_ID" TEMPLATE_POD="$pod_name" \ + TEMPLATE_GPU="$gpu" TEMPLATE_PVC="$PVC_NAME" TEMPLATE_FSGROUP="$FS_GROUP" \ + TEMPLATE_CLIENT_PORT="$CLIENT_PORT" TEMPLATE_ATTN_ROLE="$attn_role" TEMPLATE_FFN_ROLE="$ffn_role" \ + TEMPLATE_EXTRA_ENV="$extra_env" \ + envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_GPU} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_ATTN_ROLE} ${TEMPLATE_FFN_ROLE} ${TEMPLATE_EXTRA_ENV} ${VLLM_USE_V2_MODEL_RUNNER}' \ + < templates/pod.yaml | kubectl apply -f - +} +``` + +Plain recipe, one pod, both roles head, no overrides: + +```bash +deploy_pod vllm-pod "$GPU_COUNT" head head "" +``` + +Placement-driven recipe, one call per entry parsed in +[resolve-recipe.md](resolve-recipe.md), e.g. the qwen3.5 3-pod plan: + +```bash +deploy_pod vllm-pod-attention-0 2 head none "POD=ATTENTION_0" +deploy_pod vllm-pod-attention-1 2 worker none "POD=ATTENTION_1 ATTENTION_HEADLESS=1 ATTENTION_DP_START_RANK=2" +deploy_pod vllm-pod-ffn-0 2 none head "POD=FFN_0" +``` + +## 4f. Wait for Running, then for each pod's own readiness line + +Bounded 15-minute wait per pod for `phase=Running` (unschedulable pods never +reach `Failed`, so an unbounded loop would spin forever), then poll logs for +`=== pod READY ===` (budget 60 minutes -- model download and graph +capture dominate): + +```bash +for pod_name in ; do + deadline=$(( $(date +%s) + 900 )) + until [ "$(kubectl get pod "$pod_name" -o jsonpath='{.status.phase}' 2>/dev/null)" = "Running" ]; do + phase="$(kubectl get pod "$pod_name" -o jsonpath='{.status.phase}' 2>/dev/null || true)" + [ "$phase" = "Failed" ] && { kubectl logs "$pod_name" --tail=50; exit 1; } + [ "$(date +%s)" -ge "$deadline" ] && { kubectl describe pod "$pod_name" | tail -40; exit 1; } + sleep 10 + done +done + +for pod_name in ; do + deadline=$(( $(date +%s) + 3600 )) + while true; do + logs="$(kubectl logs "pod/$pod_name" 2>/dev/null || true)" + echo "$logs" | grep -q "pod ${pod_name} READY" && break + echo "$logs" | grep -q "ERROR:" && { kubectl logs "$pod_name" --tail=200; exit 1; } + [ "$(kubectl get pod "$pod_name" -o jsonpath='{.status.phase}' 2>/dev/null)" = "Failed" ] && { kubectl logs "$pod_name" --tail=200; exit 1; } + [ "$(date +%s)" -ge "$deadline" ] && { kubectl logs "$pod_name" --tail=200; exit 1; } + sleep 10 + done +done +``` + +Only report the endpoint ready once every pod in the plan has printed its +own `READY` line -- one pod becoming ready doesn't imply the others did. + +Startup markers are keyed on the fixed log names every colocation recipe +uses today (`attn.log` -> `Application startup complete`, or, for a worker +shard, the EngineCore init-complete line; `ffn.log` -> `AFD FFN EngineCore +started`). A recipe using different log names needs `templates/pod.yaml` +updated. diff --git a/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md b/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md new file mode 100644 index 000000000..4a61922f0 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md @@ -0,0 +1,31 @@ +# 5. Report back + +Tell the caller: the endpoint `http://vllm-service:${CLIENT_PORT}`, +`MODEL_ID` (and whether it's a HF repo id or an in-container path), +`PVC_NAME`, `GPU_COUNT` (per pod, if more than one), the recipe deployed, +its `--max-num-batched-tokens`, and the node(s) it landed on: + +```bash +kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}' +``` + +Every node matters if a follow-up pod needs to attach the same +`ReadWriteOnce` PVC -- it can only attach from that node. + +If the plan spans more than one pod, repeat the experimental caveat from +[resolve-recipe.md](resolve-recipe.md): correct AFD/DP flags and port +exposure, but cross-pod `P2pNcclAFDConnector` throughput is not +upstream-validated. + +# 6. Teardown + +Left running by design, so weights stay warm -- report what's up and how to +remove it; do not delete unless asked: + +```bash +kubectl delete pod +kubectl delete svc vllm-service vllm-ffn-p2p-service vllm-attn-dp-service vllm-ffn-dp-service --ignore-not-found +``` + +The model PVC is intentionally not listed -- deleting it discards the warm +cache and forces a full re-download. diff --git a/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md new file mode 100644 index 000000000..184cac6e1 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md @@ -0,0 +1,79 @@ +# 1. Resolve the recipe, its model, and its placement + +Take the recipe path from the user, or ask. It must be a colocation recipe +under `recipe/gpu/P2pNcclAFDConnector/**/prefill_decode_colocation/` (GPU +only; prefill/decode disaggregation recipes are out of scope -- see +`SKILL.md` Scope). + +Read the script and note, per `vllm serve` block (each backgrounded process +ending `> .log 2>&1 &`): + +- `CUDA_VISIBLE_DEVICES`, `--data-parallel-size`, `--tensor-parallel-size` +- the `"afd": {"role": ...}` block in `--additional-config` +- eager vs graph, and every other flag (`--enable-expert-parallel`, + `--max-num-seqs`, `--max-num-batched-tokens`, `--max-model-len`, + `--trust-remote-code`, host/port) + +Record `--max-num-batched-tokens` and report it to the caller later -- it +caps how much prefill work a single step can absorb. + +Resolve `MODEL_ID`: read the script's `MODEL_PATH` default. It is either a +HF Hub id (map the recipe's model directory name to one, e.g. +`deepseek_v2_lite` -> `deepseek-ai/DeepSeek-V2-Lite`; ask if the mapping +isn't obvious) or a literal in-container path (e.g. +`/path/model_weights/Qwen3.5-122B-A10B-FP8`) when weights are staged on the +PVC. Tell the caller which form it is. + +Set `CLIENT_PORT` to the `--port` value on the attention (or sole, for a +single-pod recipe) `vllm serve` block -- step 2 must rebind and expose +whatever value is actually there. + +## Read the placement plan -- never ask + +Look at the top-of-file comment block (above the first `vllm serve`/env-var +line). If it says nothing about per-pod placement, this is a plain +single-pod recipe: one pod, `POD` unset, both roles run unconditionally -- +continue to [deploy.md](deploy.md) with no plan. + +If it does describe a placement across multiple pods, read and interpret +that description directly -- don't assume one fixed grammar. Both recipes +written so far happen to spell it the same way, one line per pod in file +order: + +``` +# For pod 1: POD=ATTENTION_0 +# For pod 2: POD=ATTENTION_1, ATTENTION_HEADLESS=1, ATTENTION_DP_START_RANK=2 +# For pod 3: POD=FFN_0 +``` + +but a future recipe may instead describe the same plan in prose (e.g. "runs +on 3 pods: the first two run ATTENTION_0 and ATTENTION_1 -- the second one +headless, starting at data-parallel rank 2 -- and the third runs FFN_0"). +Whatever the phrasing, read it and extract, per pod, in the order the +comment presents them: + +- that pod's `POD=` identity +- any further container-env overrides for that pod only (e.g. + `ATTENTION_HEADLESS=1`, `ATTENTION_DP_START_RANK=2`) + +The recipe already states ranks/ordering per pod -- there is nothing to +derive or ask about, regardless of how it's phrased. (A quick +`grep -E '^# For pod [0-9]+: '` is a fine way to spot today's convention at +a glance, but its absence doesn't mean "no plan" -- check the comment block +itself before concluding this is a plain single-pod recipe.) + +For each entry, derive (used by [deploy.md](deploy.md) and +[services.md](services.md)): + +- **k8s Pod name**: `vllm-pod-$(echo "$POD" | tr 'A-Z_' 'a-z-')`, e.g. + `vllm-pod-attention-0`. +- **`afd-attn-node-role`**: `none` unless `POD` starts with `ATTENTION_`, in + which case `worker` if that line's `ATTENTION_HEADLESS` override is `1`, + else `head`. +- **`afd-ffn-node-role`**: symmetric, `none` unless `POD` starts with `FFN_`. + +Treat any placement spanning more than one Pod as **experimental**: upstream +documents cross-node `P2pNcclAFDConnector` use as "not established by the +current recipes ... treated as unverified" +(`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`). Deploy it as written, but +don't present throughput as validated. diff --git a/.agents/skills/deploy-afd-k8s/references/services.md b/.agents/skills/deploy-afd-k8s/references/services.md new file mode 100644 index 000000000..ae59e07ad --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/references/services.md @@ -0,0 +1,64 @@ +# Services + +Create as step 4d of [deploy.md](deploy.md) -- **before** deploying any Pod +(step 4e), not after (idempotent -- safe to re-apply either way). The +headless Services below are what a role's head Pod *binds* its own +rendezvous/DP-RPC server to during its own startup, within seconds of the +container starting; if that Service doesn't exist yet, the bind/resolve +fails fast and, since every pod has `restartPolicy: Never`, it never gets a +second chance. Which of the four to create is derived from the plan parsed +in [resolve-recipe.md](resolve-recipe.md), never asked. + +## `vllm-service` -- always + +Client-facing. Selects whichever pod holds `afd-attn-node-role: head` -- +true for the sole pod of a plain recipe, and for whichever pod is the +attention head of a placement plan. + +```bash +CLIENT_PORT="$CLIENT_PORT" envsubst '${CLIENT_PORT}' \ + < templates/service-client.yaml | kubectl apply -f - +``` + +## `vllm-ffn-p2p-service` -- iff the plan has more than one pod + +Internal only, carries AFD rendezvous traffic to whichever pod holds +`afd-ffn-node-role: head`. Must exist before any multi-pod deploy, since the +recipe's own `AFD_CONNECTOR_HOST` default now points at this name directly +(fixed at the recipe level -- this skill no longer computes that value). +Exposes the control-plane port plus one derived port per FFN rank +(`[AFD_CONNECTOR_PORT, AFD_CONNECTOR_PORT + NUM_FFN_RANKS]` -- read both +constants off the recipe's own defaults/assignments): + +```bash +FFN_PORT_START="$(grep -oE 'AFD_CONNECTOR_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" +NUM_FFN_RANKS="$(grep -oE 'NUM_FFN_RANKS=[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+$' | head -1)" +PORT_ENTRIES="$(for p in $(seq "$FFN_PORT_START" "$((FFN_PORT_START + NUM_FFN_RANKS))"); do + printf ' - {name: p%s, port: %s, targetPort: %s}\n' "$p" "$p" "$p" +done)" +PORT_ENTRIES="$PORT_ENTRIES" envsubst '${PORT_ENTRIES}' \ + < templates/service-ffn-p2p.yaml | kubectl apply -f - +``` + +## `vllm-attn-dp-service` / `vllm-ffn-dp-service` -- iff that role is split + +Only when **more than one** plan entry's `POD` carries the same role (i.e. +that role is split across pods, not just placed in its own single pod). +Carries that role's DP-RPC coordination traffic to its head pod. One per +split role, generic over role via `templates/service-dp-coord.yaml`: + +```bash +TEMPLATE_ROLE=attn TEMPLATE_PORT=13345 \ + envsubst '${TEMPLATE_ROLE} ${TEMPLATE_PORT}' < templates/service-dp-coord.yaml | kubectl apply -f - +# symmetrically: TEMPLATE_ROLE=ffn TEMPLATE_PORT=13346, if FFN is ever split +``` + +All three internal Services (`vllm-ffn-p2p-service`, `vllm-attn-dp-service`, +`vllm-ffn-dp-service`) are headless (`clusterIP: None`) -- the role's head +pod *binds* its rendezvous/DP-RPC server to that name, and a normal +ClusterIP has no real interface to bind to, only to connect through. Their +templates also set `publishNotReadyAddresses: true`: headless-Service DNS +by default only lists Pods that have already passed readiness, but a head +pod resolving its own Service name to bind is inherently not-yet-ready at +that moment -- without this flag, any future readinessProbe added to +`templates/pod.yaml` would deadlock every head pod at startup. diff --git a/.agents/skills/deploy-afd-k8s/templates/pod.yaml b/.agents/skills/deploy-afd-k8s/templates/pod.yaml new file mode 100644 index 000000000..02056dd4d --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/pod.yaml @@ -0,0 +1,190 @@ +apiVersion: v1 +kind: Pod +metadata: + name: ${TEMPLATE_POD} + labels: + app: afd-recipe + role: serve + afd-attn-node-role: ${TEMPLATE_ATTN_ROLE} + afd-ffn-node-role: ${TEMPLATE_FFN_ROLE} +spec: + restartPolicy: Never + securityContext: + fsGroup: ${TEMPLATE_FSGROUP} + volumes: + - name: model-storage + persistentVolumeClaim: + claimName: ${TEMPLATE_PVC} + - name: dshm + emptyDir: + medium: Memory + sizeLimit: 16Gi + - name: work + emptyDir: {} + - name: recipe-script + configMap: + name: afd-recipe-script + containers: + - name: afd + image: ${TEMPLATE_IMAGE} + imagePullPolicy: Always + command: + - /bin/bash + - -c + - | + set -euo pipefail + RECIPE_SCRIPT=/recipe/recipe.sh + mkdir -p /work/bin + # Resolve the real uv BEFORE putting the shim on PATH (afterwards + # `command -v uv` would find the shim and recurse). + REAL_UV="$(command -v uv || true)" + printf '#!/bin/bash\nif [ "$1" = "run" ]; then\n shift\n exec "$@"\nfi\nREAL_UV="%s"\n[ -n "$REAL_UV" ] || { echo "uv not found in image" >&2; exit 127; }\nexec "$REAL_UV" "$@"\n' "$REAL_UV" > /work/bin/uv + chmod +x /work/bin/uv + export PATH="/work/bin:$PATH" + + echo "=== launching $(basename "$RECIPE_SCRIPT") (POD=${POD:-} attn=${AFD_ATTN_NODE_ROLE} ffn=${AFD_FFN_NODE_ROLE}) ===" + + # Rebind the client-facing port off loopback: the recipe binds + # :${TEMPLATE_CLIENT_PORT} on 127.0.0.1 (normally driven from the + # same host), but the Service needs 0.0.0.0. Every internal worker + # port stays on loopback. + RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" + PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" + awk ' + { line[NR] = $0 } + END { + for (i = 1; i <= NR; i++) { + l = line[i] + if (l ~ /--host 127\.0\.0\.1/ && i < NR && line[i+1] ~ /--port ${TEMPLATE_CLIENT_PORT}/) { + gsub(/127\.0\.0\.1/, "0.0.0.0", l) + } + print l + } + }' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" + + cd /work + bash "$PATCHED_SCRIPT" & + RECIPE_PID=$! + + # Which log(s) this pod will actually produce, and this pod's own + # readiness marker per log -- both keyed on this pod's node-role + # labels, not re-derived from $POD. Fixed to the attn.log/ffn.log + # names every colocation recipe uses today. + LOGS="" + [ "$AFD_ATTN_NODE_ROLE" != "none" ] && LOGS="$LOGS attn.log" + [ "$AFD_FFN_NODE_ROLE" != "none" ] && LOGS="$LOGS ffn.log" + + fail() { + echo "ERROR: $*" + for log in $LOGS; do + echo "----- tail $log -----" + tail -n 100 "/work/$log" 2>/dev/null || echo "(missing)" + done + sleep infinity + } + + [ -n "$LOGS" ] || fail "this pod's node-role labels are both 'none' -- nothing to run" + + # A worker (headless) shard of a split role never starts an API + # server, so it never logs "Application startup complete" -- fall + # back to the EngineCore's own init-complete line instead. This + # fallback marker is unverified: not yet observed on a live headless + # run, only inferred from the vLLM startup sequence. + ready_marker() { + case "$1" in + ffn.log) echo "AFD FFN EngineCore started" ;; + attn.log) + if [ "$AFD_ATTN_NODE_ROLE" = "worker" ]; then + echo "init engine (profile, create kv cache, warmup model) took" + else + echo "Application startup complete" + fi + ;; + esac + } + + echo "=== wait for every worker to report startup ===" + for log in $LOGS; do + marker="$(ready_marker "$log")" + echo "--- waiting on $log (marker: $marker) ---" + ready=0 + for i in $(seq 1 240); do + if [ -f "/work/$log" ] && grep -q "$marker" "/work/$log"; then + echo "$log: ready" + ready=1 + break + fi + kill -0 "$RECIPE_PID" 2>/dev/null || fail "recipe process exited before $log came up" + sleep 10 + done + [ "$ready" = "1" ] || fail "timed out waiting for $log" + done + + if [ "$AFD_ATTN_NODE_ROLE" = "head" ]; then + echo "--- waiting on attention server /health (127.0.0.1:${TEMPLATE_CLIENT_PORT}) ---" + endpoint_ready=0 + for i in $(seq 1 60); do + if curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1; then + echo "attention server: ready" + endpoint_ready=1 + break + fi + sleep 5 + done + [ "$endpoint_ready" = "1" ] || fail "attention server did not answer /health" + fi + + echo "=== pod ${TEMPLATE_POD} READY ===" + sleep infinity + env: + - name: USER + value: "vllm" + - name: HOME + value: "/work/home" + - name: MODEL_PATH + value: "${TEMPLATE_MODEL}" + - name: HF_HOME + value: "/models/.hf_home" + - name: XDG_CACHE_HOME + value: "/work/xdg" + - name: TORCHINDUCTOR_CACHE_DIR + value: "/work/inductor" + - name: TRITON_CACHE_DIR + value: "/work/triton" + - name: VLLM_CACHE_ROOT + value: "/work/vllm" + - name: UV_CACHE_DIR + value: "/work/uv" + - name: VLLM_LOGGING_LEVEL + value: "INFO" + - name: VLLM_USE_V2_MODEL_RUNNER + value: "${VLLM_USE_V2_MODEL_RUNNER}" + - name: AFD_ATTN_NODE_ROLE + value: "${TEMPLATE_ATTN_ROLE}" + - name: AFD_FFN_NODE_ROLE + value: "${TEMPLATE_FFN_ROLE}" +${TEMPLATE_EXTRA_ENV} + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: token + resources: + requests: + nvidia.com/gpu: "${TEMPLATE_GPU}" + cpu: "16" + memory: 128Gi + limits: + nvidia.com/gpu: "${TEMPLATE_GPU}" + cpu: "32" + memory: 200Gi + volumeMounts: + - name: model-storage + mountPath: /models + - name: dshm + mountPath: /dev/shm + - name: work + mountPath: /work + - name: recipe-script + mountPath: /recipe + readOnly: true diff --git a/.agents/skills/deploy-afd-k8s/templates/pvc.yaml b/.agents/skills/deploy-afd-k8s/templates/pvc.yaml new file mode 100644 index 000000000..19d28d4ea --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/pvc.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ${PVC_NAME} + labels: + app: afd-recipe +spec: + accessModes: + - ${PVC_ACCESS_MODE} + storageClassName: ${STORAGE_CLASS} + resources: + requests: + storage: ${MODEL_PVC_SIZE} diff --git a/.agents/skills/deploy-afd-k8s/templates/service-client.yaml b/.agents/skills/deploy-afd-k8s/templates/service-client.yaml new file mode 100644 index 000000000..9f32260c0 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/service-client.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Service +metadata: + name: vllm-service + labels: + app: afd-recipe + role: serve +spec: + selector: + app: afd-recipe + afd-attn-node-role: head + ports: + - name: http + port: ${CLIENT_PORT} + targetPort: ${CLIENT_PORT} diff --git a/.agents/skills/deploy-afd-k8s/templates/service-dp-coord.yaml b/.agents/skills/deploy-afd-k8s/templates/service-dp-coord.yaml new file mode 100644 index 000000000..c68f907a6 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/service-dp-coord.yaml @@ -0,0 +1,23 @@ +apiVersion: v1 +kind: Service +metadata: + name: vllm-${TEMPLATE_ROLE}-dp-service + labels: + app: afd-recipe + role: serve +spec: + # Headless for the same reason as vllm-ffn-p2p-service: the role's head + # pod binds --data-parallel-address to this name, so it must resolve to a + # real pod IP, not a virtual ClusterIP. + clusterIP: None + # The head pod resolves/binds this name during its own startup, before it + # can be "ready" -- without this, headless DNS would omit it until ready, + # deadlocking startup. + publishNotReadyAddresses: true + selector: + app: afd-recipe + afd-${TEMPLATE_ROLE}-node-role: head + ports: + - name: dp-rpc + port: ${TEMPLATE_PORT} + targetPort: ${TEMPLATE_PORT} diff --git a/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml b/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml new file mode 100644 index 000000000..66d5581b9 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml @@ -0,0 +1,20 @@ +apiVersion: v1 +kind: Service +metadata: + name: vllm-ffn-p2p-service + labels: + app: afd-recipe + role: serve +spec: + # Must be headless: the FFN-head pod binds its own rendezvous server to + # this name, and a normal ClusterIP has no real interface to bind to. + clusterIP: None + # The FFN-head pod resolves/binds this name during its own startup, before + # it can be "ready" -- without this, headless DNS would omit it until + # ready, deadlocking startup. + publishNotReadyAddresses: true + selector: + app: afd-recipe + afd-ffn-node-role: head + ports: +${PORT_ENTRIES} diff --git a/recipe/gpu/P2pNcclAFDConnector/README.md b/recipe/gpu/P2pNcclAFDConnector/README.md index 59484e71b..43bbb8956 100644 --- a/recipe/gpu/P2pNcclAFDConnector/README.md +++ b/recipe/gpu/P2pNcclAFDConnector/README.md @@ -52,10 +52,10 @@ and waits until an OpenAI-compatible endpoint answers. The skill requires the recipe path (e.g. `recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh`) -plus the image, model id, and PVC name it needs. It derives `GPU_COUNT` -itself for a plain single-node recipe; for a multi-node-capable recipe it -instead confirms the recipe's attention/FFN rank totals and asks you for a -placement plan (see below). +plus the image, model id, and PVC name it needs. Placement is never asked +for: the skill derives `GPU_COUNT` itself for a plain single-pod recipe, and +for a multi-pod recipe it reads the exact per-pod placement straight off the +recipe's own header comment (see below). **Scope.** The skill only covers `prefill_decode_colocation` recipes (and their `baseline*` counterparts) under `recipe/gpu/P2pNcclAFDConnector/**`. @@ -69,16 +69,18 @@ the simple, non-experimental path. The skill computes `GPU_COUNT` itself from the script's `CUDA_VISIBLE_DEVICES` blocks and deploys a single `vllm-pod` / `vllm-service` pair. -**Multi-node.** Only recipes that gate their `vllm serve` blocks on -`ATTENTION_DP_RANKS`/`FFN_DP_RANKS` and read the AFD `host` field from an -`AFD_CONNECTOR_HOST` env var support spreading attention/FFN ranks across -more than one pod (e.g. `deepseek_v2_lite/prefill_decode_colocation/*.sh`). -For these, the skill reads the recipe's fixed attention/FFN rank totals and -asks you for an explicit **placement plan** -- how many of those ranks go in -each pod, and how many pods -- rather than assuming a shape. It then creates -one pod per plan entry, an internal FFN rendezvous Service when the plan -spans more than one pod, and DP-coordination Services for any role that is -itself split across pods. +**Multi-node.** A recipe that can spread attention/FFN ranks across more +than one pod says so directly, via a top-of-file comment describing the +placement (e.g. `qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh`, +`deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh`). +The skill reads that comment and deploys exactly what it describes -- it +never asks for or guesses a placement. Recipes without such a comment +deploy as a single pod. For a multi-pod plan, the skill creates one pod per +entry, an internal FFN rendezvous Service whenever the plan spans more than +one pod, and DP-coordination Services for any role that is itself split +across pods -- all of these Services are created *before* any pod, since a +role's head pod binds its own rendezvous server to its Service's name +within seconds of starting. Pods and Services are left running after deployment so weights stay warm for follow-up runs; see the skill's teardown step to tear them down explicitly. diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md index d5cc57e41..132f15528 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md @@ -27,11 +27,14 @@ End-to-end launch scripts for running DeepSeek-V2-Lite with the AFD ├── prefill_decode_disaggregation/ # prefill_decode_disaggregation, 2P1A1F topology │ ├── 2p1a1f_eager_dbo.sh │ └── 2p1a1f_graph_dbo.sh -└── prefill_decode_colocation/ # prefill_decode_colocation, 2A2F topology +└── prefill_decode_colocation/ # prefill_decode_colocation, 2A2F/4A4F topology ├── 2a2f_eager_dbo_dp1tp2.sh ├── 2a2f_eager_dbo_dp2tp1.sh ├── 2a2f_graph_dbo_dp1tp2.sh - └── 2a2f_graph_dbo_dp2tp1.sh + ├── 2a2f_graph_dbo_dp2tp1.sh + ├── 4a4f_eager_dbo_dp2tp2.sh + ├── 4a4f_graph_dbo_dp2tp2.sh + └── multipod_2a_2f_graph_dbo_dp1tp2.sh # 2a2f_graph_dbo_dp1tp2, split across 2 k8s pods ``` ### 1. Prefill/Decode Disaggregation — `1a1f` @@ -55,12 +58,22 @@ End-to-end launch scripts for running DeepSeek-V2-Lite with the AFD | 0, 1 | Attention | 18305 | | 2, 3 | FFN | 18305 | -The four variants cover the TP/DP cross product: +The four `2a2f_*` variants cover the TP/DP cross product; `4a4f_*` doubles +both attention and FFN ranks (DP=2 on each side); `multipod_2a_2f_graph_dbo_dp1tp2.sh` +is the same `2a2f_graph_dbo_dp1tp2` config, but gated on a `POD` env var +(`ATTENTION_0` or `FFN_0`) for the `deploy-afd-k8s` skill to deploy it +across 2 pods (see repo-root `recipe/gpu/P2pNcclAFDConnector/README.md`) -- +unlike the scripts above, it is **not** meant to be run directly: with +`POD` unset, neither `vllm serve` block executes, and its +`AFD_CONNECTOR_HOST` default (`vllm-ffn-p2p-service`) only resolves inside +the cluster the skill deploys it to: | File | DP | TP | |---------------------------------|----|----| | `2a2f_*_dp1tp2.sh` | 1 | 2 | | `2a2f_*_dp2tp1.sh` | 2 | 1 | +| `4a4f_*_dp2tp2.sh` | 2 | 2 | +| `multipod_2a_2f_graph_dbo_dp1tp2.sh` | 1 | 2 | ## Running diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh index f6ecc29d1..ba5db48c2 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp1tp2.sh @@ -1,104 +1,52 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -# Fixed topology for this recipe -- always sent to AFD for rendezvous, -# regardless of which role(s) this node launches locally. -ATTENTION_DP_SIZE=1 -ATTENTION_TP_SIZE=2 -FFN_DP_SIZE=1 -FFN_TP_SIZE=2 -NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) -NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) - -# How many DP ranks of each role to launch on THIS node. 0 skips that role. -# Default runs the full topology on one node (single-node colocation); -# a multi-node deploy overrides one of these to 0 per pod. -ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} -FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} - -# DP sharding: lets a single role's DP group be split across multiple pods. -# Defaults reproduce today's single-pod-per-role behavior exactly (local -# rank count == full DP size, start rank 0, loopback DP-RPC address). -ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} -FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} -ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} -FFN_HEADLESS=${FFN_HEADLESS:-0} -ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} -FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} -ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} -FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} - -if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then - ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$ATTENTION_DP_SIZE" \ - --data-parallel-size-local "$ATTENTION_DP_RANKS" \ - --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ - --data-parallel-address "$ATTENTION_DP_ADDRESS" \ - --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ - --tensor-parallel-size "$ATTENTION_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & -fi - -if [ "$FFN_DP_RANKS" -gt 0 ]; then - FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) - FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$FFN_DP_SIZE" \ - --data-parallel-size-local "$FFN_DP_RANKS" \ - --data-parallel-start-rank "$FFN_DP_START_RANK" \ - --data-parallel-address "$FFN_DP_ADDRESS" \ - --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ - --tensor-parallel-size "$FFN_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & -fi +CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 1 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --enforce-eager \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & + +CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 1 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --enforce-eager \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh index b85c28512..a51058278 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh @@ -1,104 +1,52 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -# Fixed topology for this recipe -- always sent to AFD for rendezvous, -# regardless of which role(s) this node launches locally. -ATTENTION_DP_SIZE=2 -ATTENTION_TP_SIZE=1 -FFN_DP_SIZE=2 -FFN_TP_SIZE=1 -NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) -NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) - -# How many DP ranks of each role to launch on THIS node. 0 skips that role. -# Default runs the full topology on one node (single-node colocation); -# a multi-node deploy overrides one of these to 0 per pod. -ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} -FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} - -# DP sharding: lets a single role's DP group be split across multiple pods. -# Defaults reproduce today's single-pod-per-role behavior exactly (local -# rank count == full DP size, start rank 0, loopback DP-RPC address). -ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} -FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} -ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} -FFN_HEADLESS=${FFN_HEADLESS:-0} -ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} -FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} -ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} -FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} - -if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then - ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$ATTENTION_DP_SIZE" \ - --data-parallel-size-local "$ATTENTION_DP_RANKS" \ - --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ - --data-parallel-address "$ATTENTION_DP_ADDRESS" \ - --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ - --tensor-parallel-size "$ATTENTION_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & -fi - -if [ "$FFN_DP_RANKS" -gt 0 ]; then - FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) - FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$FFN_DP_SIZE" \ - --data-parallel-size-local "$FFN_DP_RANKS" \ - --data-parallel-start-rank "$FFN_DP_START_RANK" \ - --data-parallel-address "$FFN_DP_ADDRESS" \ - --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ - --tensor-parallel-size "$FFN_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & -fi +CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --enforce-eager \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & + +CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --enforce-eager \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh index a2b119716..2beabcad1 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh @@ -1,110 +1,58 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -# Fixed topology for this recipe -- always sent to AFD for rendezvous, -# regardless of which role(s) this node launches locally. -ATTENTION_DP_SIZE=1 -ATTENTION_TP_SIZE=2 -FFN_DP_SIZE=1 -FFN_TP_SIZE=2 -NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) -NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) - -# How many DP ranks of each role to launch on THIS node. 0 skips that role. -# Default runs the full topology on one node (single-node colocation); -# a multi-node deploy overrides one of these to 0 per pod. -ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} -FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} - -# DP sharding: lets a single role's DP group be split across multiple pods. -# Defaults reproduce today's single-pod-per-role behavior exactly (local -# rank count == full DP size, start rank 0, loopback DP-RPC address). -ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} -FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} -ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} -FFN_HEADLESS=${FFN_HEADLESS:-0} -ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} -FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} -ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} -FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} - -if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then - ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$ATTENTION_DP_SIZE" \ - --data-parallel-size-local "$ATTENTION_DP_RANKS" \ - --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ - --data-parallel-address "$ATTENTION_DP_ADDRESS" \ - --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ - --tensor-parallel-size "$ATTENTION_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & -fi - -if [ "$FFN_DP_RANKS" -gt 0 ]; then - FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) - FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$FFN_DP_SIZE" \ - --data-parallel-size-local "$FFN_DP_RANKS" \ - --data-parallel-start-rank "$FFN_DP_START_RANK" \ - --data-parallel-address "$FFN_DP_ADDRESS" \ - --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ - --tensor-parallel-size "$FFN_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & -fi +CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 1 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & + +CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 1 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh index 54fcef80a..72db1a705 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp2tp1.sh @@ -1,109 +1,58 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -# Fixed topology for this recipe -- always sent to AFD for rendezvous, -# regardless of which role(s) this node launches locally. -ATTENTION_DP_SIZE=2 -ATTENTION_TP_SIZE=1 -FFN_DP_SIZE=2 -FFN_TP_SIZE=1 -NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) -NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) - -# How many DP ranks of each role to launch on THIS node. 0 skips that role. -# Default runs the full topology on one node (single-node colocation); -# a multi-node deploy overrides one of these to 0 per pod. -ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} -FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} - -# DP sharding: lets a single role's DP group be split across multiple pods. -# Defaults reproduce today's single-pod-per-role behavior exactly (local -# rank count == full DP size, start rank 0, loopback DP-RPC address). -ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} -FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} -ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} -FFN_HEADLESS=${FFN_HEADLESS:-0} -ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} -FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} -ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} -FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} - -if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then - ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$ATTENTION_DP_SIZE" \ - --data-parallel-size-local "$ATTENTION_DP_RANKS" \ - --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ - --data-parallel-address "$ATTENTION_DP_ADDRESS" \ - --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ - --tensor-parallel-size "$ATTENTION_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & -fi - -if [ "$FFN_DP_RANKS" -gt 0 ]; then - FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) - FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$FFN_DP_SIZE" \ - --data-parallel-size-local "$FFN_DP_RANKS" \ - --data-parallel-start-rank "$FFN_DP_START_RANK" \ - --data-parallel-address "$FFN_DP_ADDRESS" \ - --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ - --tensor-parallel-size "$FFN_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & -fi +CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & + +CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh index 0eb987ef9..7b7e9291e 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_eager_dbo_dp2tp2.sh @@ -1,104 +1,52 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -# Fixed topology for this recipe -- always sent to AFD for rendezvous, -# regardless of which role(s) this node launches locally. -ATTENTION_DP_SIZE=2 -ATTENTION_TP_SIZE=2 -FFN_DP_SIZE=2 -FFN_TP_SIZE=2 -NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) -NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) - -# How many DP ranks of each role to launch on THIS node. 0 skips that role. -# Default runs the full topology on one node (single-node colocation); -# a multi-node deploy overrides one of these to 0 per pod. -ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} -FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} - -# DP sharding: lets a single role's DP group be split across multiple pods. -# Defaults reproduce today's single-pod-per-role behavior exactly (local -# rank count == full DP size, start rank 0, loopback DP-RPC address). -ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} -FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} -ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} -FFN_HEADLESS=${FFN_HEADLESS:-0} -ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} -FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} -ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} -FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} - -if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then - ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$ATTENTION_DP_SIZE" \ - --data-parallel-size-local "$ATTENTION_DP_RANKS" \ - --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ - --data-parallel-address "$ATTENTION_DP_ADDRESS" \ - --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ - --tensor-parallel-size "$ATTENTION_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --enforce-eager \ - $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & -fi - -if [ "$FFN_DP_RANKS" -gt 0 ]; then - FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) - FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$FFN_DP_SIZE" \ - --data-parallel-size-local "$FFN_DP_RANKS" \ - --data-parallel-start-rank "$FFN_DP_START_RANK" \ - --data-parallel-address "$FFN_DP_ADDRESS" \ - --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ - --tensor-parallel-size "$FFN_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --enforce-eager \ - $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & -fi +CUDA_VISIBLE_DEVICES=0,1,2,3 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 4, + "num_ffn_ranks": 4 + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --enforce-eager \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & + +CUDA_VISIBLE_DEVICES=4,5,6,7 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 4, + "num_ffn_ranks": 4 + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --enforce-eager \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh index 2b87d9624..baa00323c 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh @@ -1,110 +1,58 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} export VLLM_USE_V2_MODEL_RUNNER=0 -# Fixed topology for this recipe -- always sent to AFD for rendezvous, -# regardless of which role(s) this node launches locally. -ATTENTION_DP_SIZE=2 -ATTENTION_TP_SIZE=2 -FFN_DP_SIZE=2 -FFN_TP_SIZE=2 -NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) -NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) - -# How many DP ranks of each role to launch on THIS node. 0 skips that role. -# Default runs the full topology on one node (single-node colocation); -# a multi-node deploy overrides one of these to 0 per pod. -ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} -FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} - -# DP sharding: lets a single role's DP group be split across multiple pods. -# Defaults reproduce today's single-pod-per-role behavior exactly (local -# rank count == full DP size, start rank 0, loopback DP-RPC address). -ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} -FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} -ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} -FFN_HEADLESS=${FFN_HEADLESS:-0} -ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} -FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} -ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} -FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} - -if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then - ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$ATTENTION_DP_SIZE" \ - --data-parallel-size-local "$ATTENTION_DP_RANKS" \ - --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ - --data-parallel-address "$ATTENTION_DP_ADDRESS" \ - --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ - --tensor-parallel-size "$ATTENTION_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --max-num-batched-tokens 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > attn.log 2>&1 & -fi - -if [ "$FFN_DP_RANKS" -gt 0 ]; then - FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) - FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$FFN_DP_SIZE" \ - --data-parallel-size-local "$FFN_DP_RANKS" \ - --data-parallel-start-rank "$FFN_DP_START_RANK" \ - --data-parallel-address "$FFN_DP_ADDRESS" \ - --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ - --tensor-parallel-size "$FFN_TP_SIZE" \ - --enable-expert-parallel \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 64 \ - --enable-dbo \ - --dbo-decode-token-threshold 2 \ - --dbo-prefill-token-threshold 12 \ - --max-num-batched-tokens 64 \ - --max-cudagraph-capture-size 64 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] - }' \ - $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 \ - --trust-remote-code > ffn.log 2>&1 & -fi +CUDA_VISIBLE_DEVICES=0,1,2,3 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 4, + "num_ffn_ranks": 4 + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & + +CUDA_VISIBLE_DEVICES=4,5,6,7 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "127.0.0.1", + "port": 6269, + "num_attention_ranks": 4, + "num_ffn_ranks": 4 + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & wait diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh new file mode 100644 index 000000000..b44372728 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + +# Use .agents/skills/deploy-afd-k8s/SKILL.md to deploy this recipe +# on 2 pods using the setup below. +# For pod 1: POD=ATTENTION_0 +# For pod 2: POD=FFN_0 + +MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} +export VLLM_USE_V2_MODEL_RUNNER=0 + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-vllm-ffn-p2p-service} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} + +if [ "$POD" == "ATTENTION_0" ]; then + CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 1 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & +fi + +if [ "$POD" == "FFN_0" ]; then + CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 1 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & +fi + +wait diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh deleted file mode 100644 index ac7206c6b..000000000 --- a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/4a2f_graph.sh +++ /dev/null @@ -1,114 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project - -MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3.5-122B-A10B-FP8} -export VLLM_USE_V2_MODEL_RUNNER=0 - -# Fixed topology for this recipe -- always sent to AFD for rendezvous, -# regardless of which role(s) this node launches locally. -ATTENTION_DP_SIZE=4 -ATTENTION_TP_SIZE=1 -FFN_DP_SIZE=2 -FFN_TP_SIZE=1 -NUM_ATTENTION_RANKS=$((ATTENTION_DP_SIZE * ATTENTION_TP_SIZE)) -NUM_FFN_RANKS=$((FFN_DP_SIZE * FFN_TP_SIZE)) - -# How many DP ranks of each role to launch on THIS node. 0 skips that role. -# Default runs the full topology on one node (single-node colocation); -# a multi-node deploy overrides one of these to 0 per pod. -ATTENTION_DP_RANKS=${ATTENTION_DP_RANKS:-$ATTENTION_DP_SIZE} -FFN_DP_RANKS=${FFN_DP_RANKS:-$FFN_DP_SIZE} - -AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-127.0.0.1} -AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-1239} - -# DP sharding: lets a single role's DP group be split across multiple pods. -# Defaults reproduce today's single-pod-per-role behavior exactly (local -# rank count == full DP size, start rank 0, loopback DP-RPC address). -ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} -FFN_DP_START_RANK=${FFN_DP_START_RANK:-0} -ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} -FFN_HEADLESS=${FFN_HEADLESS:-0} -ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-127.0.0.1} -FFN_DP_ADDRESS=${FFN_DP_ADDRESS:-127.0.0.1} -ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} -FFN_DP_RPC_PORT=${FFN_DP_RPC_PORT:-13346} - -if [ "$ATTENTION_DP_RANKS" -gt 0 ]; then - ATTN_DEVICES=$(seq -s, 0 $((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$ATTN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$ATTENTION_DP_SIZE" \ - --data-parallel-size-local "$ATTENTION_DP_RANKS" \ - --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ - --data-parallel-address "$ATTENTION_DP_ADDRESS" \ - --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ - --tensor-parallel-size "$ATTENTION_TP_SIZE" \ - --enable-expert-parallel \ - --dtype bfloat16 \ - --language-model-only \ - --max-model-len 114688 \ - --mamba-cache-mode align \ - --all2all-backend allgather_reducescatter \ - --seed 0 \ - --additional-config '{ - "afd": { - "role": "attention", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 32 \ - --max-num-batched-tokens 8192 \ - --max-cudagraph-capture-size 32 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] - }' \ - $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 > attn.log 2>&1 & -fi - -if [ "$FFN_DP_RANKS" -gt 0 ]; then - FFN_DEVICE_START=$((ATTENTION_DP_RANKS * ATTENTION_TP_SIZE)) - FFN_DEVICES=$(seq -s, "$FFN_DEVICE_START" $((FFN_DEVICE_START + FFN_DP_RANKS * FFN_TP_SIZE - 1))) - # shellcheck disable=SC2046 - CUDA_VISIBLE_DEVICES="$FFN_DEVICES" uv run vllm serve "$MODEL_PATH" \ - --data-parallel-size "$FFN_DP_SIZE" \ - --data-parallel-size-local "$FFN_DP_RANKS" \ - --data-parallel-start-rank "$FFN_DP_START_RANK" \ - --data-parallel-address "$FFN_DP_ADDRESS" \ - --data-parallel-rpc-port "$FFN_DP_RPC_PORT" \ - --tensor-parallel-size "$FFN_TP_SIZE" \ - --enable-expert-parallel \ - --dtype bfloat16 \ - --language-model-only \ - --max-model-len 114688 \ - --mamba-cache-mode align \ - --all2all-backend allgather_reducescatter \ - --seed 0 \ - --additional-config '{ - "afd": { - "role": "ffn", - "connector": "P2pNcclAFDConnector", - "host": "'"${AFD_CONNECTOR_HOST}"'", - "port": '"${AFD_CONNECTOR_PORT}"', - "num_attention_ranks": '"${NUM_ATTENTION_RANKS}"', - "num_ffn_ranks": '"${NUM_FFN_RANKS}"' - } - }' \ - --max-num-seqs 32 \ - --max-num-batched-tokens 8192 \ - --max-cudagraph-capture-size 32 \ - --compilation-config '{ - "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] - }' \ - $([ "$FFN_HEADLESS" = "1" ] && echo --headless) \ - --host 127.0.0.1 \ - --port 18305 > ffn.log 2>&1 & -fi - -wait diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh new file mode 100644 index 000000000..8f8656915 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh @@ -0,0 +1,88 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + +# Use .agents/skills/deploy-afd-k8s/SKILL.md to deploy this recipe +# on 3 pods using the setup below. +# For pod 1: POD=ATTENTION_0 +# For pod 2: POD=ATTENTION_1, ATTENTION_HEADLESS=1, ATTENTION_DP_START_RANK=2 +# For pod 3: POD=FFN_0 + +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3.5-122B-A10B-FP8} +export VLLM_USE_V2_MODEL_RUNNER=0 + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-vllm-ffn-p2p-service} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-1239} + +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-vllm-attn-dp-service} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} + +if [[ "$POD" == ATTENTION_* ]]; then + CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 4 \ + --data-parallel-size-local 2 \ + --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --dtype bfloat16 \ + --language-model-only \ + --max-model-len 114688 \ + --mamba-cache-mode align \ + --all2all-backend allgather_reducescatter \ + --seed 0 \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": 4, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 32 \ + --max-num-batched-tokens 8192 \ + --max-cudagraph-capture-size 32 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] + }' \ + $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ + --host 127.0.0.1 \ + --port 18305 > attn.log 2>&1 & +fi + +if [[ "$POD" == FFN_* ]]; then + CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --dtype bfloat16 \ + --language-model-only \ + --max-model-len 114688 \ + --mamba-cache-mode align \ + --all2all-backend allgather_reducescatter \ + --seed 0 \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": 4, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 32 \ + --max-num-batched-tokens 8192 \ + --max-cudagraph-capture-size 32 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] + }' \ + --host 127.0.0.1 \ + --port 18305 > ffn.log 2>&1 & +fi + +wait From 7c14d26403cb87e78b09af32b8947cb310fa7931 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Sun, 4 Oct 2026 16:53:54 +0300 Subject: [PATCH 08/15] increasing timeout Signed-off-by: Shlomit Koyfman --- .../prefill_decode_colocation/multipod_2a_2a_2f_graph.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh index 8f8656915..a7137c5d9 100644 --- a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh @@ -37,6 +37,7 @@ if [[ "$POD" == ATTENTION_* ]]; then "afd": { "role": "attention", "connector": "P2pNcclAFDConnector", + "afd_process_group_timeout_s": 120, "host": "'"${AFD_CONNECTOR_HOST}"'", "port": '"${AFD_CONNECTOR_PORT}"', "num_attention_ranks": 4, @@ -69,6 +70,7 @@ if [[ "$POD" == FFN_* ]]; then "afd": { "role": "ffn", "connector": "P2pNcclAFDConnector", + "afd_process_group_timeout_s": 120, "host": "'"${AFD_CONNECTOR_HOST}"'", "port": '"${AFD_CONNECTOR_PORT}"', "num_attention_ranks": 4, From 02ef6e8b630348a896ce8efdc4d3cb7973cc8c2f Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Sun, 4 Oct 2026 17:10:11 +0300 Subject: [PATCH 09/15] pre-commit error fixes Signed-off-by: Shlomit Koyfman --- .../deepseek_v2_lite/README.md | 19 ++++++++++--------- .../multipod_2a_2a_2f_graph.sh | 7 ++++++- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md index 132f15528..04462135f 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md @@ -22,7 +22,7 @@ End-to-end launch scripts for running DeepSeek-V2-Lite with the AFD ## Directory layout -``` +```text . ├── prefill_decode_disaggregation/ # prefill_decode_disaggregation, 2P1A1F topology │ ├── 2p1a1f_eager_dbo.sh @@ -36,6 +36,7 @@ End-to-end launch scripts for running DeepSeek-V2-Lite with the AFD ├── 4a4f_graph_dbo_dp2tp2.sh └── multipod_2a_2f_graph_dbo_dp1tp2.sh # 2a2f_graph_dbo_dp1tp2, split across 2 k8s pods ``` + ### 1. Prefill/Decode Disaggregation — `1a1f` 5 processes, 4 GPU workers + 1 proxy server: @@ -48,7 +49,6 @@ End-to-end launch scripts for running DeepSeek-V2-Lite with the AFD | 3 | Decode (FFN) | 18304 | | / | Proxy Server | 18305 | - ### 2. Prefill/Decode Colocation — `2a2f` 2 processes, two GPUs each: @@ -68,12 +68,12 @@ unlike the scripts above, it is **not** meant to be run directly: with `AFD_CONNECTOR_HOST` default (`vllm-ffn-p2p-service`) only resolves inside the cluster the skill deploys it to: -| File | DP | TP | -|---------------------------------|----|----| -| `2a2f_*_dp1tp2.sh` | 1 | 2 | -| `2a2f_*_dp2tp1.sh` | 2 | 1 | -| `4a4f_*_dp2tp2.sh` | 2 | 2 | -| `multipod_2a_2f_graph_dbo_dp1tp2.sh` | 1 | 2 | +| File | DP | TP | +|---------------------------------------|----|----| +| `2a2f_*_dp1tp2.sh` | 1 | 2 | +| `2a2f_*_dp2tp1.sh` | 2 | 1 | +| `4a4f_*_dp2tp2.sh` | 2 | 2 | +| `multipod_2a_2f_graph_dbo_dp1tp2.sh` | 1 | 2 | ## Running @@ -84,6 +84,7 @@ Wait for `attn.log` (and `afd_prefill0.log`, `afd_prefill1.log` in disaggregatio before sending traffic. ### prefill_decode_colocation + ```bash export MODEL_PATH=/path/model_weights/DeepSeek-V2-Lite export VLLM_USE_V2_MODEL_RUNNER=0 @@ -145,7 +146,7 @@ and deployment configuration. Graph mode replaces `--enforce-eager` with: -``` +```text --max-cudagraph-capture-size 64 --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64]}' diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh index a7137c5d9..dcea163a0 100644 --- a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh @@ -18,6 +18,11 @@ ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-vllm-attn-dp-service} ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} +HEADLESS_ARGS=() +if [[ "$ATTENTION_HEADLESS" == "1" ]]; then + HEADLESS_ARGS=(--headless) +fi + if [[ "$POD" == ATTENTION_* ]]; then CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ --data-parallel-size 4 \ @@ -50,7 +55,7 @@ if [[ "$POD" == ATTENTION_* ]]; then --compilation-config '{ "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] }' \ - $([ "$ATTENTION_HEADLESS" = "1" ] && echo --headless) \ + "${HEADLESS_ARGS[@]}" \ --host 127.0.0.1 \ --port 18305 > attn.log 2>&1 & fi From 0dd4c81b9b79ef889db4980cb1d92cd3cfa6e4cc Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Sun, 4 Oct 2026 17:20:46 +0300 Subject: [PATCH 10/15] pre-commit error fixes Signed-off-by: Shlomit Koyfman --- .agents/skills/deploy-afd-k8s/references/deploy.md | 7 +++++-- .agents/skills/deploy-afd-k8s/references/services.md | 2 +- .agents/skills/deploy-afd-k8s/templates/pod.yaml | 2 +- .../skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml | 2 +- 4 files changed, 8 insertions(+), 5 deletions(-) diff --git a/.agents/skills/deploy-afd-k8s/references/deploy.md b/.agents/skills/deploy-afd-k8s/references/deploy.md index 3a27501e7..8eb587d26 100644 --- a/.agents/skills/deploy-afd-k8s/references/deploy.md +++ b/.agents/skills/deploy-afd-k8s/references/deploy.md @@ -138,10 +138,13 @@ deploy_pod() { # extra_env_pairs: space-separated VAR=value tokens straight off this # plan entry's header line (including POD=... itself), empty for a plain # recipe. Build the spliced env-entries block for templates/pod.yaml. + # Leading (not trailing) newlines keep the `# ${TEMPLATE_EXTRA_ENV}` + # placeholder line a harmless comment when this is empty, and valid + # sibling list items when it isn't. local extra_env="" for pair in $extra_env_pairs; do - extra_env="$extra_env - {name: ${pair%%=*}, value: \"${pair#*=}\"} -" + extra_env="$extra_env + - {name: ${pair%%=*}, value: \"${pair#*=}\"}" done kubectl delete pod "$pod_name" --ignore-not-found diff --git a/.agents/skills/deploy-afd-k8s/references/services.md b/.agents/skills/deploy-afd-k8s/references/services.md index ae59e07ad..dea5a5fea 100644 --- a/.agents/skills/deploy-afd-k8s/references/services.md +++ b/.agents/skills/deploy-afd-k8s/references/services.md @@ -34,7 +34,7 @@ constants off the recipe's own defaults/assignments): FFN_PORT_START="$(grep -oE 'AFD_CONNECTOR_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" NUM_FFN_RANKS="$(grep -oE 'NUM_FFN_RANKS=[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+$' | head -1)" PORT_ENTRIES="$(for p in $(seq "$FFN_PORT_START" "$((FFN_PORT_START + NUM_FFN_RANKS))"); do - printf ' - {name: p%s, port: %s, targetPort: %s}\n' "$p" "$p" "$p" + printf '\n - {name: p%s, port: %s, targetPort: %s}' "$p" "$p" "$p" done)" PORT_ENTRIES="$PORT_ENTRIES" envsubst '${PORT_ENTRIES}' \ < templates/service-ffn-p2p.yaml | kubectl apply -f - diff --git a/.agents/skills/deploy-afd-k8s/templates/pod.yaml b/.agents/skills/deploy-afd-k8s/templates/pod.yaml index 02056dd4d..997e90be4 100644 --- a/.agents/skills/deploy-afd-k8s/templates/pod.yaml +++ b/.agents/skills/deploy-afd-k8s/templates/pod.yaml @@ -163,7 +163,7 @@ spec: value: "${TEMPLATE_ATTN_ROLE}" - name: AFD_FFN_NODE_ROLE value: "${TEMPLATE_FFN_ROLE}" -${TEMPLATE_EXTRA_ENV} + # ${TEMPLATE_EXTRA_ENV} - name: HF_TOKEN valueFrom: secretKeyRef: diff --git a/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml b/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml index 66d5581b9..216e6f3f2 100644 --- a/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml +++ b/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml @@ -17,4 +17,4 @@ spec: app: afd-recipe afd-ffn-node-role: head ports: -${PORT_ENTRIES} + # ${PORT_ENTRIES} From a7b4d25a2c49263b3f005f694d808877898298d7 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Mon, 5 Oct 2026 11:18:28 +0300 Subject: [PATCH 11/15] fix service generation Signed-off-by: Shlomit Koyfman --- .agents/skills/deploy-afd-k8s/references/services.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/.agents/skills/deploy-afd-k8s/references/services.md b/.agents/skills/deploy-afd-k8s/references/services.md index dea5a5fea..3a59b9ecd 100644 --- a/.agents/skills/deploy-afd-k8s/references/services.md +++ b/.agents/skills/deploy-afd-k8s/references/services.md @@ -24,15 +24,16 @@ CLIENT_PORT="$CLIENT_PORT" envsubst '${CLIENT_PORT}' \ Internal only, carries AFD rendezvous traffic to whichever pod holds `afd-ffn-node-role: head`. Must exist before any multi-pod deploy, since the -recipe's own `AFD_CONNECTOR_HOST` default now points at this name directly -(fixed at the recipe level -- this skill no longer computes that value). +recipe's own `AFD_CONNECTOR_HOST` default now points at this name directly. Exposes the control-plane port plus one derived port per FFN rank -(`[AFD_CONNECTOR_PORT, AFD_CONNECTOR_PORT + NUM_FFN_RANKS]` -- read both -constants off the recipe's own defaults/assignments): +(`[AFD_CONNECTOR_PORT, AFD_CONNECTOR_PORT + NUM_FFN_RANKS]` -- read +`AFD_CONNECTOR_PORT` off the recipe's own shell default and `NUM_FFN_RANKS` +off its `"num_ffn_ranks"` JSON config key, the *total* rank count for the +FFN role across all pods, not a per-pod count): ```bash FFN_PORT_START="$(grep -oE 'AFD_CONNECTOR_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" -NUM_FFN_RANKS="$(grep -oE 'NUM_FFN_RANKS=[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+$' | head -1)" +NUM_FFN_RANKS="$(grep -oE '"num_ffn_ranks":[[:space:]]*[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+$' | head -1)" PORT_ENTRIES="$(for p in $(seq "$FFN_PORT_START" "$((FFN_PORT_START + NUM_FFN_RANKS))"); do printf '\n - {name: p%s, port: %s, targetPort: %s}' "$p" "$p" "$p" done)" From c7881a0048efcb82495aaf7d20de2217d26822af Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Mon, 5 Oct 2026 12:13:08 +0300 Subject: [PATCH 12/15] fix PR comments Signed-off-by: Shlomit Koyfman --- .agents/skills/deploy-afd-k8s/SKILL.md | 20 ++++----- .../deploy-afd-k8s/references/deploy.md | 37 +++++---------- .../references/report-and-teardown.md | 9 ++-- .../references/resolve-recipe.md | 17 ++++--- .../deploy-afd-k8s/references/services.md | 31 +++++-------- .../skills/deploy-afd-k8s/templates/pod.yaml | 40 ++++------------- .../templates/service-ffn-p2p.yaml | 4 +- recipe/gpu/P2pNcclAFDConnector/README.md | 45 +++++-------------- 8 files changed, 68 insertions(+), 135 deletions(-) diff --git a/.agents/skills/deploy-afd-k8s/SKILL.md b/.agents/skills/deploy-afd-k8s/SKILL.md index bcbfe1b9d..f51ae4190 100644 --- a/.agents/skills/deploy-afd-k8s/SKILL.md +++ b/.agents/skills/deploy-afd-k8s/SKILL.md @@ -1,6 +1,6 @@ --- name: deploy-afd-k8s -description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is read directly off the recipe's own top-of-file comment block (however that block describes it, strict or prose): a recipe whose comment describes a per-pod placement deploys exactly that many pods with exactly that per-pod setup; a recipe without one deploys as a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). To drive load against what this deploys, see run-vllm-bench-k8s. +description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is read directly off the recipe's own top-of-file comment block (however that block describes it, strict or prose): a recipe whose comment describes a per-pod placement deploys exactly that many pods with exactly that per-pod setup; a recipe without one deploys as a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). --- # Deploy an AFD GPU recipe on Kubernetes @@ -31,7 +31,7 @@ than one pod) -- callers never talk to these directly. See - Recipes under `recipe/gpu/P2pNcclAFDConnector/**` (GPU only, not NPU), **`prefill_decode_colocation` only** (attention + FFN workers, no prefill - split) and their `baseline*.sh` counterparts. + split). `prefill_decode_disaggregation` recipes are **out of scope**: they need a NIXL-enabled image plus a proxy, and resolve a `SCRIPT_DIR`-relative path that this skill's flat script mount (step 4b) breaks. Ask the user for a @@ -64,7 +64,8 @@ than one pod) -- callers never talk to these directly. See The image supplies only the `afd-plugin` install. The recipe script is mounted fresh from local disk on every run (step 4b), so editing or adding a recipe never requires a rebuild. -- An `hf-token-secret` Secret with a `token` key: +- An `hf-token-secret` Secret with a `token` key, needed when `MODEL_ID` is + a HF Hub repo id (not needed when weights are already staged on the PVC): ```bash kubectl create secret generic hf-token-secret --from-literal=token= ``` @@ -72,19 +73,14 @@ than one pod) -- callers never talk to these directly. See ## Workflow -Load these in order; each covers one phase and nothing else. `deploy.md` -hands off to `services.md` partway through (its own step 4d) and back -- -Services must exist **before** any Pod is deployed (step 4e), since a -role's head Pod binds its own rendezvous/DP-RPC server to its Service's -name within seconds of starting, and `restartPolicy: Never` means a bind -that fails because the Service doesn't exist yet never retries: +Load these in order; each covers one phase and nothing else. Services must +exist **before** any Pod is deployed -- see +[services.md](references/services.md) for why: | Reference | Covers | |---|---| | [resolve-recipe.md](references/resolve-recipe.md) | Step 1: read the recipe's serve-block fields and model id; read its placement (or lack of one) from its header comment -- never ask. | -| [deploy.md](references/deploy.md) | Steps 2-4c: GPU_COUNT, confirm, PVC, ConfigMap, fsGroup. | -| [services.md](references/services.md) | Step 4d: which of the four Services to create, derived from the parsed plan -- before any Pod. | -| [deploy.md](references/deploy.md) (resume) | Steps 4e-4f: pod(s) via `templates/pod.yaml`, wait for Running and for each pod's own readiness line. | +| [deploy.md](references/deploy.md) | Steps 2-4f: GPU_COUNT, confirm, PVC, ConfigMap, fsGroup, Services (step 4d -- which of the four, see [services.md](references/services.md)), pod(s) via `templates/pod.yaml`, wait for Running and for each pod's own readiness line. | | [report-and-teardown.md](references/report-and-teardown.md) | Step 5: report the endpoint/model/nodes. Step 6: teardown (pods/Services only -- never the PVC unless asked). | All k8s manifests live in `templates/*.yaml`, applied via `envsubst | kubectl diff --git a/.agents/skills/deploy-afd-k8s/references/deploy.md b/.agents/skills/deploy-afd-k8s/references/deploy.md index 8eb587d26..eccd5423a 100644 --- a/.agents/skills/deploy-afd-k8s/references/deploy.md +++ b/.agents/skills/deploy-afd-k8s/references/deploy.md @@ -43,7 +43,6 @@ MODEL_ID= PVC_NAME= CLIENT_PORT= # from the script's --port; default 18305 RECIPE_SCRIPT_PATH= -VLLM_USE_V2_MODEL_RUNNER=${VLLM_USE_V2_MODEL_RUNNER:-0} ``` **Node placement is left to the scheduler by default.** Only add @@ -66,9 +65,8 @@ asks to exercise cross-node placement -- add this to every Pod's ## 4a. Model PVC -Size for the model (e.g. Qwen3.5-122B-A10B-FP8 needs `200Gi` minimum). An -existing undersized PVC is reused silently and fails mid-download, so set -`MODEL_PVC_SIZE` deliberately. `PVC_ACCESS_MODE` is derived, not asked: +Size for the model being served. An existing undersized PVC is reused +silently and fails mid-download, so set `MODEL_PVC_SIZE` deliberately. `PVC_ACCESS_MODE` is derived, not asked: `ReadWriteMany` whenever the plan (if any) has more than one pod -- an RWO volume can only attach from one node, and the second Pod deadlocks `Pending` with `Multi-Attach error` otherwise. @@ -118,14 +116,11 @@ FS_GROUP="${FS_GROUP:-1000}" ## 4d. Services -Create every Service that applies -- see [services.md](services.md) -- -**before** deploying any Pod in 4e below. Everything a Service needs -(ports, selectors, which of the four apply) comes from the plan parsed in -resolve-recipe.md, so none of this waits on a Pod existing. Creating -Services late is the actual failure mode this guards against: a role's -head Pod binds its own rendezvous/DP-RPC server to its Service's DNS name -within seconds of starting, and with `restartPolicy: Never` a bind that -fails because the Service doesn't exist yet never gets retried. +Create every Service that applies -- see [services.md](services.md) for +which, and why this must happen **before** deploying any Pod in 4e below. +Everything a Service needs (ports, selectors, which of the four apply) +comes from the plan parsed in resolve-recipe.md, so none of this waits on a +Pod existing. ## 4e. Deploy pod(s) @@ -153,7 +148,7 @@ deploy_pod() { TEMPLATE_GPU="$gpu" TEMPLATE_PVC="$PVC_NAME" TEMPLATE_FSGROUP="$FS_GROUP" \ TEMPLATE_CLIENT_PORT="$CLIENT_PORT" TEMPLATE_ATTN_ROLE="$attn_role" TEMPLATE_FFN_ROLE="$ffn_role" \ TEMPLATE_EXTRA_ENV="$extra_env" \ - envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_GPU} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_ATTN_ROLE} ${TEMPLATE_FFN_ROLE} ${TEMPLATE_EXTRA_ENV} ${VLLM_USE_V2_MODEL_RUNNER}' \ + envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_GPU} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_ATTN_ROLE} ${TEMPLATE_FFN_ROLE} ${TEMPLATE_EXTRA_ENV}' \ < templates/pod.yaml | kubectl apply -f - } ``` @@ -182,13 +177,8 @@ capture dominate): ```bash for pod_name in ; do - deadline=$(( $(date +%s) + 900 )) - until [ "$(kubectl get pod "$pod_name" -o jsonpath='{.status.phase}' 2>/dev/null)" = "Running" ]; do - phase="$(kubectl get pod "$pod_name" -o jsonpath='{.status.phase}' 2>/dev/null || true)" - [ "$phase" = "Failed" ] && { kubectl logs "$pod_name" --tail=50; exit 1; } - [ "$(date +%s)" -ge "$deadline" ] && { kubectl describe pod "$pod_name" | tail -40; exit 1; } - sleep 10 - done + kubectl wait pod "$pod_name" --for=jsonpath='{.status.phase}'=Running --timeout=15m \ + || { kubectl describe pod "$pod_name" | tail -40; exit 1; } done for pod_name in ; do @@ -197,7 +187,6 @@ for pod_name in ; do logs="$(kubectl logs "pod/$pod_name" 2>/dev/null || true)" echo "$logs" | grep -q "pod ${pod_name} READY" && break echo "$logs" | grep -q "ERROR:" && { kubectl logs "$pod_name" --tail=200; exit 1; } - [ "$(kubectl get pod "$pod_name" -o jsonpath='{.status.phase}' 2>/dev/null)" = "Failed" ] && { kubectl logs "$pod_name" --tail=200; exit 1; } [ "$(date +%s)" -ge "$deadline" ] && { kubectl logs "$pod_name" --tail=200; exit 1; } sleep 10 done @@ -208,7 +197,5 @@ Only report the endpoint ready once every pod in the plan has printed its own `READY` line -- one pod becoming ready doesn't imply the others did. Startup markers are keyed on the fixed log names every colocation recipe -uses today (`attn.log` -> `Application startup complete`, or, for a worker -shard, the EngineCore init-complete line; `ffn.log` -> `AFD FFN EngineCore -started`). A recipe using different log names needs `templates/pod.yaml` -updated. +uses today -- see `templates/pod.yaml`'s `ready_marker()` for the exact +lines. A recipe using different log names needs that template updated. diff --git a/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md b/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md index 4a61922f0..72cc7f1f3 100644 --- a/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md +++ b/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md @@ -6,16 +6,15 @@ Tell the caller: the endpoint `http://vllm-service:${CLIENT_PORT}`, its `--max-num-batched-tokens`, and the node(s) it landed on: ```bash -kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}' +kubectl get pods -l app=afd-recipe -o wide ``` Every node matters if a follow-up pod needs to attach the same `ReadWriteOnce` PVC -- it can only attach from that node. -If the plan spans more than one pod, repeat the experimental caveat from -[resolve-recipe.md](resolve-recipe.md): correct AFD/DP flags and port -exposure, but cross-pod `P2pNcclAFDConnector` throughput is not -upstream-validated. +If the plan spans more than one pod, repeat the verification caveat from +[resolve-recipe.md](resolve-recipe.md): verified for DeepSeek-V2-Lite +`2A2F` only, every other cross-pod topology is still unverified. # 6. Teardown diff --git a/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md index 184cac6e1..ac7753d39 100644 --- a/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md +++ b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md @@ -25,8 +25,9 @@ isn't obvious) or a literal in-container path (e.g. PVC. Tell the caller which form it is. Set `CLIENT_PORT` to the `--port` value on the attention (or sole, for a -single-pod recipe) `vllm serve` block -- step 2 must rebind and expose -whatever value is actually there. +single-pod recipe) `vllm serve` block -- `templates/pod.yaml` rebinds this +port off loopback and `vllm-service` (step 4d, see +[services.md](services.md)) exposes it, whatever value is actually there. ## Read the placement plan -- never ask @@ -72,8 +73,10 @@ For each entry, derive (used by [deploy.md](deploy.md) and else `head`. - **`afd-ffn-node-role`**: symmetric, `none` unless `POD` starts with `FFN_`. -Treat any placement spanning more than one Pod as **experimental**: upstream -documents cross-node `P2pNcclAFDConnector` use as "not established by the -current recipes ... treated as unverified" -(`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`). Deploy it as written, but -don't present throughput as validated. +Upstream verifies cross-node `P2pNcclAFDConnector` placement specifically for +DeepSeek-V2-Lite `2A2F` with the FFN ranks on separate hosts, over TCP (ENA) +and over EFA; every other topology -- including the qwen3.5 3-pod plan above +-- is still **unverified** +(`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`). Deploy any placement as +written, but only present throughput as validated for that one verified +case. diff --git a/.agents/skills/deploy-afd-k8s/references/services.md b/.agents/skills/deploy-afd-k8s/references/services.md index 3a59b9ecd..3aa423a8d 100644 --- a/.agents/skills/deploy-afd-k8s/references/services.md +++ b/.agents/skills/deploy-afd-k8s/references/services.md @@ -25,19 +25,13 @@ CLIENT_PORT="$CLIENT_PORT" envsubst '${CLIENT_PORT}' \ Internal only, carries AFD rendezvous traffic to whichever pod holds `afd-ffn-node-role: head`. Must exist before any multi-pod deploy, since the recipe's own `AFD_CONNECTOR_HOST` default now points at this name directly. -Exposes the control-plane port plus one derived port per FFN rank -(`[AFD_CONNECTOR_PORT, AFD_CONNECTOR_PORT + NUM_FFN_RANKS]` -- read -`AFD_CONNECTOR_PORT` off the recipe's own shell default and `NUM_FFN_RANKS` -off its `"num_ffn_ranks"` JSON config key, the *total* rank count for the -FFN role across all pods, not a per-pod count): +Traffic to a headless Service goes straight to the pod IP, and there's one +rendezvous port, so a single static port entry is enough -- read +`AFD_CONNECTOR_PORT` off the recipe's own shell default: ```bash -FFN_PORT_START="$(grep -oE 'AFD_CONNECTOR_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" -NUM_FFN_RANKS="$(grep -oE '"num_ffn_ranks":[[:space:]]*[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+$' | head -1)" -PORT_ENTRIES="$(for p in $(seq "$FFN_PORT_START" "$((FFN_PORT_START + NUM_FFN_RANKS))"); do - printf '\n - {name: p%s, port: %s, targetPort: %s}' "$p" "$p" "$p" -done)" -PORT_ENTRIES="$PORT_ENTRIES" envsubst '${PORT_ENTRIES}' \ +AFD_CONNECTOR_PORT="$(grep -oE 'AFD_CONNECTOR_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" +AFD_CONNECTOR_PORT="$AFD_CONNECTOR_PORT" envsubst '${AFD_CONNECTOR_PORT}' \ < templates/service-ffn-p2p.yaml | kubectl apply -f - ``` @@ -49,17 +43,12 @@ Carries that role's DP-RPC coordination traffic to its head pod. One per split role, generic over role via `templates/service-dp-coord.yaml`: ```bash -TEMPLATE_ROLE=attn TEMPLATE_PORT=13345 \ +TEMPLATE_ROLE=attn +TEMPLATE_PORT="$(grep -oE 'ATTENTION_DP_RPC_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" +TEMPLATE_ROLE="$TEMPLATE_ROLE" TEMPLATE_PORT="$TEMPLATE_PORT" \ envsubst '${TEMPLATE_ROLE} ${TEMPLATE_PORT}' < templates/service-dp-coord.yaml | kubectl apply -f - -# symmetrically: TEMPLATE_ROLE=ffn TEMPLATE_PORT=13346, if FFN is ever split ``` All three internal Services (`vllm-ffn-p2p-service`, `vllm-attn-dp-service`, -`vllm-ffn-dp-service`) are headless (`clusterIP: None`) -- the role's head -pod *binds* its rendezvous/DP-RPC server to that name, and a normal -ClusterIP has no real interface to bind to, only to connect through. Their -templates also set `publishNotReadyAddresses: true`: headless-Service DNS -by default only lists Pods that have already passed readiness, but a head -pod resolving its own Service name to bind is inherently not-yet-ready at -that moment -- without this flag, any future readinessProbe added to -`templates/pod.yaml` would deadlock every head pod at startup. +`vllm-ffn-dp-service`) are headless with `publishNotReadyAddresses: true` -- +see the comments in their own templates for why. diff --git a/.agents/skills/deploy-afd-k8s/templates/pod.yaml b/.agents/skills/deploy-afd-k8s/templates/pod.yaml index 997e90be4..f85e8261d 100644 --- a/.agents/skills/deploy-afd-k8s/templates/pod.yaml +++ b/.agents/skills/deploy-afd-k8s/templates/pod.yaml @@ -35,32 +35,21 @@ spec: set -euo pipefail RECIPE_SCRIPT=/recipe/recipe.sh mkdir -p /work/bin - # Resolve the real uv BEFORE putting the shim on PATH (afterwards - # `command -v uv` would find the shim and recurse). - REAL_UV="$(command -v uv || true)" - printf '#!/bin/bash\nif [ "$1" = "run" ]; then\n shift\n exec "$@"\nfi\nREAL_UV="%s"\n[ -n "$REAL_UV" ] || { echo "uv not found in image" >&2; exit 127; }\nexec "$REAL_UV" "$@"\n' "$REAL_UV" > /work/bin/uv + # Recipes only ever call `uv run `; drop the `run` and exec + # the rest directly instead of invoking uv at all. + printf '#!/bin/bash\nshift\nexec "$@"\n' > /work/bin/uv chmod +x /work/bin/uv export PATH="/work/bin:$PATH" echo "=== launching $(basename "$RECIPE_SCRIPT") (POD=${POD:-} attn=${AFD_ATTN_NODE_ROLE} ffn=${AFD_FFN_NODE_ROLE}) ===" - # Rebind the client-facing port off loopback: the recipe binds - # :${TEMPLATE_CLIENT_PORT} on 127.0.0.1 (normally driven from the - # same host), but the Service needs 0.0.0.0. Every internal worker - # port stays on loopback. + # Rebind the client-facing port off loopback: every `--host + # 127.0.0.1` on the branch is an API-server flag (the recipe binds + # :${TEMPLATE_CLIENT_PORT} on 127.0.0.1), but the Service needs + # 0.0.0.0. RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" - awk ' - { line[NR] = $0 } - END { - for (i = 1; i <= NR; i++) { - l = line[i] - if (l ~ /--host 127\.0\.0\.1/ && i < NR && line[i+1] ~ /--port ${TEMPLATE_CLIENT_PORT}/) { - gsub(/127\.0\.0\.1/, "0.0.0.0", l) - } - print l - } - }' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" + sed 's/--host 127\.0\.0\.1/--host 0.0.0.0/' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" cd /work bash "$PATCHED_SCRIPT" & @@ -145,20 +134,8 @@ spec: value: "${TEMPLATE_MODEL}" - name: HF_HOME value: "/models/.hf_home" - - name: XDG_CACHE_HOME - value: "/work/xdg" - name: TORCHINDUCTOR_CACHE_DIR value: "/work/inductor" - - name: TRITON_CACHE_DIR - value: "/work/triton" - - name: VLLM_CACHE_ROOT - value: "/work/vllm" - - name: UV_CACHE_DIR - value: "/work/uv" - - name: VLLM_LOGGING_LEVEL - value: "INFO" - - name: VLLM_USE_V2_MODEL_RUNNER - value: "${VLLM_USE_V2_MODEL_RUNNER}" - name: AFD_ATTN_NODE_ROLE value: "${TEMPLATE_ATTN_ROLE}" - name: AFD_FFN_NODE_ROLE @@ -169,6 +146,7 @@ spec: secretKeyRef: name: hf-token-secret key: token + optional: true resources: requests: nvidia.com/gpu: "${TEMPLATE_GPU}" diff --git a/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml b/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml index 216e6f3f2..b327f8886 100644 --- a/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml +++ b/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml @@ -17,4 +17,6 @@ spec: app: afd-recipe afd-ffn-node-role: head ports: - # ${PORT_ENTRIES} + - name: rendezvous + port: ${AFD_CONNECTOR_PORT} + targetPort: ${AFD_CONNECTOR_PORT} diff --git a/recipe/gpu/P2pNcclAFDConnector/README.md b/recipe/gpu/P2pNcclAFDConnector/README.md index 43bbb8956..58862a6f0 100644 --- a/recipe/gpu/P2pNcclAFDConnector/README.md +++ b/recipe/gpu/P2pNcclAFDConnector/README.md @@ -15,7 +15,7 @@ for the connector's rank mapping and configuration contract. │ ├── prefill_decode_colocation/ # 2A2F, DP/TP variants, eager/graph │ └── prefill_decode_disaggregation/ # 2P1A1F └── qwen3_5_122b_a10b_fp8/ - └── prefill_decode_colocation/ # 4A2F, graph + └── prefill_decode_colocation/ # multipod_2a_2a_2f_graph.sh: 4A2F split across 2 attention pods, graph ``` Each model directory holds its own recipe scripts; open its `README.md` @@ -45,42 +45,21 @@ weights, ports) and the benchmark command for that model. ## Deploying on Kubernetes Use the **`deploy-afd-k8s`** skill (`.agents/skills/deploy-afd-k8s/SKILL.md`) -to stand up a recipe from this folder on a Kubernetes/OpenShift cluster. It -provisions the model PVC, mounts the chosen recipe script into a serve pod -via a ConfigMap (no image rebuild needed to try an edited or new recipe), -and waits until an OpenAI-compatible endpoint answers. +to stand up a recipe from this folder on a Kubernetes/OpenShift cluster -- +see that skill for the full contract (PVC/ConfigMap/pod/Service lifecycle, +scope, and requirements). The skill requires the recipe path (e.g. `recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh`) plus the image, model id, and PVC name it needs. Placement is never asked -for: the skill derives `GPU_COUNT` itself for a plain single-pod recipe, and -for a multi-pod recipe it reads the exact per-pod placement straight off the -recipe's own header comment (see below). - -**Scope.** The skill only covers `prefill_decode_colocation` recipes (and -their `baseline*` counterparts) under `recipe/gpu/P2pNcclAFDConnector/**`. -`prefill_decode_disaggregation` recipes are out of scope for it -- they need -a NIXL-enabled image and a proxy, and rely on a `SCRIPT_DIR`-relative path -that breaks once the script is mounted flat into the pod. Run those locally -instead, or ask for a different workflow. - -**Single-node.** Any recipe deploys as one pod holding every rank -- this is -the simple, non-experimental path. The skill computes `GPU_COUNT` itself from -the script's `CUDA_VISIBLE_DEVICES` blocks and deploys a single `vllm-pod` / -`vllm-service` pair. - -**Multi-node.** A recipe that can spread attention/FFN ranks across more -than one pod says so directly, via a top-of-file comment describing the -placement (e.g. `qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh`, -`deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh`). -The skill reads that comment and deploys exactly what it describes -- it -never asks for or guesses a placement. Recipes without such a comment -deploy as a single pod. For a multi-pod plan, the skill creates one pod per -entry, an internal FFN rendezvous Service whenever the plan spans more than -one pod, and DP-coordination Services for any role that is itself split -across pods -- all of these Services are created *before* any pod, since a -role's head pod binds its own rendezvous server to its Service's name -within seconds of starting. +for: a recipe without a top-of-file placement comment deploys as a single +pod; a recipe whose header describes a per-pod placement (e.g. +`qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh`, +`deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh`) +deploys exactly that many pods with exactly that per-pod setup -- see the +skill's `resolve-recipe.md` for the header convention. +`prefill_decode_disaggregation` recipes are out of scope for the skill; run +those locally instead. Pods and Services are left running after deployment so weights stay warm for follow-up runs; see the skill's teardown step to tear them down explicitly. From ebfd3d5ae9d2265c405bd19faabd9929f4cde072 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Mon, 5 Oct 2026 12:49:26 +0300 Subject: [PATCH 13/15] fix of an incorrect model path Signed-off-by: Shlomit Koyfman --- .../references/resolve-recipe.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md index ac7753d39..633a4dcc9 100644 --- a/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md +++ b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md @@ -20,9 +20,22 @@ caps how much prefill work a single step can absorb. Resolve `MODEL_ID`: read the script's `MODEL_PATH` default. It is either a HF Hub id (map the recipe's model directory name to one, e.g. `deepseek_v2_lite` -> `deepseek-ai/DeepSeek-V2-Lite`; ask if the mapping -isn't obvious) or a literal in-container path (e.g. -`/path/model_weights/Qwen3.5-122B-A10B-FP8`) when weights are staged on the -PVC. Tell the caller which form it is. +isn't obvious) or a **placeholder** path of the form +`/path/model_weights/` -- every colocation recipe uses exactly this +literal `/path/` stand-in today, regardless of what's actually staged on any +given cluster's PVC. Never pass this through verbatim: it names only the +model directory (``), not a real in-container path. Derive the real +`MODEL_ID` as `/`, where `` is the PVC mount +point fixed by `templates/pod.yaml` (`/models`) -- e.g. +`/path/model_weights/Qwen3.5-122B-A10B-FP8` resolves to +`/models/Qwen3.5-122B-A10B-FP8`. Before deploying, confirm that directory +actually exists on the target `PVC_NAME` (if it already exists, `kubectl +exec` into any pod already mounting it, e.g. `ls /models`, or spin up a +short-lived busybox pod mounting the PVC) -- don't rely on the recipe's +claim; a mismatch here fails fast inside the pod with a confusing +HF-repo-id validation error instead of a clear message. + +Tell the caller which form it is, and the resolved value if derived. Set `CLIENT_PORT` to the `--port` value on the attention (or sole, for a single-pod recipe) `vllm serve` block -- `templates/pod.yaml` rebinds this From 1dad15b503b78ce8da29b8bf92c1451f1e3e5629 Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Thu, 8 Oct 2026 19:28:48 +0300 Subject: [PATCH 14/15] add readiness probe Signed-off-by: Shlomit Koyfman --- .../references/report-and-teardown.md | 4 --- .../references/resolve-recipe.md | 21 +++++------ .../skills/deploy-afd-k8s/templates/pod.yaml | 35 ++++++++++++++++--- 3 files changed, 38 insertions(+), 22 deletions(-) diff --git a/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md b/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md index 72cc7f1f3..70fbb7e03 100644 --- a/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md +++ b/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md @@ -12,10 +12,6 @@ kubectl get pods -l app=afd-recipe -o wide Every node matters if a follow-up pod needs to attach the same `ReadWriteOnce` PVC -- it can only attach from that node. -If the plan spans more than one pod, repeat the verification caveat from -[resolve-recipe.md](resolve-recipe.md): verified for DeepSeek-V2-Lite -`2A2F` only, every other cross-pod topology is still unverified. - # 6. Teardown Left running by design, so weights stay warm -- report what's up and how to diff --git a/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md index 633a4dcc9..84e4ac2a9 100644 --- a/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md +++ b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md @@ -50,19 +50,22 @@ single-pod recipe: one pod, `POD` unset, both roles run unconditionally -- continue to [deploy.md](deploy.md) with no plan. If it does describe a placement across multiple pods, read and interpret -that description directly -- don't assume one fixed grammar. Both recipes -written so far happen to spell it the same way, one line per pod in file -order: +that description directly. +Example of a placement description: ``` # For pod 1: POD=ATTENTION_0 # For pod 2: POD=ATTENTION_1, ATTENTION_HEADLESS=1, ATTENTION_DP_START_RANK=2 # For pod 3: POD=FFN_0 ``` -but a future recipe may instead describe the same plan in prose (e.g. "runs +Or +``` +"runs on 3 pods: the first two run ATTENTION_0 and ATTENTION_1 -- the second one -headless, starting at data-parallel rank 2 -- and the third runs FFN_0"). +headless, starting at data-parallel rank 2 -- and the third runs FFN_0" +``` + Whatever the phrasing, read it and extract, per pod, in the order the comment presents them: @@ -85,11 +88,3 @@ For each entry, derive (used by [deploy.md](deploy.md) and which case `worker` if that line's `ATTENTION_HEADLESS` override is `1`, else `head`. - **`afd-ffn-node-role`**: symmetric, `none` unless `POD` starts with `FFN_`. - -Upstream verifies cross-node `P2pNcclAFDConnector` placement specifically for -DeepSeek-V2-Lite `2A2F` with the FFN ranks on separate hosts, over TCP (ENA) -and over EFA; every other topology -- including the qwen3.5 3-pod plan above --- is still **unverified** -(`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`). Deploy any placement as -written, but only present throughput as validated for that one verified -case. diff --git a/.agents/skills/deploy-afd-k8s/templates/pod.yaml b/.agents/skills/deploy-afd-k8s/templates/pod.yaml index f85e8261d..225524486 100644 --- a/.agents/skills/deploy-afd-k8s/templates/pod.yaml +++ b/.agents/skills/deploy-afd-k8s/templates/pod.yaml @@ -54,6 +54,7 @@ spec: cd /work bash "$PATCHED_SCRIPT" & RECIPE_PID=$! + echo "$RECIPE_PID" > /work/recipe.pid # Which log(s) this pod will actually produce, and this pod's own # readiness marker per log -- both keyed on this pod's node-role @@ -65,20 +66,19 @@ spec: fail() { echo "ERROR: $*" + rm -f /work/ready for log in $LOGS; do echo "----- tail $log -----" tail -n 100 "/work/$log" 2>/dev/null || echo "(missing)" done - sleep infinity + exit 1 } [ -n "$LOGS" ] || fail "this pod's node-role labels are both 'none' -- nothing to run" # A worker (headless) shard of a split role never starts an API # server, so it never logs "Application startup complete" -- fall - # back to the EngineCore's own init-complete line instead. This - # fallback marker is unverified: not yet observed on a live headless - # run, only inferred from the vLLM startup sequence. + # back to the EngineCore's own init-complete line instead. ready_marker() { case "$1" in ffn.log) echo "AFD FFN EngineCore started" ;; @@ -124,7 +124,17 @@ spec: fi echo "=== pod ${TEMPLATE_POD} READY ===" - sleep infinity + touch /work/ready + + echo "=== monitoring serving process (pid $RECIPE_PID) ===" + while true; do + sleep 10 + kill -0 "$RECIPE_PID" 2>/dev/null || fail "serving process exited unexpectedly" + if [ "$AFD_ATTN_NODE_ROLE" = "head" ]; then + curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1 \ + || fail "attention server health check failed" + fi + done env: - name: USER value: "vllm" @@ -147,6 +157,21 @@ spec: name: hf-token-secret key: token optional: true + readinessProbe: + exec: + command: + - /bin/bash + - -c + - | + [ -f /work/ready ] || exit 1 + kill -0 "$(cat /work/recipe.pid)" 2>/dev/null || exit 1 + if [ "$AFD_ATTN_NODE_ROLE" = "head" ]; then + curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1 + fi + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 resources: requests: nvidia.com/gpu: "${TEMPLATE_GPU}" From f51132d30c4bb04476fe2f75d41a6e0c327bab7b Mon Sep 17 00:00:00 2001 From: Shlomit Koyfman Date: Sun, 11 Oct 2026 16:34:10 +0300 Subject: [PATCH 15/15] should not pass --data-parallel-start-rank 0 for the coordinator pod since vllm 0.30.0 Signed-off-by: Shlomit Koyfman --- .../prefill_decode_colocation/multipod_2a_2a_2f_graph.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh index dcea163a0..c7bf95452 100644 --- a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh @@ -23,11 +23,16 @@ if [[ "$ATTENTION_HEADLESS" == "1" ]]; then HEADLESS_ARGS=(--headless) fi +DP_START_RANK_ARGS=() +if [[ "$ATTENTION_DP_START_RANK" != "0" ]]; then + DP_START_RANK_ARGS=(--data-parallel-start-rank "$ATTENTION_DP_START_RANK") +fi + if [[ "$POD" == ATTENTION_* ]]; then CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ --data-parallel-size 4 \ --data-parallel-size-local 2 \ - --data-parallel-start-rank "$ATTENTION_DP_START_RANK" \ + "${DP_START_RANK_ARGS[@]}" \ --data-parallel-address "$ATTENTION_DP_ADDRESS" \ --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ --tensor-parallel-size 1 \