diff --git a/.agents/skills/deploy-afd-k8s/SKILL.md b/.agents/skills/deploy-afd-k8s/SKILL.md new file mode 100644 index 000000000..f51ae4190 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/SKILL.md @@ -0,0 +1,87 @@ +--- +name: deploy-afd-k8s +description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is read directly off the recipe's own top-of-file comment block (however that block describes it, strict or prose): a recipe whose comment describes a per-pod placement deploys exactly that many pods with exactly that per-pod setup; a recipe without one deploys as a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e). +--- + +# Deploy an AFD GPU recipe on Kubernetes + +Stand up a serve pod (or pods) running a local AFD recipe script on a +Kubernetes/OpenShift cluster and leave a reachable OpenAI-compatible +endpoint behind. + +## Contract + +On success this skill guarantees, in the target namespace: + +| | | +|---|---| +| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`) | +| Serving | the model named by `MODEL_ID` | +| Pod(s) / Service | `vllm-pod`, or one pod per pod the recipe's header comment describes (named `vllm-pod-`); Service `vllm-service`, labels `app=afd-recipe,role=serve` | +| Model cache | `PVC_NAME`, mounted at `/models`, `HF_HOME=/models/.hf_home` | +| Left running | yes, deliberately -- so weights stay warm for follow-up runs | + +Whenever a recipe's header spans more than one pod, two more internal-only +Services may also exist (`vllm-ffn-p2p-service` always; `vllm-attn-dp-service` +and/or `vllm-ffn-dp-service` only if that role is itself split across more +than one pod) -- callers never talk to these directly. See +[services.md](references/services.md). + +## Scope + +- Recipes under `recipe/gpu/P2pNcclAFDConnector/**` (GPU only, not NPU), + **`prefill_decode_colocation` only** (attention + FFN workers, no prefill + split). + `prefill_decode_disaggregation` recipes are **out of scope**: they need a + NIXL-enabled image plus a proxy, and resolve a `SCRIPT_DIR`-relative path + that this skill's flat script mount (step 4b) breaks. Ask the user for a + different workflow for disaggregation. +- Recipes are hand-written shell scripts, not JSON configs -- there is no + generator step and no manifest to read. +- A recipe is **placement-driven** (more than one pod) exactly when its + top-of-file comment block describes a per-pod placement -- read that + block, don't infer shape from flags. Today's recipes spell it as one + `# For pod N: POD=...` line per pod, but the comment isn't guaranteed to + use that exact grammar going forward. See + [resolve-recipe.md](references/resolve-recipe.md). +- Not for `tools/benchmarks/decode_bench_server.sh` (local, non-k8s). + +## Requirements + +- A live, authenticated `kubectl`/`oc` session able to create/delete Pods, + Services, ConfigMaps, and PVCs in the target namespace. +- `envsubst` (from `gettext`) locally. +- An image, pushed where the cluster can pull it. Ask the user whether to + use an image they already have, or to build one from `docker/Dockerfile.ci` + and push it to a registry they provide: + + ```bash + IMAGE=/: + docker build -t "$IMAGE" -f docker/Dockerfile.ci . + docker push "$IMAGE" + ``` + + The image supplies only the `afd-plugin` install. The recipe script is + mounted fresh from local disk on every run (step 4b), so editing or adding + a recipe never requires a rebuild. +- An `hf-token-secret` Secret with a `token` key, needed when `MODEL_ID` is + a HF Hub repo id (not needed when weights are already staged on the PVC): + ```bash + kubectl create secret generic hf-token-secret --from-literal=token= + ``` +- Nodes with enough free GPUs for `GPU_COUNT` (per pod). + +## Workflow + +Load these in order; each covers one phase and nothing else. Services must +exist **before** any Pod is deployed -- see +[services.md](references/services.md) for why: + +| Reference | Covers | +|---|---| +| [resolve-recipe.md](references/resolve-recipe.md) | Step 1: read the recipe's serve-block fields and model id; read its placement (or lack of one) from its header comment -- never ask. | +| [deploy.md](references/deploy.md) | Steps 2-4f: GPU_COUNT, confirm, PVC, ConfigMap, fsGroup, Services (step 4d -- which of the four, see [services.md](references/services.md)), pod(s) via `templates/pod.yaml`, wait for Running and for each pod's own readiness line. | +| [report-and-teardown.md](references/report-and-teardown.md) | Step 5: report the endpoint/model/nodes. Step 6: teardown (pods/Services only -- never the PVC unless asked). | + +All k8s manifests live in `templates/*.yaml`, applied via `envsubst | kubectl +apply -f -`; no manifest is inlined in these docs. diff --git a/.agents/skills/deploy-afd-k8s/agents/openai.yaml b/.agents/skills/deploy-afd-k8s/agents/openai.yaml new file mode 100644 index 000000000..e2f15ea63 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Deploy AFD on Kubernetes" + short_description: "Deploy an AFD GPU recipe on k8s and expose a ready endpoint" + default_prompt: "Use $deploy-afd-k8s to deploy an AFD GPU colocation recipe on Kubernetes and wait until the endpoint is ready." diff --git a/.agents/skills/deploy-afd-k8s/references/deploy.md b/.agents/skills/deploy-afd-k8s/references/deploy.md new file mode 100644 index 000000000..eccd5423a --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/references/deploy.md @@ -0,0 +1,201 @@ +# 2-4. Compute GPU_COUNT, confirm, and deploy + +## 2. GPU_COUNT + +Plain single-pod recipe (no plan from [resolve-recipe.md](resolve-recipe.md)): +union of every literal `CUDA_VISIBLE_DEVICES=` in the script. + +```bash +GPU_COUNT="$(grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' "$RECIPE_SCRIPT_PATH" \ + | cut -d= -f2 | tr ',' '\n' | sort -u | wc -l)" +``` + +Per plan entry: count the devices in the literal `CUDA_VISIBLE_DEVICES=` inside +the recipe's `if` block that tests this pod's role (`ATTENTION` or `FFN` -- +whichever prefix `POD` starts with). Every pod sharing a role shares that +literal, so this is a plain text extraction, not a per-pod computation: + +```bash +role_block() { # $1=script $2=ATTENTION|FFN + awk -v role="$2" ' + /if.*\$POD/ && $0 ~ role { inblock=1; next } + inblock && /^fi/ { inblock=0 } + inblock { print } + ' "$1" +} +devices="$(role_block "$RECIPE_SCRIPT_PATH" "$ROLE" \ + | grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' | head -1 | cut -d= -f2)" +GPU_COUNT=$(( $(grep -o ',' <<<"$devices" | wc -l) + 1 )) +``` + +## 3. Confirm before touching the cluster + +State and confirm: target cluster/namespace, `AFD_PLUGIN_IMAGE`, `MODEL_ID`, +`PVC_NAME`, `CLIENT_PORT`, `RECIPE_SCRIPT_PATH`, and either `GPU_COUNT` +(plain recipe) or the plan as a table of `pod_name | role | GPU_COUNT` +(placement-driven recipe, straight from resolve-recipe.md -- no shape to +negotiate, just restate what was parsed). Flag that step 4 **deletes any +existing Pod(s)** with the same name(s) about to be deployed. + +```bash +AFD_PLUGIN_IMAGE= +MODEL_ID= +PVC_NAME= +CLIENT_PORT= # from the script's --port; default 18305 +RECIPE_SCRIPT_PATH= +``` + +**Node placement is left to the scheduler by default.** Only add +`podAntiAffinity` forcing Pods onto separate nodes if the caller explicitly +asks to exercise cross-node placement -- add this to every Pod's +`metadata.labels` and `spec` in `templates/pod.yaml` before applying: + +```yaml + labels: + afd-multinode-group: afd-multinode + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - {key: afd-multinode-group, operator: In, values: ["afd-multinode"]} + topologyKey: kubernetes.io/hostname +``` + +## 4a. Model PVC + +Size for the model being served. An existing undersized PVC is reused +silently and fails mid-download, so set `MODEL_PVC_SIZE` deliberately. `PVC_ACCESS_MODE` is derived, not asked: +`ReadWriteMany` whenever the plan (if any) has more than one pod -- an RWO +volume can only attach from one node, and the second Pod deadlocks +`Pending` with `Multi-Attach error` otherwise. + +```bash +MODEL_PVC_SIZE=${MODEL_PVC_SIZE:-100Gi} +STORAGE_CLASS=${STORAGE_CLASS:-} +PVC_ACCESS_MODE=ReadWriteOnce # ReadWriteMany if the plan has >1 pod + +if kubectl get pvc "${PVC_NAME}" >/dev/null 2>&1; then + echo "PVC ${PVC_NAME} exists; reusing its warm HF_HOME cache for ${MODEL_ID}" + echo "NOTE: verify its size fits ${MODEL_ID}, and its accessModes include" + echo " ReadWriteMany if this plan has >1 pod -- an RWO reuse deadlocks the 2nd pod" +else + envsubst '${PVC_NAME} ${MODEL_PVC_SIZE} ${STORAGE_CLASS} ${PVC_ACCESS_MODE}' \ + < templates/pvc.yaml | grep -v 'storageClassName: *$' | kubectl apply -f - +fi +``` + +If an existing single-pod RWO PVC already holds downloaded weights you want +to reuse for a multi-pod plan, provision a new RWX PVC and `cp -a` the data +across (via a short-lived Pod pinned to the node that holds the RWO mount) +rather than re-downloading. + +## 4b. Recipe ConfigMap + +Lets the pod run any local recipe -- edited, uncommitted, or new -- without +rebuilding the image. Run from wherever `RECIPE_SCRIPT_PATH` resolves: + +```bash +kubectl create configmap afd-recipe-script \ + --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +## 4c. fsGroup + +Must be inside the namespace's allowed range or the pod is rejected at +admission. Look it up once, reuse for every pod: + +```bash +FS_GROUP="$(kubectl get ns "$(kubectl config view --minify -o jsonpath='{..namespace}')" \ + -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' 2>/dev/null \ + | cut -d/ -f1)" +FS_GROUP="${FS_GROUP:-1000}" +``` + +## 4d. Services + +Create every Service that applies -- see [services.md](services.md) for +which, and why this must happen **before** deploying any Pod in 4e below. +Everything a Service needs (ports, selectors, which of the four apply) +comes from the plan parsed in resolve-recipe.md, so none of this waits on a +Pod existing. + +## 4e. Deploy pod(s) + +One call per plan entry (or a single call with no overrides for a plain +recipe) against the one shared `templates/pod.yaml`: + +```bash +deploy_pod() { + local pod_name="$1" gpu="$2" attn_role="$3" ffn_role="$4" extra_env_pairs="$5" + # extra_env_pairs: space-separated VAR=value tokens straight off this + # plan entry's header line (including POD=... itself), empty for a plain + # recipe. Build the spliced env-entries block for templates/pod.yaml. + # Leading (not trailing) newlines keep the `# ${TEMPLATE_EXTRA_ENV}` + # placeholder line a harmless comment when this is empty, and valid + # sibling list items when it isn't. + local extra_env="" + for pair in $extra_env_pairs; do + extra_env="$extra_env + - {name: ${pair%%=*}, value: \"${pair#*=}\"}" + done + + kubectl delete pod "$pod_name" --ignore-not-found + + TEMPLATE_IMAGE="$AFD_PLUGIN_IMAGE" TEMPLATE_MODEL="$MODEL_ID" TEMPLATE_POD="$pod_name" \ + TEMPLATE_GPU="$gpu" TEMPLATE_PVC="$PVC_NAME" TEMPLATE_FSGROUP="$FS_GROUP" \ + TEMPLATE_CLIENT_PORT="$CLIENT_PORT" TEMPLATE_ATTN_ROLE="$attn_role" TEMPLATE_FFN_ROLE="$ffn_role" \ + TEMPLATE_EXTRA_ENV="$extra_env" \ + envsubst '${TEMPLATE_IMAGE} ${TEMPLATE_MODEL} ${TEMPLATE_POD} ${TEMPLATE_GPU} ${TEMPLATE_PVC} ${TEMPLATE_FSGROUP} ${TEMPLATE_CLIENT_PORT} ${TEMPLATE_ATTN_ROLE} ${TEMPLATE_FFN_ROLE} ${TEMPLATE_EXTRA_ENV}' \ + < templates/pod.yaml | kubectl apply -f - +} +``` + +Plain recipe, one pod, both roles head, no overrides: + +```bash +deploy_pod vllm-pod "$GPU_COUNT" head head "" +``` + +Placement-driven recipe, one call per entry parsed in +[resolve-recipe.md](resolve-recipe.md), e.g. the qwen3.5 3-pod plan: + +```bash +deploy_pod vllm-pod-attention-0 2 head none "POD=ATTENTION_0" +deploy_pod vllm-pod-attention-1 2 worker none "POD=ATTENTION_1 ATTENTION_HEADLESS=1 ATTENTION_DP_START_RANK=2" +deploy_pod vllm-pod-ffn-0 2 none head "POD=FFN_0" +``` + +## 4f. Wait for Running, then for each pod's own readiness line + +Bounded 15-minute wait per pod for `phase=Running` (unschedulable pods never +reach `Failed`, so an unbounded loop would spin forever), then poll logs for +`=== pod READY ===` (budget 60 minutes -- model download and graph +capture dominate): + +```bash +for pod_name in ; do + kubectl wait pod "$pod_name" --for=jsonpath='{.status.phase}'=Running --timeout=15m \ + || { kubectl describe pod "$pod_name" | tail -40; exit 1; } +done + +for pod_name in ; do + deadline=$(( $(date +%s) + 3600 )) + while true; do + logs="$(kubectl logs "pod/$pod_name" 2>/dev/null || true)" + echo "$logs" | grep -q "pod ${pod_name} READY" && break + echo "$logs" | grep -q "ERROR:" && { kubectl logs "$pod_name" --tail=200; exit 1; } + [ "$(date +%s)" -ge "$deadline" ] && { kubectl logs "$pod_name" --tail=200; exit 1; } + sleep 10 + done +done +``` + +Only report the endpoint ready once every pod in the plan has printed its +own `READY` line -- one pod becoming ready doesn't imply the others did. + +Startup markers are keyed on the fixed log names every colocation recipe +uses today -- see `templates/pod.yaml`'s `ready_marker()` for the exact +lines. A recipe using different log names needs that template updated. diff --git a/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md b/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md new file mode 100644 index 000000000..70fbb7e03 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/references/report-and-teardown.md @@ -0,0 +1,26 @@ +# 5. Report back + +Tell the caller: the endpoint `http://vllm-service:${CLIENT_PORT}`, +`MODEL_ID` (and whether it's a HF repo id or an in-container path), +`PVC_NAME`, `GPU_COUNT` (per pod, if more than one), the recipe deployed, +its `--max-num-batched-tokens`, and the node(s) it landed on: + +```bash +kubectl get pods -l app=afd-recipe -o wide +``` + +Every node matters if a follow-up pod needs to attach the same +`ReadWriteOnce` PVC -- it can only attach from that node. + +# 6. Teardown + +Left running by design, so weights stay warm -- report what's up and how to +remove it; do not delete unless asked: + +```bash +kubectl delete pod +kubectl delete svc vllm-service vllm-ffn-p2p-service vllm-attn-dp-service vllm-ffn-dp-service --ignore-not-found +``` + +The model PVC is intentionally not listed -- deleting it discards the warm +cache and forces a full re-download. diff --git a/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md new file mode 100644 index 000000000..84e4ac2a9 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/references/resolve-recipe.md @@ -0,0 +1,90 @@ +# 1. Resolve the recipe, its model, and its placement + +Take the recipe path from the user, or ask. It must be a colocation recipe +under `recipe/gpu/P2pNcclAFDConnector/**/prefill_decode_colocation/` (GPU +only; prefill/decode disaggregation recipes are out of scope -- see +`SKILL.md` Scope). + +Read the script and note, per `vllm serve` block (each backgrounded process +ending `> .log 2>&1 &`): + +- `CUDA_VISIBLE_DEVICES`, `--data-parallel-size`, `--tensor-parallel-size` +- the `"afd": {"role": ...}` block in `--additional-config` +- eager vs graph, and every other flag (`--enable-expert-parallel`, + `--max-num-seqs`, `--max-num-batched-tokens`, `--max-model-len`, + `--trust-remote-code`, host/port) + +Record `--max-num-batched-tokens` and report it to the caller later -- it +caps how much prefill work a single step can absorb. + +Resolve `MODEL_ID`: read the script's `MODEL_PATH` default. It is either a +HF Hub id (map the recipe's model directory name to one, e.g. +`deepseek_v2_lite` -> `deepseek-ai/DeepSeek-V2-Lite`; ask if the mapping +isn't obvious) or a **placeholder** path of the form +`/path/model_weights/` -- every colocation recipe uses exactly this +literal `/path/` stand-in today, regardless of what's actually staged on any +given cluster's PVC. Never pass this through verbatim: it names only the +model directory (``), not a real in-container path. Derive the real +`MODEL_ID` as `/`, where `` is the PVC mount +point fixed by `templates/pod.yaml` (`/models`) -- e.g. +`/path/model_weights/Qwen3.5-122B-A10B-FP8` resolves to +`/models/Qwen3.5-122B-A10B-FP8`. Before deploying, confirm that directory +actually exists on the target `PVC_NAME` (if it already exists, `kubectl +exec` into any pod already mounting it, e.g. `ls /models`, or spin up a +short-lived busybox pod mounting the PVC) -- don't rely on the recipe's +claim; a mismatch here fails fast inside the pod with a confusing +HF-repo-id validation error instead of a clear message. + +Tell the caller which form it is, and the resolved value if derived. + +Set `CLIENT_PORT` to the `--port` value on the attention (or sole, for a +single-pod recipe) `vllm serve` block -- `templates/pod.yaml` rebinds this +port off loopback and `vllm-service` (step 4d, see +[services.md](services.md)) exposes it, whatever value is actually there. + +## Read the placement plan -- never ask + +Look at the top-of-file comment block (above the first `vllm serve`/env-var +line). If it says nothing about per-pod placement, this is a plain +single-pod recipe: one pod, `POD` unset, both roles run unconditionally -- +continue to [deploy.md](deploy.md) with no plan. + +If it does describe a placement across multiple pods, read and interpret +that description directly. + +Example of a placement description: +``` +# For pod 1: POD=ATTENTION_0 +# For pod 2: POD=ATTENTION_1, ATTENTION_HEADLESS=1, ATTENTION_DP_START_RANK=2 +# For pod 3: POD=FFN_0 +``` + +Or +``` +"runs +on 3 pods: the first two run ATTENTION_0 and ATTENTION_1 -- the second one +headless, starting at data-parallel rank 2 -- and the third runs FFN_0" +``` + +Whatever the phrasing, read it and extract, per pod, in the order the +comment presents them: + +- that pod's `POD=` identity +- any further container-env overrides for that pod only (e.g. + `ATTENTION_HEADLESS=1`, `ATTENTION_DP_START_RANK=2`) + +The recipe already states ranks/ordering per pod -- there is nothing to +derive or ask about, regardless of how it's phrased. (A quick +`grep -E '^# For pod [0-9]+: '` is a fine way to spot today's convention at +a glance, but its absence doesn't mean "no plan" -- check the comment block +itself before concluding this is a plain single-pod recipe.) + +For each entry, derive (used by [deploy.md](deploy.md) and +[services.md](services.md)): + +- **k8s Pod name**: `vllm-pod-$(echo "$POD" | tr 'A-Z_' 'a-z-')`, e.g. + `vllm-pod-attention-0`. +- **`afd-attn-node-role`**: `none` unless `POD` starts with `ATTENTION_`, in + which case `worker` if that line's `ATTENTION_HEADLESS` override is `1`, + else `head`. +- **`afd-ffn-node-role`**: symmetric, `none` unless `POD` starts with `FFN_`. diff --git a/.agents/skills/deploy-afd-k8s/references/services.md b/.agents/skills/deploy-afd-k8s/references/services.md new file mode 100644 index 000000000..3aa423a8d --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/references/services.md @@ -0,0 +1,54 @@ +# Services + +Create as step 4d of [deploy.md](deploy.md) -- **before** deploying any Pod +(step 4e), not after (idempotent -- safe to re-apply either way). The +headless Services below are what a role's head Pod *binds* its own +rendezvous/DP-RPC server to during its own startup, within seconds of the +container starting; if that Service doesn't exist yet, the bind/resolve +fails fast and, since every pod has `restartPolicy: Never`, it never gets a +second chance. Which of the four to create is derived from the plan parsed +in [resolve-recipe.md](resolve-recipe.md), never asked. + +## `vllm-service` -- always + +Client-facing. Selects whichever pod holds `afd-attn-node-role: head` -- +true for the sole pod of a plain recipe, and for whichever pod is the +attention head of a placement plan. + +```bash +CLIENT_PORT="$CLIENT_PORT" envsubst '${CLIENT_PORT}' \ + < templates/service-client.yaml | kubectl apply -f - +``` + +## `vllm-ffn-p2p-service` -- iff the plan has more than one pod + +Internal only, carries AFD rendezvous traffic to whichever pod holds +`afd-ffn-node-role: head`. Must exist before any multi-pod deploy, since the +recipe's own `AFD_CONNECTOR_HOST` default now points at this name directly. +Traffic to a headless Service goes straight to the pod IP, and there's one +rendezvous port, so a single static port entry is enough -- read +`AFD_CONNECTOR_PORT` off the recipe's own shell default: + +```bash +AFD_CONNECTOR_PORT="$(grep -oE 'AFD_CONNECTOR_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" +AFD_CONNECTOR_PORT="$AFD_CONNECTOR_PORT" envsubst '${AFD_CONNECTOR_PORT}' \ + < templates/service-ffn-p2p.yaml | kubectl apply -f - +``` + +## `vllm-attn-dp-service` / `vllm-ffn-dp-service` -- iff that role is split + +Only when **more than one** plan entry's `POD` carries the same role (i.e. +that role is split across pods, not just placed in its own single pod). +Carries that role's DP-RPC coordination traffic to its head pod. One per +split role, generic over role via `templates/service-dp-coord.yaml`: + +```bash +TEMPLATE_ROLE=attn +TEMPLATE_PORT="$(grep -oE 'ATTENTION_DP_RPC_PORT:-[0-9]+' "$RECIPE_SCRIPT_PATH" | grep -oE '[0-9]+')" +TEMPLATE_ROLE="$TEMPLATE_ROLE" TEMPLATE_PORT="$TEMPLATE_PORT" \ + envsubst '${TEMPLATE_ROLE} ${TEMPLATE_PORT}' < templates/service-dp-coord.yaml | kubectl apply -f - +``` + +All three internal Services (`vllm-ffn-p2p-service`, `vllm-attn-dp-service`, +`vllm-ffn-dp-service`) are headless with `publishNotReadyAddresses: true` -- +see the comments in their own templates for why. diff --git a/.agents/skills/deploy-afd-k8s/templates/pod.yaml b/.agents/skills/deploy-afd-k8s/templates/pod.yaml new file mode 100644 index 000000000..225524486 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/pod.yaml @@ -0,0 +1,193 @@ +apiVersion: v1 +kind: Pod +metadata: + name: ${TEMPLATE_POD} + labels: + app: afd-recipe + role: serve + afd-attn-node-role: ${TEMPLATE_ATTN_ROLE} + afd-ffn-node-role: ${TEMPLATE_FFN_ROLE} +spec: + restartPolicy: Never + securityContext: + fsGroup: ${TEMPLATE_FSGROUP} + volumes: + - name: model-storage + persistentVolumeClaim: + claimName: ${TEMPLATE_PVC} + - name: dshm + emptyDir: + medium: Memory + sizeLimit: 16Gi + - name: work + emptyDir: {} + - name: recipe-script + configMap: + name: afd-recipe-script + containers: + - name: afd + image: ${TEMPLATE_IMAGE} + imagePullPolicy: Always + command: + - /bin/bash + - -c + - | + set -euo pipefail + RECIPE_SCRIPT=/recipe/recipe.sh + mkdir -p /work/bin + # Recipes only ever call `uv run `; drop the `run` and exec + # the rest directly instead of invoking uv at all. + printf '#!/bin/bash\nshift\nexec "$@"\n' > /work/bin/uv + chmod +x /work/bin/uv + export PATH="/work/bin:$PATH" + + echo "=== launching $(basename "$RECIPE_SCRIPT") (POD=${POD:-} attn=${AFD_ATTN_NODE_ROLE} ffn=${AFD_FFN_NODE_ROLE}) ===" + + # Rebind the client-facing port off loopback: every `--host + # 127.0.0.1` on the branch is an API-server flag (the recipe binds + # :${TEMPLATE_CLIENT_PORT} on 127.0.0.1), but the Service needs + # 0.0.0.0. + RECIPE_BASENAME="$(basename "$RECIPE_SCRIPT" .sh)" + PATCHED_SCRIPT="/work/${RECIPE_BASENAME}.patched.sh" + sed 's/--host 127\.0\.0\.1/--host 0.0.0.0/' "$RECIPE_SCRIPT" > "$PATCHED_SCRIPT" + + cd /work + bash "$PATCHED_SCRIPT" & + RECIPE_PID=$! + echo "$RECIPE_PID" > /work/recipe.pid + + # Which log(s) this pod will actually produce, and this pod's own + # readiness marker per log -- both keyed on this pod's node-role + # labels, not re-derived from $POD. Fixed to the attn.log/ffn.log + # names every colocation recipe uses today. + LOGS="" + [ "$AFD_ATTN_NODE_ROLE" != "none" ] && LOGS="$LOGS attn.log" + [ "$AFD_FFN_NODE_ROLE" != "none" ] && LOGS="$LOGS ffn.log" + + fail() { + echo "ERROR: $*" + rm -f /work/ready + for log in $LOGS; do + echo "----- tail $log -----" + tail -n 100 "/work/$log" 2>/dev/null || echo "(missing)" + done + exit 1 + } + + [ -n "$LOGS" ] || fail "this pod's node-role labels are both 'none' -- nothing to run" + + # A worker (headless) shard of a split role never starts an API + # server, so it never logs "Application startup complete" -- fall + # back to the EngineCore's own init-complete line instead. + ready_marker() { + case "$1" in + ffn.log) echo "AFD FFN EngineCore started" ;; + attn.log) + if [ "$AFD_ATTN_NODE_ROLE" = "worker" ]; then + echo "init engine (profile, create kv cache, warmup model) took" + else + echo "Application startup complete" + fi + ;; + esac + } + + echo "=== wait for every worker to report startup ===" + for log in $LOGS; do + marker="$(ready_marker "$log")" + echo "--- waiting on $log (marker: $marker) ---" + ready=0 + for i in $(seq 1 240); do + if [ -f "/work/$log" ] && grep -q "$marker" "/work/$log"; then + echo "$log: ready" + ready=1 + break + fi + kill -0 "$RECIPE_PID" 2>/dev/null || fail "recipe process exited before $log came up" + sleep 10 + done + [ "$ready" = "1" ] || fail "timed out waiting for $log" + done + + if [ "$AFD_ATTN_NODE_ROLE" = "head" ]; then + echo "--- waiting on attention server /health (127.0.0.1:${TEMPLATE_CLIENT_PORT}) ---" + endpoint_ready=0 + for i in $(seq 1 60); do + if curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1; then + echo "attention server: ready" + endpoint_ready=1 + break + fi + sleep 5 + done + [ "$endpoint_ready" = "1" ] || fail "attention server did not answer /health" + fi + + echo "=== pod ${TEMPLATE_POD} READY ===" + touch /work/ready + + echo "=== monitoring serving process (pid $RECIPE_PID) ===" + while true; do + sleep 10 + kill -0 "$RECIPE_PID" 2>/dev/null || fail "serving process exited unexpectedly" + if [ "$AFD_ATTN_NODE_ROLE" = "head" ]; then + curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1 \ + || fail "attention server health check failed" + fi + done + env: + - name: USER + value: "vllm" + - name: HOME + value: "/work/home" + - name: MODEL_PATH + value: "${TEMPLATE_MODEL}" + - name: HF_HOME + value: "/models/.hf_home" + - name: TORCHINDUCTOR_CACHE_DIR + value: "/work/inductor" + - name: AFD_ATTN_NODE_ROLE + value: "${TEMPLATE_ATTN_ROLE}" + - name: AFD_FFN_NODE_ROLE + value: "${TEMPLATE_FFN_ROLE}" + # ${TEMPLATE_EXTRA_ENV} + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: token + optional: true + readinessProbe: + exec: + command: + - /bin/bash + - -c + - | + [ -f /work/ready ] || exit 1 + kill -0 "$(cat /work/recipe.pid)" 2>/dev/null || exit 1 + if [ "$AFD_ATTN_NODE_ROLE" = "head" ]; then + curl -fsS "http://127.0.0.1:${TEMPLATE_CLIENT_PORT}/health" >/dev/null 2>&1 + fi + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + resources: + requests: + nvidia.com/gpu: "${TEMPLATE_GPU}" + cpu: "16" + memory: 128Gi + limits: + nvidia.com/gpu: "${TEMPLATE_GPU}" + cpu: "32" + memory: 200Gi + volumeMounts: + - name: model-storage + mountPath: /models + - name: dshm + mountPath: /dev/shm + - name: work + mountPath: /work + - name: recipe-script + mountPath: /recipe + readOnly: true diff --git a/.agents/skills/deploy-afd-k8s/templates/pvc.yaml b/.agents/skills/deploy-afd-k8s/templates/pvc.yaml new file mode 100644 index 000000000..19d28d4ea --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/pvc.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ${PVC_NAME} + labels: + app: afd-recipe +spec: + accessModes: + - ${PVC_ACCESS_MODE} + storageClassName: ${STORAGE_CLASS} + resources: + requests: + storage: ${MODEL_PVC_SIZE} diff --git a/.agents/skills/deploy-afd-k8s/templates/service-client.yaml b/.agents/skills/deploy-afd-k8s/templates/service-client.yaml new file mode 100644 index 000000000..9f32260c0 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/service-client.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Service +metadata: + name: vllm-service + labels: + app: afd-recipe + role: serve +spec: + selector: + app: afd-recipe + afd-attn-node-role: head + ports: + - name: http + port: ${CLIENT_PORT} + targetPort: ${CLIENT_PORT} diff --git a/.agents/skills/deploy-afd-k8s/templates/service-dp-coord.yaml b/.agents/skills/deploy-afd-k8s/templates/service-dp-coord.yaml new file mode 100644 index 000000000..c68f907a6 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/service-dp-coord.yaml @@ -0,0 +1,23 @@ +apiVersion: v1 +kind: Service +metadata: + name: vllm-${TEMPLATE_ROLE}-dp-service + labels: + app: afd-recipe + role: serve +spec: + # Headless for the same reason as vllm-ffn-p2p-service: the role's head + # pod binds --data-parallel-address to this name, so it must resolve to a + # real pod IP, not a virtual ClusterIP. + clusterIP: None + # The head pod resolves/binds this name during its own startup, before it + # can be "ready" -- without this, headless DNS would omit it until ready, + # deadlocking startup. + publishNotReadyAddresses: true + selector: + app: afd-recipe + afd-${TEMPLATE_ROLE}-node-role: head + ports: + - name: dp-rpc + port: ${TEMPLATE_PORT} + targetPort: ${TEMPLATE_PORT} diff --git a/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml b/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml new file mode 100644 index 000000000..b327f8886 --- /dev/null +++ b/.agents/skills/deploy-afd-k8s/templates/service-ffn-p2p.yaml @@ -0,0 +1,22 @@ +apiVersion: v1 +kind: Service +metadata: + name: vllm-ffn-p2p-service + labels: + app: afd-recipe + role: serve +spec: + # Must be headless: the FFN-head pod binds its own rendezvous server to + # this name, and a normal ClusterIP has no real interface to bind to. + clusterIP: None + # The FFN-head pod resolves/binds this name during its own startup, before + # it can be "ready" -- without this, headless DNS would omit it until + # ready, deadlocking startup. + publishNotReadyAddresses: true + selector: + app: afd-recipe + afd-ffn-node-role: head + ports: + - name: rendezvous + port: ${AFD_CONNECTOR_PORT} + targetPort: ${AFD_CONNECTOR_PORT} diff --git a/recipe/gpu/P2pNcclAFDConnector/README.md b/recipe/gpu/P2pNcclAFDConnector/README.md new file mode 100644 index 000000000..58862a6f0 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/README.md @@ -0,0 +1,65 @@ +# P2pNcclAFDConnector Recipes + +Launch scripts for running Attention-FFN Disaggregation (AFD) with +`P2pNcclAFDConnector`, a GPU point-to-point connector built on vLLM's +`PyNcclCommunicator`. See +[`docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md`](../../../docs/gpu/NCCL_P2P_CONNECTOR_USER_GUIDE.md) +for the connector's rank mapping and configuration contract. + +## Directory layout + +```text +. +├── deepseek_v2_lite/ +│ ├── README.md +│ ├── prefill_decode_colocation/ # 2A2F, DP/TP variants, eager/graph +│ └── prefill_decode_disaggregation/ # 2P1A1F +└── qwen3_5_122b_a10b_fp8/ + └── prefill_decode_colocation/ # multipod_2a_2a_2f_graph.sh: 4A2F split across 2 attention pods, graph +``` + +Each model directory holds its own recipe scripts; open its `README.md` +(where present) before running anything in it -- it documents prerequisites, +topology, ports, and known limitations for that model. + +Recipes under `prefill_decode_colocation/` background an attention worker and +an FFN worker (no prefill split). Recipes under `prefill_decode_disaggregation/` +add a separate prefill stage and a proxy in front of it -- see the caveat in +"Deploying on Kubernetes" below. + +## Running on a local host + +Run any script directly from the repository root, e.g.: + +```bash +export MODEL_PATH=/path/model_weights/DeepSeek-V2-Lite +bash recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh +``` + +Each script backgrounds its workers and writes per-worker logs (`attn.log`, +`ffn.log`, and for disaggregation `afd_prefill*.log`) into the current +directory. Wait for `Application startup complete` in each log before +sending traffic. See the model README for exact prerequisites (GPU count, +weights, ports) and the benchmark command for that model. + +## Deploying on Kubernetes + +Use the **`deploy-afd-k8s`** skill (`.agents/skills/deploy-afd-k8s/SKILL.md`) +to stand up a recipe from this folder on a Kubernetes/OpenShift cluster -- +see that skill for the full contract (PVC/ConfigMap/pod/Service lifecycle, +scope, and requirements). + +The skill requires the recipe path (e.g. +`recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh`) +plus the image, model id, and PVC name it needs. Placement is never asked +for: a recipe without a top-of-file placement comment deploys as a single +pod; a recipe whose header describes a per-pod placement (e.g. +`qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh`, +`deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh`) +deploys exactly that many pods with exactly that per-pod setup -- see the +skill's `resolve-recipe.md` for the header convention. +`prefill_decode_disaggregation` recipes are out of scope for the skill; run +those locally instead. + +Pods and Services are left running after deployment so weights stay warm for +follow-up runs; see the skill's teardown step to tear them down explicitly. diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md index 6be8f684c..4bd062659 100644 --- a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md @@ -27,11 +27,14 @@ End-to-end launch scripts for running DeepSeek-V2-Lite with the AFD ├── prefill_decode_disaggregation/ # prefill_decode_disaggregation, 2P1A1F topology │ ├── 2p1a1f_eager_dbo.sh │ └── 2p1a1f_graph_dbo.sh -└── prefill_decode_colocation/ # prefill_decode_colocation, 2A2F topology +└── prefill_decode_colocation/ # prefill_decode_colocation, 2A2F/4A4F topology ├── 2a2f_eager_dbo_dp1tp2.sh ├── 2a2f_eager_dbo_dp2tp1.sh ├── 2a2f_graph_dbo_dp1tp2.sh - └── 2a2f_graph_dbo_dp2tp1.sh + ├── 2a2f_graph_dbo_dp2tp1.sh + ├── 4a4f_eager_dbo_dp2tp2.sh + ├── 4a4f_graph_dbo_dp2tp2.sh + └── multipod_2a_2f_graph_dbo_dp1tp2.sh # 2a2f_graph_dbo_dp1tp2, split across 2 k8s pods ``` ### 1. Prefill/Decode Disaggregation — `1a1f` @@ -55,12 +58,22 @@ End-to-end launch scripts for running DeepSeek-V2-Lite with the AFD | 0, 1 | Attention | 18305 | | 2, 3 | FFN | 18305 | -The four variants cover the TP/DP cross product: - -| File | DP | TP | -|---------------------------------|----|----| -| `2a2f_*_dp1tp2.sh` | 1 | 2 | -| `2a2f_*_dp2tp1.sh` | 2 | 1 | +The four `2a2f_*` variants cover the TP/DP cross product; `4a4f_*` doubles +both attention and FFN ranks (DP=2 on each side); `multipod_2a_2f_graph_dbo_dp1tp2.sh` +is the same `2a2f_graph_dbo_dp1tp2` config, but gated on a `POD` env var +(`ATTENTION_0` or `FFN_0`) for the `deploy-afd-k8s` skill to deploy it +across 2 pods (see repo-root `recipe/gpu/P2pNcclAFDConnector/README.md`) -- +unlike the scripts above, it is **not** meant to be run directly: with +`POD` unset, neither `vllm serve` block executes, and its +`AFD_CONNECTOR_HOST` default (`vllm-ffn-p2p-service`) only resolves inside +the cluster the skill deploys it to: + +| File | DP | TP | +|---------------------------------------|----|----| +| `2a2f_*_dp1tp2.sh` | 1 | 2 | +| `2a2f_*_dp2tp1.sh` | 2 | 1 | +| `4a4f_*_dp2tp2.sh` | 2 | 2 | +| `multipod_2a_2f_graph_dbo_dp1tp2.sh` | 1 | 2 | ## Running diff --git a/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh new file mode 100644 index 000000000..b44372728 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + +# Use .agents/skills/deploy-afd-k8s/SKILL.md to deploy this recipe +# on 2 pods using the setup below. +# For pod 1: POD=ATTENTION_0 +# For pod 2: POD=FFN_0 + +MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite} +export VLLM_USE_V2_MODEL_RUNNER=0 + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-vllm-ffn-p2p-service} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269} + +if [ "$POD" == "ATTENTION_0" ]; then + CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 1 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --max-num-batched-tokens 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > attn.log 2>&1 & +fi + +if [ "$POD" == "FFN_0" ]; then + CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 1 \ + --tensor-parallel-size 2 \ + --enable-expert-parallel \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": 2, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 64 \ + --enable-dbo \ + --dbo-decode-token-threshold 2 \ + --dbo-prefill-token-threshold 12 \ + --max-num-batched-tokens 64 \ + --max-cudagraph-capture-size 64 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64] + }' \ + --host 127.0.0.1 \ + --port 18305 \ + --trust-remote-code > ffn.log 2>&1 & +fi + +wait diff --git a/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh new file mode 100644 index 000000000..c7bf95452 --- /dev/null +++ b/recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh @@ -0,0 +1,100 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + +# Use .agents/skills/deploy-afd-k8s/SKILL.md to deploy this recipe +# on 3 pods using the setup below. +# For pod 1: POD=ATTENTION_0 +# For pod 2: POD=ATTENTION_1, ATTENTION_HEADLESS=1, ATTENTION_DP_START_RANK=2 +# For pod 3: POD=FFN_0 + +MODEL_PATH=${MODEL_PATH:-/path/model_weights/Qwen3.5-122B-A10B-FP8} +export VLLM_USE_V2_MODEL_RUNNER=0 + +AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-vllm-ffn-p2p-service} +AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-1239} + +ATTENTION_DP_START_RANK=${ATTENTION_DP_START_RANK:-0} +ATTENTION_HEADLESS=${ATTENTION_HEADLESS:-0} +ATTENTION_DP_ADDRESS=${ATTENTION_DP_ADDRESS:-vllm-attn-dp-service} +ATTENTION_DP_RPC_PORT=${ATTENTION_DP_RPC_PORT:-13345} + +HEADLESS_ARGS=() +if [[ "$ATTENTION_HEADLESS" == "1" ]]; then + HEADLESS_ARGS=(--headless) +fi + +DP_START_RANK_ARGS=() +if [[ "$ATTENTION_DP_START_RANK" != "0" ]]; then + DP_START_RANK_ARGS=(--data-parallel-start-rank "$ATTENTION_DP_START_RANK") +fi + +if [[ "$POD" == ATTENTION_* ]]; then + CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 4 \ + --data-parallel-size-local 2 \ + "${DP_START_RANK_ARGS[@]}" \ + --data-parallel-address "$ATTENTION_DP_ADDRESS" \ + --data-parallel-rpc-port "$ATTENTION_DP_RPC_PORT" \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --dtype bfloat16 \ + --language-model-only \ + --max-model-len 114688 \ + --mamba-cache-mode align \ + --all2all-backend allgather_reducescatter \ + --seed 0 \ + --additional-config '{ + "afd": { + "role": "attention", + "connector": "P2pNcclAFDConnector", + "afd_process_group_timeout_s": 120, + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": 4, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 32 \ + --max-num-batched-tokens 8192 \ + --max-cudagraph-capture-size 32 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] + }' \ + "${HEADLESS_ARGS[@]}" \ + --host 127.0.0.1 \ + --port 18305 > attn.log 2>&1 & +fi + +if [[ "$POD" == FFN_* ]]; then + CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \ + --data-parallel-size 2 \ + --tensor-parallel-size 1 \ + --enable-expert-parallel \ + --dtype bfloat16 \ + --language-model-only \ + --max-model-len 114688 \ + --mamba-cache-mode align \ + --all2all-backend allgather_reducescatter \ + --seed 0 \ + --additional-config '{ + "afd": { + "role": "ffn", + "connector": "P2pNcclAFDConnector", + "afd_process_group_timeout_s": 120, + "host": "'"${AFD_CONNECTOR_HOST}"'", + "port": '"${AFD_CONNECTOR_PORT}"', + "num_attention_ranks": 4, + "num_ffn_ranks": 2 + } + }' \ + --max-num-seqs 32 \ + --max-num-batched-tokens 8192 \ + --max-cudagraph-capture-size 32 \ + --compilation-config '{ + "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[32] + }' \ + --host 127.0.0.1 \ + --port 18305 > ffn.log 2>&1 & +fi + +wait