Ver Fonte

1017 vllm032: 512K context and the standard served-model-name

Bring the lane to the cluster standard: --max-model-len 524288 (the production
SGLang lane's window; 131072 was only the cautious first boot) and the
served-model-name deepseek-v4.1-flash instead of the local dsv41 alias, so
clients point at one id across lanes. The KV pool at 131072 was 6,895,712
tokens, i.e. a 512K request is ~8% of it.
Alexander Korolev há 12 horas atrás
pai
commit
db4c0d8e29
1 ficheiros alterados com 7 adições e 5 exclusões
  1. 7 5
      recipes/dsv41-flash-1017-4x-vllm032/recipe.toml

+ 7 - 5
recipes/dsv41-flash-1017-4x-vllm032/recipe.toml

@@ -37,14 +37,16 @@ cluster = "dgx58-switch"
 entrypoint = ["vllm", "serve"]
 
 args = [
-    "--served-model-name", "dsv41",
+    # the cluster-wide alias: every lane serves the production name, so
+    # clients (agents, gateways, benches) point at one id
+    "--served-model-name", "deepseek-v4.1-flash",
     "--trust-remote-code",
     "--distributed-executor-backend", "mp",
     "--gpu-memory-utilization", "0.88",
-    # 131072 first: the production lane runs 524288 behind SGLang's engram
-    # staging, but a vLLM boot that OOMs teaches us nothing. Raise after the
-    # baseline numbers exist.
-    "--max-model-len", "131072",
+    # 524288 — the cluster standard (same window as the production SGLang
+    # lane). 131072 was the cautious first boot; with the engram shards staged
+    # locally the KV pool came up at 6.9M tokens, so a 512K request is 8% of it.
+    "--max-model-len", "524288",
     "--max-num-seqs", "4",
     "--max-num-batched-tokens", "4096",
     "--enable-chunked-prefill",