glm53-mxfp4-fp8-4x-r39: 196K window hung the host on a 150K prefill — back to 131072 / 6 GiB KV / 2048 chunks

A ~150K prompt blew the RPC budget (EngineDead, HTTP 500) and then thrashed
dgx6+dgx7 (ping alive, sshd starved) — an HA plug cycle was needed. The KV pool
was never the limit: the prefill scratch does not fit the ~11 GiB left after an
11 GiB pin behind 98.86 GiB of weights. 131072 with a 6 GiB pin and 2048-token
chunks leaves ~16 GiB for scratch and is the configuration this lane keeps.

recipes/glm53-mxfp4-fp8-4x-r39/NOTE.md

@@ -110,3 +110,23 @@ memory left after the 98.86 GiB of weights is ~20 GiB, so:
 
 The 512K/1M windows of the Flash sibling live on DCP4, a path this checkpoint
 cannot take with our images.
+
+## Incident 2026-10-22 22:45 — long prefill hangs the host
+
+The 196608-window lane (11 GiB KV pin) answered normally at short and medium
+prompts, then a **~150K-token prompt** produced
+`TimeoutError: RPC call to sample_tokens timed out` → `EngineDeadError` →
+HTTP 500, and **dgx6 + dgx7 stopped answering SSH while still pinging** — the
+2026-09-08 unified-memory thrash signature. Recovered with an HA plug cycle of
+both nodes (off 35 s → on); no data loss (all weights are on the NAS).
+
+Reading: the killer is the **prefill scratch**, not the KV pool. After 98.86
+GiB of weights a GB10 Spark has ~22 GiB left; an 11 GiB KV pin leaves ~11 GiB,
+which a 150K-token prefill (37 chunked steps at 4096 tokens) does not fit. The
+lane therefore runs **131072 context, 6 GiB KV pin, 2048 batched tokens**
+(smaller chunks = smaller scratch) and `VLLM_RPC_TIMEOUT=3600`.
+
+Rule for this checkpoint on 4 ranks: `window_max ≈ KV_pin / 52.7 KB` **and**
+the prefill of that window must fit the ~22 GiB-left budget — test long prompts
+ascending (8K → 32K → 64K → 128K) and never send a prompt the lane has not been
+measured at.

recipes/glm53-mxfp4-fp8-4x-r39/recipe.toml

@@ -63,7 +63,7 @@ args = [
     # DCP4 (2026-10-22, third boot): context is sharded across the 4 ranks, so
     # 8 GiB/rank pins ~32 GiB aggregate — one 524288-token request needs ~27 GiB
     # aggregate (51.6 KB/token/rank unsharded, measured), which just fits.
-    "--kv-cache-memory-bytes", "10737418240",
+    "--kv-cache-memory-bytes", "6442450944",
     # Decode-context parallelism: the proven r35 4x shape for the Flash sibling
     # (glm53-flash-r35-4x-dcp4). Unverified for THIS checkpoint — if the boot
     # rejects it (the GLM C4 page-stride / interleave-multiple-of-4 gates), drop
@@ -94,9 +94,9 @@ args = [
     # fp8_block layers keep their own (working) backend.)
     "--load-format", "instanttensor",
     "--block-size", "256",
-    "--max-model-len", "196608",
+    "--max-model-len", "131072",
     "--max-num-seqs", "4",
-    "--max-num-batched-tokens", "4096",
+    "--max-num-batched-tokens", "2048",
     "--enable-chunked-prefill",
     "--async-scheduling",
     "--enable-prefix-caching",
@@ -123,6 +123,10 @@ NCCL_NCHANNELS = "16"
 VLLM_ENABLE_ROCE_ALLREDUCE = "1"
 # compile-OOM guard: 12 parallel nvcc jobs × ~3 GiB behind ~97 GiB of weights
 MAX_JOBS = "2"
+# A ~150K-token prefill runs 37 chunked steps of 4096 tokens and blew the
+# default 600 s RPC budget (2026-10-22 22:45: "RPC call to sample_tokens timed
+# out" killed the engine core, HTTP 500 with an empty reply).
+VLLM_RPC_TIMEOUT = "3600"
 # MXFP4 linears: FlashInfer's cute-dsl mm_fp4 does not support GB10 (cap 121),
 # and forcing --linear-backend marlin globally trips on the fp8_block
 # ReplicatedLinear modules. Disabling just the FlashInfer MXFP4 kernel makes