feat(recipes): redhat @ 524288 ctx — pinned 10 GiB KV, autotune off, 4096 chunks, lab image for CC-12.0 topk guards

512K needed 3 fixes, each proven by a boot that died without it:
1. flashinfer autotune + cutedsl warmup off + expandable_segments
   (autotune scratch at 512K OOMs NVRM: NV_ERR_NO_MEMORY);
2. --max-num-batched-tokens 4096 --block-size 256 (lab memory shape);
3. docker_image -> glm53-flash-nvfp4-lab:local — the FujitsuPolycom
   sparse_attn_indexer CC-12.0 guards are REQUIRED: indexer topk fits 48
   SMs at 262K but wants 62 CTAs at 512K, fallback needs 128KB smem
   (GB10: 99KB) → hard abort. Patches are arch-level; W4A4 output
   verified correct on the lab image.

Measured: KV 1,334,265 tok / 2.54x @ 524288, ~17 min boot, greedy answer
correct, finish=stop. Ops note: rank-1 headless workers survive rank-0
death and hold the node — rm -f label-filtered leftovers before retry.

recipes/glm53-flash-nvfp4-redhat/recipe.toml

@@ -5,15 +5,25 @@
 # embeddings / lm_head kept original precision. Quantizes via regex targets,
 # so no name-prefix matching bugs.
 # Model: https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4
-#        (~177 GB → ~88 GB/node on 2 Sparks; GMU 0.85 = ~101 GiB budget)
+#        (~177 GB → ~88 GB/node on 2 Sparks)
+# KV: pinned --kv-cache-memory-bytes 10 GiB (same as -lab, identical MLA
+#   geometry) → ~1.16 M-token pool @ fp8_ds_mla = 2.22× concurrency at the
+#   524288 ctx. GMU 0.90 matches the proven lab config; the pin overrides
+#   GMU-based KV sizing.
 #
 # ⚠ NO MTP DRAFTER IN THIS CHECKPOINT — the index has 146,745 tensors and
 #   zero layer-45/MTP weights. `--speculative-config mtp` WILL fail to load.
 #   This recipe is baseline decode only (~15 tok/s class on this arch —
 #   spec decode buys 1.3–2×; use glm53-flash-nvfp4[-lab] when MTP matters).
 #
-# Image: shared glm53-flash-nvfp4:local overlay (day-0 vLLM image + the
-# NoPE-MLA / sm120 sparse-MLA mod). The mod is arch-level, quant-independent.
+# Image: the -lab overlay (day-0 vLLM image + NoPE-MLA mod + FujitsuPolycom
+# arch patches). The sparse_attn_indexer CC-12.0 guards are REQUIRED at the
+# 512K ctx: at 262K the indexer topk fit 48 SMs, at 512K it wants 62 CTAs and
+# the fallback needs 128 KB smem (GB10 has 99 KB) → hard boot abort
+# (launch_persistent_topk oversubscribe, 2026-08-28). The patches are
+# arch-level, not quant-level: the modelopt quantprobe is a no-op for this
+# compressed-tensors checkpoint, the model.py naming shim anchors on
+# arch-internal names.
 #
 # GB10 gotchas handled (see glm53-flash-nvfp4 header for detail):
 #   --moe-backend marlin → if vLLM rejects marlin for W4A4 at startup (marlin
@@ -21,10 +31,13 @@
 #   token-loop garbage (the silent FLASHINFER_CUTLASS corruption) before
 #   trusting it; `--moe-backend triton` is the other candidate.
 #
-# Vendor note: RedHatAI's own README suggests --no-enable-flashinfer-autotune
-# for a fast boot — deliberately NOT set here: the GB10 sparse-MLA kernels
-# rely on the autotune result (first boot is slow by design, tune persists to
-# ~/.cache/vllm). Add it if you want to skip re-tuning after the first run.
+# GB10 gotcha (MEASURED): at 524288 ctx WITH flashinfer autotune on, rank 0
+# dies mid-boot on a driver OOM (NVRM NV_ERR_NO_MEMORY _memdescAllocInternal,
+# 2026-08-28 05:01) — autotune/graph scratch at the 512K verify shapes doesn't
+# fit the unified-memory headroom. The 262144 ctx + autotune combo booted
+# fine. Fix = the lab-proven boot: autotune + cutedsl warmup off (a perf
+# micro-opt only; the sparse-MLA kernels have a non-autotuned fallback) and
+# expandable_segments to cut fragmentation.
 #
 # PREREQUISITES: weights cached on both nodes:
 #   huggingface-cli download RedHatAI/GLM-5.3-Flash-NVFP4
@@ -35,8 +48,8 @@
 
 engine = "vllm"
 name = "GLM-5.3-Flash NVFP4-redhat (vLLM, 2x Spark, no MTP)"
-docker_image = "glm53-flash-nvfp4:local"
-docker_image_dir = "glm53-flash-nvfp4"
+docker_image = "glm53-flash-nvfp4-lab:local"
+docker_image_dir = "glm53-flash-nvfp4-lab"
 model = "RedHatAI/GLM-5.3-Flash-NVFP4"
 required_nodes = 2
 tensor_parallel_size = 2
@@ -49,8 +62,13 @@ args = [
     "--served-model-name", "glm53",
     "--trust-remote-code",
     "--distributed-executor-backend", "mp",
-    "--gpu-memory-utilization", "0.85",
-    "--max-model-len", "262144",
+    "--gpu-memory-utilization", "0.90",
+    "--kv-cache-memory-bytes", "10737418240",
+    "--max-model-len", "524288",
+    # Scan/graph scratch scales with the prefill chunk: 8192 OOMs the driver
+    # at 512K ctx (05:38 NVRM OOM), 4096 is the lab-proven ceiling here.
+    "--max-num-batched-tokens", "4096",
+    "--block-size", "256",
     "--max-num-seqs", "1",
     "--kv-cache-dtype", "fp8_ds_mla",
     "--language-model-only",
@@ -59,6 +77,8 @@ args = [
     "--tool-call-parser", "glm47",
     "--enable-auto-tool-choice",
     "--reasoning-parser", "glm45",
+    "--kernel-config", """
+'{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
 ]
 
 [env]
@@ -69,3 +89,4 @@ VLLM_ENGINE_READY_TIMEOUT_S = "3600"
 TORCH_CUDA_ARCH_LIST = "12.1a"
 CUTE_DSL_ARCH = "sm_121a"
 OMP_NUM_THREADS = "4"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"

specs/tech-architecture/glm53-recipe-state.md

@@ -51,6 +51,30 @@ algo=MXFP8` confirm the MTP fix is live; `algo=None` there = broken load.
 
 ## Operational notes (measured)
 
+### redhat @ 524288 ctx (2026-08-28)
+
+262144 booted on the base image; **512K needed three changes** (each verified
+by a real boot that died without it):
+1. `--kernel-config {"enable_flashinfer_autotune":false,"enable_cutedsl_warmup":false}`
+   + `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` — autotune scratch at
+   512K shapes OOMs the driver (`NVRM NV_ERR_NO_MEMORY _memdescAllocInternal`);
+   the sparse-MLA kernels' non-autotuned fallback is fine (lab proved it).
+2. `--max-num-batched-tokens 4096` + `--block-size 256` (lab's memory shape).
+3. **The lab image** — the killer was `launch_persistent_topk ... total_ctas=62
+   > num_sms*occupancy=48 ... fallback requires >=128KB smem (have 101376)`,
+   i.e. exactly what the FujitsuPolycom `sparse_attn_indexer(_kpool)` CC-12.0
+   guards fix. Indexer CTAs scale with ctx: 262K fit 48 SMs, 512K needs 62.
+   The patches are arch-level → safe for the compressed-tensors checkpoint
+   (quantprobe is a no-op without modelopt; output verified correct).
+
+Result: KV pool **1,334,265 tokens / 2.54× @ 524288** (10 GiB pin; lab gets
+1,164,369 from the same pin — its MTP speculator reserves memory before KV
+sizing), ~17 min clean boot, correct greedy output.
+
+Note: an orphaned **rank-1 headless worker survives rank-0 death** and keeps
+the node occupied — always `docker rm -f` label-filtered leftovers before a
+retry serve (or the launcher refuses with "only 1 nodes").
+
 - lab weight load: 92.19 GiB / 700 s sequential from the NAS HF cache;
   `--kv-cache-memory-bytes 10737418240` → KV 1,164,369 tokens / 2.22× @ 512K
   (matches upstream to the digit — do NOT switch back to GMU sizing).
@@ -69,8 +93,9 @@ algo=MXFP8` confirm the MTP fix is live; `algo=None` there = broken load.
 ## Follow-ups
 
 - Vendor instanttensor loader into the lab image (~30 s vs 700 s weight load).
-- Consider adding `--kernel-config` autotune-off + pinned KV to the redhat and
-  LibertAIDAI recipes (their boot could be minutes shorter; verify before trusting).
+- Consider adding `--kernel-config` autotune-off to the LibertAIDAI recipe
+  (boot minutes shorter; redhat now has it, see 512K section — verify before
+  trusting on LibertAIDAI).
 - MTP for redhat: RedHatAI ships MTP as **separate variant repos** (proven
   pattern: `RedHatAI/GLM-5.2-NVFP4-FP8-mtp-FP8` = full quant + `mtp.safetensors`
   alongside main weights). Nothing for 5.3-Flash yet (org search 2026-08-28: