lab-quant variant of 0rand winning config (forum 381350 #119): GMU 0.89, MNBT 2048, KV auto-size

Reproduced their knob set on ds4: KV auto-size lands 1,167,586 tokens
(same capacity as our 10 GiB pin — the pin is redundant; GMU is the
budget). Bench c1,2,4: 90/100 (158/176), tg 27.1/34.6/37.2 — c4 decode
scales (baseline has no c4 arm), but quality -2 and c2 decode -15% vs
our baseline (92, 40.5). Kept as variant; baseline stays the default.
Bench report path + verdict in the spec ledger.

recipes/glm53-flash-nvfp4-lab-0rand/recipe.toml

@@ -0,0 +1,117 @@
+# GLM-5.3-Flash NVFP4 — local-inference-lab quant (vLLM) — 2× DGX Spark, TP=2.
+#
+# local-inference-lab mixed quant: NVFP4 routed experts (layers 3–44,
+# group-16) + MXFP8 MTP experts (layer 45) + W4A4 input scales. Best agent
+# quant so far: tool-eval-bench 89/100 (vs 85 LibertAIDAI, 90 official FP8).
+# Model: https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4
+#        (~186 GB → 92.72 GiB weights/node; KV pinned at 10 GiB, see below)
+#
+# Image: glm53-flash-nvfp4-lab:local (docker-images/glm53-flash-nvfp4-lab/ —
+# day-0 arm64 image + NoPE-MLA mod + the four FujitsuPolycom lab-checkpoint
+# patches: model naming shim, modelopt MTP namespace fix, persistent_topk
+# CC-12.x guards). Their launch config is mirrored below.
+# PROVENANCE: https://github.com/FujitsuPolycom/glm53-flash-tp2-spark
+#             (Apache-2.0; see patches/UPSTREAM.md) + forum 381350 #41.
+#
+# Deviations from their launch script (intentional):
+#   * no --kda-prefill-backend flashkda (needs their base-image patch — skipped)
+#   * --kv-cache-dtype fp8_ds_mla instead of `fp8` (same packed layout; this
+#     spelling is what the NoPE-MLA mod booted with on the LibertAIDAI stack)
+#   * --language-model-only + no mm-renderer patch/env (text-only)
+#   * no bundled NCCL/LD_PRELOAD, no NCCL_* fabric block (our switched fabric
+#     is configured via [nccl.ds4] + launcher GID auto-detect)
+#   * no --moe-backend marlin: their proven stack uses the default backend for
+#     THIS quant and reports clean evals. Sanity-check output for token loops
+#     anyway (the silent FP4 MoE corruption gotcha) before trusting a new node.
+#   * flashinfer autotune disabled via --kernel-config (their proven boot);
+#     VLLM_ENGINE_READY_TIMEOUT_S stays high regardless.
+#
+# Boot success markers (engine log, in order):
+#   "Model loading took 92.72 GiB" → "[quantprobe] ... algo=MXFP8" (MTP fix
+#   active; algo=None means broken) → "GPU KV cache size: 1,164,369 tokens" →
+#   "Application startup complete". ~700 s sequential weight load.
+#
+# Test request: chat completions need "chat_template_kwargs":{"reasoning_effort"
+# :"low"} — omitted/other values select MAX reasoning effort (empty content on
+# short max_tokens).
+#
+# Optional: instanttensor loader (~30 s weight load vs ~700 s):
+#   FROM glm53-flash-nvfp4-lab:local + pip install --no-deps --no-build-isolation
+#   instanttensor==0.1.9, then add --load-format instanttensor (--no-deps is
+#   required: resolution would downgrade the image's NCCL).
+#
+# PREREQUISITES: weights cached on both nodes:
+#   hf download local-inference-lab/GLM-5.3-Flash-NVFP4
+
+engine = "vllm"
+name = "GLM-5.3-Flash NVFP4-lab - 0rand winning config (vLLM, 2x Spark, MTP-3)"
+docker_image = "glm53-flash-nvfp4-lab:local"
+docker_image_dir = "glm53-flash-nvfp4-lab"
+model = "local-inference-lab/GLM-5.3-Flash-NVFP4"
+required_nodes = 2
+tensor_parallel_size = 2
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+args = [
+    # Stable served name — quant-agnostic (shared across glm53 quants)
+    "--served-model-name", "glm53",
+    "--trust-remote-code",
+    "--distributed-executor-backend", "mp",
+    # GMU is ONLY a startup guardrail here (KV pool is pinned below): 0.90
+    # demands 109.46 GiB free on GB10's 121.63 GiB, but driver reserve +
+    # page cache routinely leave ~108.5 -> flaky
+    # "Free memory ... less than desired" boot deaths (this was the class
+    # of the old "54-second death"), and post-crash reruns see ~106.6 GiB.
+    # 0.82 = 99.7 GiB, below any realistic dip; does NOT change what gets
+    # allocated (weights ~93 + pinned KV 10 + activations).
+    # 0rand's winning config (forum 381350 #119): 0.89, MNBT 2048, KV
+    # auto-sized (1.05M-token pool) instead of our 0.82 + pinned 10 GiB.
+    "--gpu-memory-utilization", "0.89",
+    # Pinned 10 GiB KV (≈1.16M tokens) — GMU sizing has no headroom at
+    # 92.72 GiB weights/node; pinning also keeps warmup off the swap path
+    "--kv-cache-dtype", "fp8_ds_mla",
+    "--block-size", "256",
+    "--max-model-len", "524288",
+    "--max-num-seqs", "8",
+    "--max-num-batched-tokens", "2048",
+    "--enable-chunked-prefill",
+    "--async-scheduling",
+    "--enable-prefix-caching",
+    "--disable-custom-all-reduce",
+    # Their proven boot: no flashinfer autotune, no CuTe-DSL warmup
+    "--kernel-config", """
+'{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
+    # MTP-3 draft from the MXFP8 layer-45 shard (needs modelopt.patch in image)
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":3}'""",
+    "--language-model-only",
+    # MTP + CUDA graphs degrades OUTPUT QUALITY on the glm53 day-0 dev build
+    # (forum #381350/#44, CONFIRMED by A/B bench 2026-08-28: same seed/flags,
+    # graphs on = 89/100, eager = 92/100; decode unchanged 28.4 t/s @ c1,
+    # c2 -7%, TTFT +1.6s). The b12x glm52 stack does NOT need this (its
+    # graphs+MTP combo benches clean at 100/100) — bug is build-specific.
+    "--enforce-eager",
+    "--tool-call-parser", "glm47",
+    "--enable-auto-tool-choice",
+    "--reasoning-parser", "glm45",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+# NoPE-MLA rope-pad runtime gate (mod baked into the image)
+VLLM_MLA_NOPE_PAD_ROPE = "1"
+VLLM_ENGINE_READY_TIMEOUT_S = "3600"
+# Upstream launch-script env (memory + compile knobs)
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+VLLM_USE_AOT_COMPILE = "0"
+TORCH_USE_RTLD_GLOBAL = "1"
+VLLM_NO_USAGE_STATS = "1"
+
+# Architecture (aarch64 / sm_121a DGX Spark)
+TORCH_CUDA_ARCH_LIST = "12.1a"
+FLASHINFER_CUDA_ARCH_LIST = "12.1f"
+CUTE_DSL_ARCH = "sm_121a"
+CMAKE_CUDA_ARCHITECTURES = "121"
+OMP_NUM_THREADS = "4"

specs/tech-architecture/glm53-recipe-state.md

@@ -205,3 +205,21 @@ the four lanes** (lab 72, 4p67 67, sparkrun 66), driven by responsiveness
 50/100 with a 3.0 s median turn. Quality still trails lab-eager-with-MTP3
 (92) by ~4 points — the 4bpw price. Report:
 /tmp/runs/2026/08/2026-08-30T17-43-04.770487Z_63900bf7.md.
+
+## 0rand winning-config trial on the lab quant (2026-08-30 evening)
+
+Forum 381350 #117/#119 report 93/100 (164/176) for the lab quant with
+GMU 0.89, MNBT 2048, KV auto-sized (1.05M pool), MTP-3. Reproduced on
+ds4 as `glm53-flash-nvfp4-lab-0rand` (variant in tree): KV auto-size
+lands at 1,167,586 tokens — same capacity as our pinned 10 GiB, so the
+pin is unnecessary; smoke exact. Bench (--seed 42 --hardmode --perf
+--concurrency 1,2,4): quality 90 (158/176), tg 27.1/34.6/**37.2** c1/c2/c4,
+TTFT d0 1.3 s, responsiveness 25 (6.2 s median), deployability 70.
+vs our lab baseline (92 = 162/176; c1 28.4 / c2 40.5; resp 21, deploy 72):
+quality −2, c2 decode −15%, c1 ≈, but c4 decode 37.2 in an arm the
+baseline never ran, and responsiveness +4. On OUR stack the 0rand combo
+is not a strict win; keep the baseline as the daily driver, keep the
+variant in tree for concurrency-hungry serving. Possible confounder:
+their run posts a single 14.8 s median turn at responsiveness 8 — an
+earlier bench build. Report:
+/tmp/runs/2026/08/2026-08-30T19-33-22.444571Z_4616f476.md.