glm53 full MXFP4: explicit --moe-backend flashinfer_cutlass_afp8 + document the MXFP4 ingestion path

The r35/r37 tree's default moe backend is b12x (W4A4) - not the route the lab
qualifies for MXFP4 GLM experts. Per their GLM-5.1 MXFP4 runbook: use
flashinfer_cutlass_afp8 (native FlashInfer Cutlass MXFP4xMXFP8), never set
B12X_MOE_FORCE_A16, GLM gets standard SwiGLU (gpt-oss clamp defaults gated).
Verified by read-only image inspection that the June GLM-5.1 MXFP4 vLLM
changes are merged in our Sept tree (oracle/mxfp4.py:297, quantization/
mxfp4.py non-gpt_oss routing, flashinfer_cutlass_moe.py:270-280). Fallbacks
documented (compressed-tensors ingestion path differs from the lab's
quant_method=mxfp4). Also: recipe dsv41-flash-jovian-r37-4x + profiles/
jovian-r37-ds41.env (built now), and the night plan gained P4 (LMCache
RAM/FS tiers for long context on a small KV pool, GLM vision TP2 contract,
explicit max-len/KV convention, CUBLAS workspace bounding)

history/night-shifts/2026-09-13.md

@@ -202,3 +202,87 @@ nightly-20260910 prod numbers (167/176, 1,642 pp). ~2.5–3 h total.
 3. If the full model cannot hold 512K on 4×, is a smaller context
    (131K–262K) acceptable for this lane, or is 512K a hard requirement
    (= the lane is dead on this hardware)?
+
+---
+
+## Amendments 2026-09-13 ~22:00 UTC (after reading the lab's GLM docs)
+
+Operator: build R37 first, then read `models/glm-5.3-flash-spark-tp2.md` and
+`models/glm-5.1-mxfp4.md` properly ("GLM-5.3 did not fundamentally change the
+architecture") and fold the logical consequences into the plan.
+
+### R37 build started
+`sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan`
+(recipe created as a clone of the r36 one with image/profile swapped; the
+build fetches `c687594b8a8` from voipmonitor/vllm + `f1c4e9dd5b1` from
+voipmonitor/b12x with the HEAD/tree gates — both present upstream). Log:
+`/tmp/build-r37.log`. Next after it lands: transfer to dgx1–4, then P2.
+
+### P0 amendment — the MXFP4 MoE backend must be explicit
+The r35/r37 tree's **default MoE backend is b12x** (r35 changelog). The lab's
+GLM-5.1 MXFP4 runbook says for MXFP4 GLM experts:
+`--moe-backend flashinfer_cutlass_afp8` → native **FlashInfer Cutlass
+MXFP4×MXFP8**; do **not** set `B12X_MOE_FORCE_A16`; GLM must get **standard
+SwiGLU** (gpt-oss alpha/beta/clamp defaults must not apply).
+The recipe now sets `flashinfer_cutlass_afp8` explicitly.
+- Already in our Sept tree (verified by image inspection): `oracle/mxfp4.py:297`
+  maps that name to `FLASHINFER_CUTLASS_MXFP4_MXFP8`; `quantization/mxfp4.py`
+  routes non-gpt_oss to the native `Mxfp4MoEMethod`; the gpt-oss SwiGLU gates
+  in `experts/flashinfer_cutlass_moe.py:270-280` abstain — i.e. the June
+  GLM-5.1 MXFP4 vLLM changes are merged.
+- RedHatAI ships `quant_method=compressed-tensors` / `mxfp4-pack-quantized`
+  (`CompressedTensorsW4A4Mxfp4`), a *different* ingestion path from the lab's
+  `quant_method=mxfp4` — hence the fallback order: `flashinfer_cutlass_afp8`
+  → `b12x` (proven on our DS4.1 MXFP4 experts) → drop the flag and read the
+  engine's selected backend from the log.
+- **Numerics gate either way**: a missing SwiGLU clamp (r33/r34 lesson) and
+  W4A4 token-loop corruption both present as degenerate repetition. Run the
+  P0.2 smoke with ~300 tokens and look at it, before any bench.
+
+### P4 — capability lanes worth a logical slot (from the lab docs)
+
+**(a) LMCache RAM + filesystem tiers → long context on a tiny GPU KV pool.**
+Their TP2 GLM-5.3-Flash contract is `MAX_MODEL_LEN=1048576` with only
+`KV_CACHE_MEMORY_BYTES=4294967296` (4 GiB/rank) and a 1,048,320-token cold
+text run in 139.7 s, RAM restore 0.808 s, filesystem restore 1.279 s, zero
+recompute. Options (before the image name):
+`-e CACHE_MODE=lmcache -e LMCACHE_L1_SIZE_GB=64 -e LMCACHE_L2_ROOT=/cache/lmcache-l2`.
+Caveats: the 64 GiB L1 arena is **preallocated pinned host shm**, so host
+`/dev/shm` must be big enough (Docker `--shm-size` does NOT resize host shm);
+`LMCACHE_L2_ENABLED=0` = RAM-only; the `/cache` volume persists FS objects
+across container replacement; objects cover 4,096 global tokens, attention
+pages 2,048 tokens per DCP rank; checkpoint files are hashed at startup (first
+log delayed while weights are read — keep them immutable while serving).
+Why it matters for us: it decouples "≥512K context" from "GPU KV pool size" —
+exactly the wall the full MXFP4 model hits on 4×. Worth a lane on the r35 GLM
+Flash profile (cheap: env only, same image).
+
+**(b) GLM vision on 2 GPUs.** The lab's TP2/DCP2 GLM-5.3-Flash contract
+includes vision: one image per API request (across all turns), native
+~8,000-feature ceiling, video disabled, full-image FlashAttention for the
+image pass, target KV FP8 + FP32 recurrent state, MTP3 probabilistic. Cold
+1,048,320-token image input: correct answer in 143.0 s with **min free VRAM
+232.31 MiB/GPU**. All our GLM lanes are text-only (`--language-model-only`) —
+this is a capability upgrade (screenshots/diagrams in agent traffic), and it
+is a documented, qualified contract rather than guesswork.
+
+**(c) Two conventions to copy into our recipes.**
+- Explicit `MAX_MODEL_LEN` + explicit `KV_CACHE_MEMORY_BYTES` (they never let
+  a lane silently shorten context; `KV_CACHE_MEMORY_BYTES=auto` is for
+  auto-profiling and `MAX_MODEL_LEN=-1` is the opt-in for accepting a shorter
+  context; GMU does not resize an explicit KV allocation). We already pin both
+  — keep it that way, and quote the KV line in every run report.
+- `CUBLAS_WORKSPACE_CONFIG=:4096:1` bounds the cuBLAS workspace to 4 MiB per
+  handle → 140 MiB less live allocation per rank (they measured it; not a
+  confirmed cure for every community cuBLAS crash). Cheap, consider for the
+  full-model lane where every 100 MiB counts.
+
+**Important caveat from their doc:** "The memory mechanisms can also apply to
+TP4, but their magnitude depends on sharding and workload. **No TP4 memory or
+performance improvement is qualified by the TP2 measurements.** The TP2 NCCL
+configuration is not a TP4 recommendation." (They use 2 NCCL channels / 1 MiB
+buffer on TP2; we run 16 channels / 2 MiB on TP4 — do not copy the TP2 values.)
+
+**Reference numbers from that doc** (2× GB10, TP2/DCP2, MTP3, 4 GiB KV/rank,
+1M ctx): 32K prefill 10,648.9 tok/s; C1 MTP3 median output 170.77 tok/s;
+verifier 67.5 steps/s; R2 vs R1 = +0.41% prefill / +4.52% C1.

recipes/glm53-full-mxfp4-4x/recipe.toml

@@ -103,6 +103,19 @@
 #   compile-OOM guard (12 parallel nvcc jobs × ~3 GiB behind 94 GiB of weights
 #   = cicc OOM-killed, 2026-09-04 lesson).
 #
+# ── MXFP4 vs NVFP4: what changed, and what is already in our tree ──────────
+# The full-model NVFP4 lane is retired (see above), so this checkpoint is the
+# lab's MXFP4 family: routed-expert weights FP4-e2m1 with power-of-two (E8M0)
+# group-32 scales — the same shape as their own GLM-5.1 MXFP4 repack, which is
+# why their GLM-5.1 runbook (models/glm-5.1-mxfp4.md) is the reference for
+# this lane even though GLM-5.3 moved on: the MXFP4 ingestion path, the
+# FlashInfer Cutlass MXFP4xMXFP8 MoE route, the standard-SwiGLU rule and the
+# "do not set B12X_MOE_FORCE_A16" warning all still apply. That runbook also
+# unsets empty NCCL graph variables before launch
+# (NCCL_GRAPH_FILE / NCCL_GRAPH_DUMP_FILE / VLLM_B12X_MLA_EXTEND_MAX_CHUNKS)
+# — unset them in the shell if a previous lane exported them empty; an empty
+# value is not the same as an unset one to NCCL.
+#
 # ── SPECULATIVE DECODING ───────────────────────────────────────────────────
 #   NOT enabled on the first boot: the checkpoint's MTP head is real
 #   (num_nextn_predict_layers=1) and the card suggests mtp k=5, but spec adds
@@ -164,6 +177,31 @@ args = [
     # knobs). The card's --kv-cache-dtype fp8_e4m3 is the fallback spelling
     # if the pool needs to shrink.
     "--attention-backend", "B12X",
+    # MXFP4 MoE path — MUST be explicit: the r35/r37 tree's DEFAULT moe
+    # backend is b12x (r35 changelog), which is the W4A4 path we use for the
+    # DS4.1 MXFP4 experts, not the route the lab qualifies for MXFP4 GLM
+    # experts. Their GLM-5.1 MXFP4 runbook (models/glm-5.1-mxfp4.md,
+    # "Runtime Rules") is explicit: `--moe-backend flashinfer_cutlass_afp8`
+    # selects the native FlashInfer Cutlass MXFP4xMXFP8 MoE path, and
+    # B12X_MOE_FORCE_A16 must NOT be set for that checkpoint. Verified present
+    # in OUR Sept tree (read-only image inspection 2026-09-13):
+    #   fused_moe/oracle/mxfp4.py:297  "flashinfer_cutlass_afp8" ->
+    #                                 Mxfp4MoeBackend.FLASHINFER_CUTLASS_MXFP4_MXFP8
+    #   quantization/mxfp4.py:79/134  non-gpt_oss routes to the native
+    #                                 Mxfp4MoEMethod (GptOssMxfp4Config abstains)
+    #   fused_moe/experts/flashinfer_cutlass_moe.py:270-280  the gpt-oss
+    #                                 alpha/beta/clamp SwiGLU defaults are gated
+    #                                 -> GLM gets standard SwiGLU
+    # so the June GLM-5.1 MXFP4 vLLM changes are merged upstream already.
+    # FALLBACK if this backend rejects the compressed-tensors checkpoint
+    # (RedHatAI ships quant_method=compressed-tensors / format=
+    # mxfp4-pack-quantized, i.e. CompressedTensorsW4A4Mxfp4 — a different
+    # ingestion path from the lab's quant_method=mxfp4): try "b12x" (proven on
+    # our DS4.1 MXFP4 experts) or drop the flag and read the engine's selected
+    # MoE backend from the log. WHICHEVER backend runs, validate the numerics:
+    # the missing-SwiGLU-clamp class of bug (r33/r34: runaway generation) and
+    # the W4A4 token-loop corruption both show up as degenerate repeats.
+    "--moe-backend", "flashinfer_cutlass_afp8",
     # pure I/O loader (kept across every GLM config)
     "--load-format", "instanttensor",
     "--block-size", "256",