glm53-mxfp4-fp8-4x-karmic: A/B vs r39 — native b12x MoE experts buy +1% (19.2 vs 19.0 t/s)

Same RedHat checkpoint, same cluster, same args. karmic's B12X_MXFP4_MXFP8
expert kernel vs r39's MarlinExperts: decode is bandwidth-bound at 96 GiB per
rank, so the Marlin A16 dequant was never the limiter. MXFP4 linears stay on
Marlin in both (--linear-backend b12x does not change the selection for these
shapes).

Karmic-only boot requirements recorded: --block-size 64 (256 gives a padded
BLNHC layout the engine refuses) and enable_jit_warmup=false (the b12x tuning
cache binds a temporary padded KV pool and trips the MLA bind assert).

recipes/glm53-mxfp4-fp8-4x-karmic/NOTE.md

@@ -0,0 +1,37 @@
+# karmic A/B — same checkpoint, native b12x MoE instead of Marlin experts
+
+`spark-glm53-karmic:local` (local-inference-lab fork, `dev/karmic-kraken`
+636afde9, vLLM 0.1.dev21553+gab86b7073) vs `jovian-judgement:r39-glm53` on the
+same `RedHatAI/GLM-5.3-MXFP4xFP8` checkpoint, same cluster (dgx5–8, TP4), same
+args, measured 2026-10-22.
+
+| | r39 (jovian lock) | karmic (lab fork) |
+|---|---|---|
+| MXFP4 MoE experts | `MarlinExperts` | **`B12X_MXFP4_MXFP8`** (native) |
+| MXFP4 linears | `MarlinMxFp4LinearKernel` | `MarlinMxFp4LinearKernel` (the same — `--linear-backend b12x` does not change the selection for these shapes) |
+| weights/rank | 96.34 GiB | 96.34 GiB |
+| decode, single request | **19.0 t/s** | **19.2 t/s** |
+| quality | 88/100 (v2.8.0) | not benched (same weights, same parsers) |
+
+**Conclusion: the native b12x expert kernel buys +1%.** At TP4 with 96 GiB of
+weights per Spark the decode is bandwidth-bound, not kernel-bound — the Marlin
+A16 dequant was never the limiter. Further speed has to come from elsewhere:
+MTP (the checkpoint ships a head), fewer bytes per token (TP8 halves the
+weights per rank), or more decode-parallel lanes, not from a faster GEMM.
+
+## Boot notes (karmic-specific)
+
+Two karmic-only failures on the way, both fixed in `recipe.toml`:
+
+1. `--block-size 256` → `The resolved KV cache layout (BLNHC) does not store
+   blocks as dense, unpadded pages (block stride 13808…)`. **64 works.**
+2. `--block-size 64` then tripped `AssertionError` in
+   `mla_attention.py:925` (`assert self.kv_cache.stride(0) % row_width == 0`),
+   raised from `b12x_prepare.initialize_b12x_tuning_cache` — the tuning pass
+   binds a temporary padded KV pool. The assert sits behind
+   `enable_jit_warmup`, so the kernel-config sets it false; the DSA indexer
+   kernel then compiles on the first request (`b12x autotuning uses 32`
+   temporary blocks appears in the log right before startup completes).
+
+Boot: load ~30 min from the NAS cache, `Application startup complete` ~2 min
+after the load, KV 79,616 tokens at `--max-model-len 65536`.

recipes/glm53-mxfp4-fp8-4x-karmic/recipe.toml

@@ -0,0 +1,160 @@
+# GLM-5.3 (FULL, non-Flash) — RedHatAI MXFP4×FP8 re-cut — 4× DGX Spark, TP=4.
+#
+# The 2026-10-21 re-cut of the RedHat quant: `RedHatAI/GLM-5.3-MXFP4xFP8`
+# (415.4 GB, 141 shards) instead of the 2026-09 `RedHatAI/GLM-5.3-MXFP4`
+# (403.4 GB). Mixed precision by layer group: routed experts + dense MLPs at
+# MXFP4 weights (FP4-e2m1, E8M0 group-32) with dynamic FP8 activations,
+# attention / shared experts / MTP head at FP8 block-wise (128). Architecture
+# and geometry are unchanged: GlmMoeDsaForCausalLM, 78 layers, MLA + DSA
+# lightning indexer, 1M native window, MTP head n=1 in-checkpoint.
+#
+# Stack: `spark-glm53-karmic:local` — our clone-at-pin build of the
+# local-inference-lab vLLM fork (`dev/karmic-kraken` tip 636afde9, 2026-10-08,
+# CUDA 13.4.2 base, B12X taken from FlashInfer 0.7.1), vLLM
+# 0.1.dev21553+gab86b7073. A/B against the r39 lane (88/100, ~19 t/s decode):
+# karmic's MXFP4 linears can run on the NATIVE b12x FP4 GEMM
+# (`--linear-backend b12x`, plans per token count) instead of Marlin's
+# weight-only A16 dequant fallback, which is what caps r39's decode.
+#
+# Known-bad in this fork too (checked statically in the image, 2026-10-22):
+#   * `FlashInferMxFp4LinearKernel.is_supported()` gates on
+#     `has_device_capability(100)`, which passes on GB10 (121) while the
+#     cute-dsl mm_fp4 rejects 121 at runtime — hence the disable env below.
+#   * `--attention-backend B12X` would hit the same
+#     `DeepseekV32B12xAttention.__init__() unexpected 'index_group_builder'`
+#     that r39 did (the model passes the kwarg, the b12x class lacks it), so
+#     the attention backend stays at the fork default.
+#
+# ORIGINAL r39 header follows.
+#
+# Stack: our own jovian r39 lock (`jovian-judgement:r39-glm53`) — the same
+# image that booted and scored **91/100** on this checkpoint family at 4x on
+# 2026-09-14 (`glm53-full-mxfp4-4x-probe`, harness dev65). r39 registers
+# GlmMoeDsaForCausalLM through the NEW `deepseek_v32` NVIDIA path (the GB10
+# target), ships the compressed-tensors W4A4-MXFP4 scheme and fp8_block, and
+# is our reproducible clone-at-pin build.
+#
+# Rejected for this lane (measured or verified 2026-10-22):
+#   * `spark-glm53:v10` (kindling gx10 fork) — registers the arch, loads all
+#     141 shards twice (~45 min/pass) and then dies in multi-node worker init:
+#     `is_in_the_same_node: gloo/tcp/pair.cc` → "WorkerProc failed to start".
+#     The fork is tuned for TP2 on two nodes.
+#   * `eugr/spark-vllm-b12x:nightly-20261015` (vLLM main 8244d1c44) — has the
+#     arch, `mxfp4`/`mxfp8`/`mxfp4_csf` methods and ct 0.17.0 schemes; kept as
+#     the fallback (`glm53-mxfp4-fp8-4x-eugr`) if r39 trips on the re-cut.
+#   * `ghcr.io/fujitsupolycom/glm53-flash-sparkring:v14-arm64` — a pinned
+#     contract for GLM-5.3-**Flash** NVFP4 (InstantTensor + MTP4 sampling +
+#     SparkRing NCCL), not a generic vLLM; nothing to reuse for this checkpoint
+#     beyond its ideas.
+#
+# MEMORY (the binding constraint — 121.63 GiB unified/node):
+#   weights 103.9 GB = 96.7 GiB/rank (was 93.9 GiB on the old cut, +2.8 GiB)
+#   vLLM procs + docker ~12 GiB · KV pin 4 GiB · activations/graphs ~4 GiB
+#   ⇒ ~117 GiB, inside the node but with no room for a second surprise, so
+#   this first boot is 64K context and 4 sequences. Ladder after a clean
+#   boot: 131072 (needs 6.6 GiB/rank) → DCP4 + 512K (see the probe's header).
+#
+# BOOT HAZARD (2026-09-08): a post-load crash with auto-restart began a
+# SECOND full load on top of the first → all four nodes OOM-thrashed. Watch
+# for a second "Loading safetensors" pass and kill immediately.
+
+engine = "vllm"
+name = "GLM-5.3 (full) MXFP4xFP8, 4x TP4 (karmic local-inference-lab fork)"
+docker_image = "spark-glm53-karmic:local"
+docker_image_dir = "karmic-kraken"
+entrypoint = ["vllm", "serve"]
+model = "RedHatAI/GLM-5.3-MXFP4xFP8"
+parent_model = "zai-org/GLM-5.3"
+port = 30000
+required_nodes = 4
+tensor_parallel_size = 4
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+cluster = "dgx58-switch"
+
+args = [
+    "--served-model-name", "glm53-full",
+    "--trust-remote-code",
+    "--prefill-compute-share", "0.4",
+    # Admission guardrail only (the KV pool is pinned): 0.85 × 121.63 = 103.4 GiB.
+    "--gpu-memory-utilization", "0.85",
+    # 4 GiB/rank: enough for one 65536-token request (51.6 KB/token/rank).
+    "--kv-cache-memory-bytes", "4294967296",
+    # 2026-10-22 05:41: explicit B12X dies in the worker with
+    #   TypeError: DeepseekV32B12xAttention.__init__() got an unexpected
+    #   keyword argument 'index_group_builder'
+    # (the re-cut's indexer API is newer than r39's B12X attention class), so
+    # the first boot lets the fork pick its default backend.
+    # "--attention-backend", "B12X",
+    # MXFP4 GEMM: the FlashInfer kernel accepts capability 121 in is_supported()
+    # and then fails at runtime (mm_fp4, cute-dsl); the native b12x warmup
+    # crashes on this checkpoint's layer ranks. Marlin is the working path.
+    # 2026-10-22 06:15: forcing marlin kills the boot inside
+    #   prepare_fp8_layer_for_marlin: AttributeError: 'ReplicatedLinear' object
+    #   has no attribute 'output_size_per_partition'
+    # — the re-cut puts fp8_block on ReplicatedLinear modules (shared experts /
+    # indexer) and marlin's fp8 prep only handles the parallel-linear shapes.
+    # Marlin IS the right kernel for the MXFP4 linears on GB10 (weight-only A16
+    # dequant), but the flag is global, so the default per-scheme dispatch runs
+    # instead.
+    # karmic A/B #1 (2026-10-22 10:34): --linear-backend b12x dies at the end of
+    # the load pass with
+    #   vllm.utils.b12x.PreparationResourceUnavailableError:
+    #   B12x block-FP8 linear has no prepared capacity plan
+    # (the fork prepares b12x plans from the *workload* in its preparation
+    # stages, and the fp8_block linears have no plan by the time the first
+    # forward runs at 4 GiB KV / 64K ctx). Retry keeps the default linear
+    # dispatch (Marlin for MXFP4 via the disable env below) and moves the
+    # dominant cost — the MoE experts — onto b12x instead.
+    "--moe-backend", "b12x",
+    # "--linear-backend", "b12x",
+    # "--linear-backend", "marlin",
+    # (2026-10-22 06:44, no flag at all: default dispatch sends the MXFP4
+    # linears to FlashInfer's mm_fp4, which rejects capability 121 at runtime
+    # — flashinfer.utils.BackendSupportedError: mm_fp4 does not support backend
+    # 'cute-dsl' with capability 121. The surgical fix is the env lever below:
+    # disable ONLY that kernel so MXFP4 falls through to Marlin while the
+    # fp8_block layers keep their own (working) backend.)
+    "--load-format", "instanttensor",
+    # b12x wants dense, unpadded KV pages: with 256 the resolved layout is
+    # BLNHC with a padded block stride and the engine refuses to start
+    # ("does not store blocks as dense, unpadded pages", 2026-10-22 11:05).
+    "--block-size", "64",
+    "--max-model-len", "65536",
+    "--max-num-seqs", "4",
+    "--max-num-batched-tokens", "4096",
+    "--enable-chunked-prefill",
+    "--async-scheduling",
+    "--enable-prefix-caching",
+    # No CuTe-DSL warmup / FlashInfer autotune: their scratch at large shapes
+    # does not fit this headroom.
+    # b12x tuning-cache warmup binds a TEMPORARY KV pool and the MLA layer
+    # asserts on the padded page stride (mla_attention.py:925,
+    # `assert self.kv_cache.stride(0) % row_width == 0`) — reached from
+    # b12x_prepare.initialize_b12x_tuning_cache, 2026-10-22 11:46. The assert
+    # sits behind enable_jit_warmup, so the JIT warmup is off: the DSA indexer
+    # kernel then compiles on the first request instead of at startup.
+    "--kernel-config", """
+'{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false,"enable_jit_warmup":false}'""",
+    "--language-model-only",
+    "--enable-auto-tool-choice",
+    "--tool-call-parser", "glm47",
+    "--reasoning-parser", "glm45",
+    # No --speculative-config on the first boot: the MTP head is real
+    # (num_nextn_predict_layers=1) but draft KV + graph shapes need headroom
+    # this lane does not have yet. Add mtp k=3 after a clean boot.
+]
+
+[env]
+VLLM_ENGINE_READY_TIMEOUT_S = "5400"
+HF_HUB_OFFLINE = "1"
+NCCL_NCHANNELS = "16"
+VLLM_ENABLE_ROCE_ALLREDUCE = "1"
+# compile-OOM guard: 12 parallel nvcc jobs × ~3 GiB behind ~97 GiB of weights
+MAX_JOBS = "2"
+# MXFP4 linears: FlashInfer's cute-dsl mm_fp4 does not support GB10 (cap 121),
+# and forcing --linear-backend marlin globally trips on the fp8_block
+# ReplicatedLinear modules. Disabling just the FlashInfer MXFP4 kernel makes
+# the dispatcher fall through to Marlin for FP4 and leave FP8 alone.
+VLLM_DISABLED_KERNELS = "FlashInferMxFp4LinearKernel"