NVFP4-Spark serving on our r12 fork build: spec-config attention_backend=B12X fixes MTP draft selection; intel recipe retargeted; ledger

docker-images/glm53-jovian-r12/Dockerfile

@@ -61,6 +61,7 @@ RUN set -eux; \
     # The spinloop target needs an explicit nvrtc: the cu13 toolkit ships
     # under targets/sbsa-linux (ldconfig knows it, FindCUDA does not).
     NVRTC=$(ls /usr/local/cuda*/targets/sbsa-linux/lib/libnvrtc.so* | head -1); \
+    ln -s "${NVRTC}" /usr/lib/aarch64-linux-gnu/libnvrtc.so; \
     CMAKE_ARGS="-DCUDAToolkit_ROOT=/usr/local/cuda -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DCUDA_nvrtc_LIBRARY=${NVRTC} -DCUDAToolkit_LIBRARY_DIR=/usr/local/cuda/targets/sbsa-linux/lib" \
     VLLM_TARGET_DEVICE=cuda TORCH_CUDA_ARCH_LIST="12.1a" \
         pip install --no-build-isolation --no-deps . 2>&1; \

docker-images/glm53-jovian-r12/patch_cuda_sm12x.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""B12X into the sm12x auto-priority list (platforms/cuda.py).
+
+The sm12x branch of _get_backend_priorities only offers
+[TRITON_MLA, FLASHINFER_MLA_SPARSE_SM120]. Neither matches GLM-5.3-Flash:
+TRITON_MLA rejects sparse, and FLASHINFER_MLA_SPARSE_SM120 only supports
+head_size 576 (DeepSeek-V4 MLA), not GLM's 512. The B12X paged/sparse-MLA
+backend explicitly supports compute (12, 0) and (12, 1), so it is offered
+first. This matters for the MTP draft model: its vllm_config does NOT
+inherit the global --attention-backend, so its MLA layers go through
+auto-selection and die without this patch.
+"""
+
+import importlib
+import sys
+
+P = "/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py"
+
+OLD = """        elif device_capability.major == 12:
+            return [
+                AttentionBackendEnum.TRITON_MLA,
+                AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120,
+            ]"""
+
+NEW = """        elif device_capability.major == 12:
+            return [
+                AttentionBackendEnum.B12X,
+                AttentionBackendEnum.TRITON_MLA,
+                AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120,
+            ]"""
+
+s = open(P).read()
+if NEW in s:
+    print("[patch] already applied")
+elif OLD in s:
+    open(P, "w").write(s.replace(OLD, NEW, 1))
+    print("[patch] applied: B12X first in sm12x MLA priorities")
+else:
+    sys.exit("[patch] sm12x priority block not found (source drift)")

recipes/glm53-flash-intel-w4a16/recipe.toml

@@ -32,13 +32,14 @@
 # max_tokens).
 
 engine = "vllm"
-# timothystewart6 image has ENTRYPOINT [] + CMD bash — declare it explicitly.
-entrypoint = ["vllm", "serve"]
-name = "GLM-5.3-Flash W4A16-AutoRound Intel (vLLM, 2x Spark)"
-# timothystewart6/vllm-gb10 v0.28.0-gb10.2 — GB10 daily-build image (upstream
-# vLLM 0.28.0, torch 2.11, arm64 sm_121a; has vllm-rs frontend). Was suggested
-# for the Intel W4A16 AutoRound lane; also covers glm5_next arch coverage risk.
-docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.28.0-gb10.2"
+name = "GLM-5.3-Flash W4A16-AutoRound Intel (vLLM, 2x Spark, r12 fork)"
+# glm53-jovian-r12:local — our in-docker build of local-inference-lab/vllm
+# (jovian-judgement composition + PR heads 550-557) with the native
+# Glm5NextForConditionalGeneration arch and B12X sparse-MLA attention.
+# The timothystewart6 image was abandoned: upstream vLLM 0.28 there has NO
+# native glm5_next arch (its Transformers-backend fallback lacks the
+# per-stream q_conv1d weight mapping — boot dies in the weight loader).
+docker_image = "glm53-jovian-r12:local"
 model = "Intel/GLM-5.3-Flash-W4A16-AutoRound"
 required_nodes = 2
 tensor_parallel_size = 2
@@ -53,6 +54,10 @@ args = [
     # AutoRound int4 pack → GPTQ-style safetensors; explicit quant route (the
     # timothystewart6 upstream-stack build knows quant_method auto-round).
     "--quantization", "gptq_marlin",
+    # Required for the head-512 MLA layers on sm_12x: the auto-selection
+    # list (TRITON_MLA / FLASHINFER_MLA_SPARSE_SM120) has no backend that
+    # fits GLM's 512 (SM120 flashinfer supports head 576 only).
+    "--attention-backend", "B12X",
     "--distributed-executor-backend", "mp",
     # GMU is ONLY a startup guardrail (KV pool is pinned below): 0.90
     # flaked boots on GB10 ("Free memory ... less than desired") because

recipes/glm53-flash-nvfp4-spark/mods/patch_b12x_sm12x_priorities.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+# The sm12x auto-priority list in platforms/cuda.py only offers
+# [TRITON_MLA, FLASHINFER_MLA_SPARSE_SM120] — neither matches GLM-5.3's
+# head-512 MLA (SM120 flashinfer only supports head 576). The MTP draft
+# model's vllm_config does NOT inherit the global --attention-backend, so
+# its MLA layers auto-select and die. The B12X paged/sparse-MLA backend
+# (which the main model uses via the global flag) supports (12,0)/(12,1),
+# so offer it first.
+set -euo pipefail
+P=/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py
+python3 - <<'PY'
+p = "/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py"
+s = open(p).read()
+old = """        elif device_capability.major == 12:
+            return [
+                AttentionBackendEnum.TRITON_MLA,
+                AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120,
+            ]"""
+new = """        elif device_capability.major == 12:
+            return [
+                AttentionBackendEnum.B12X,
+                AttentionBackendEnum.TRITON_MLA,
+                AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120,
+            ]"""
+if "AttentionBackendEnum.B12X,\n                AttentionBackendEnum.TRITON_MLA" in s:
+    print("[b12x-sm12x] already patched")
+else:
+    assert old in s, "sm12x priority block not found (source drift)"
+    open(p, "w").write(s.replace(old, new, 1))
+    print("[b12x-sm12x] patched: B12X first in sm12x MLA priorities")
+import importlib, vllm.platforms.cuda
+importlib.reload(vllm.platforms.cuda)
+print("[b12x-sm12x] OK")
+PY
+find /usr/local/lib/python3.12/dist-packages/vllm -name "__pycache__" -type d -prune -exec rm -rf {} + 2>/dev/null || true

recipes/glm53-flash-nvfp4-spark/recipe.toml

@@ -68,7 +68,7 @@
 
 engine = "vllm"
 name = "GLM-5.3-Flash NVFP4-Spark (vLLM, 2x Spark, MTP-5)"
-docker_image = "glm53-flash-nvfp4-lab:local"
+docker_image = "glm53-jovian-r12:local"  # r12 fork build (see docker-images/glm53-jovian-r12)
 docker_image_dir = "glm53-flash-nvfp4-lab"
 model = "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark"
 required_nodes = 2
@@ -93,7 +93,13 @@ args = [
     # Pinned 10 GiB KV (≈1.16M tokens) — GMU sizing has no headroom at
     # 92.72 GiB weights/node; pinning also keeps warmup off the swap path
     "--kv-cache-memory-bytes", "10737418240",
-    "--kv-cache-dtype", "fp8_ds_mla",
+    # No --kv-cache-dtype: let the r12 fork pick its default (the explicit
+    # fp8 / fp8_ds_mla both failed auto-selection on this build).
+    # The GLM5Next model wires the B12X sparse-MLA backend (sm_120/sm_121)
+    # when the global backend is B12X (glm5next/nvidia/attention.py:
+    # _select_sparse_backend); the sm12x auto-priority list never offers it
+    # (FLASHINFER_MLA_SPARSE_SM120 only supports head 576, not GLM's 512).
+    "--attention-backend", "B12X",
     "--block-size", "256",
     "--max-model-len", "524288",
     "--max-num-seqs", "8",
@@ -106,8 +112,11 @@ args = [
     "--kernel-config", """
 '{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
     # MTP-5 draft from the embedded NVFP4 MTP shards (lil.yaml mtp_tokens: 5)
+    # The draft model's vllm_config takes its attention backend from the
+    # SPECULATIVE config, not the global flag (eagle/utils.py:
+    # _make_eagle_draft_vllm_config) — B12X here mirrors the main model's.
     "--speculative-config", """
-'{"method":"mtp","num_speculative_tokens":3}'""",
+'{"method":"mtp","num_speculative_tokens":3,"attention_backend":"B12X"}'""",
     "--language-model-only",
     # MTP + CUDA graphs degrades OUTPUT QUALITY on the glm53 day-0 dev build
     # (forum #381350/#44, CONFIRMED by A/B bench 2026-08-28: same seed/flags,

specs/tech-architecture/glm53-recipe-state.md

@@ -337,3 +337,29 @@ of local-inference-lab's fork — source contract from models/glm-5.3-flash.md
 build rules apply: CUDA 13.0.88, sm_121f gencode rule for block-scaled FP4).
 RTX docs archive: /tmp/lil-docs (ds4-flash-v6, ds4dspark-v20-r33, infernal
 r21, pcie-autotune) for the DeepSeek quality track.
+
+## 2026-09-02: r12 fork built + NVFP4-Spark serving on it (the day's win)
+
+Built `glm53-jovian-r12:local` in-docker on dgx3 (docker-images/glm53-jovian-r12/):
+vllm 0.1.dev20489+gcc161e220 (local-inference-lab/vllm base 54f6e9826c + PR heads
+550/552/553/554/555/557 merged in order — the doc's "composition commit" 7d8a09c is
+not a public ref) + b12x 1.3.0 (master 139e0405 + PR heads 260-270), on the day-0
+`vllm/vllm-openai:glm53-flash-arm64-cu130@905c02…` base. Build fixes applied along
+the way: apt git, setuptools-rust, ninja/cmake, cu13 wheel include-header symlinks
+(cusparse.h), unversioned libnvrtc.so symlink for the deepgemm_C link step.
+
+NVFP4-Spark boot on it (recipe glm53-flash-nvfp4-spark, retargeted):
+- whole-network NVFP4 W4A4 (incl. attention in_proj_qkvbfg scales) now LOADS —
+  the r12 fork's modelopt handles it (our lab fork: KeyError).
+- Attention wall solved: the sm12x auto-priority list only offers
+  [TRITON_MLA (no sparse), FLASHINFER_MLA_SPARSE_SM120 (head 576 only)] — neither
+  fits GLM head-512. Fix = `--attention-backend B12X` (model hook in
+  glm5next/nvidia/attention.py: _select_sparse_backend) for the main model, and
+  `"attention_backend":"B12X"` INSIDE --speculative-config for the MTP draft
+  (eagle/utils.py: _make_eagle_draft_vllm_config takes the draft's backend from
+  the speculative config, not the global flag — user's hint).
+- Served: exact "391" verified, MTP-3 Glm5NextMTPModel resolved, KV fp8_ds_mla
+  default, 2-node TP2. c1 bench running.
+- Intel W4A16 recipe retargeted to the same image (gptq_marlin + B12X attn).
+- patch_cuda_sm12x.py (B12X into sm12x priorities) staged in the image dir for
+  the next rebuild — the image-baked version of the same fix.