recipes: intel glm53 W4A16-AutoRound (tjstewart image, gptq_marlin, entrypoint) + lab NVFP4-Spark (mtp-5 embedded; blocked on whole-network W4A4 loader); ledger: rtx6kpro source contract for own fork build

recipes/glm53-flash-intel-w4a16/recipe.toml

@@ -0,0 +1,135 @@
+# GLM-5.3-Flash NVFP4 (lab quant) on the COMMUNITY sparkrun image — 2× DGX
+# Spark, TP=2.
+#
+# Image: aidendle94/sparkrun-vllm-glm53-gb10 (Spark Arena / sparkrun
+# orchestrator's GLM-5.3 vLLM build). Verified 2026-08-29: same day-0 vLLM
+# dev base as our lab image reports (v0.1.dev20051+g487ecf187, torch
+# 2.13.0+cu130), arm64-only (no manifest list — sm_121 target), generic
+# `vllm serve` entrypoint, no baked weights. Digest pin (single-arch
+# manifest, checked with `docker manifest inspect`):
+#   sha256:9b0592293a0b0161a040568578aca330227a03d140fa1c5f8965f64292431ef0
+#
+# WHY THIS RECIPE: cross-checks our home-built lab image against the Spark
+# Arena community stack — same checkpoint, same proven flag set, different
+# runtime. If both boot and score alike, the flag set (not our patches) is
+# the load-bearing part; any delta localizes to the image overlay.
+# Flags are mirrored verbatim from recipes/glm53-flash-nvfp4-lab (their
+# launch config + our fabric/env deviations — see that header for the full
+# rationale of each knob).
+#
+# Weights live on the NAS (mounted at /mnt/nas on the nodes) and are
+# exposed to the container via extra_mounts + HF_HOME — no per-node
+# `hf download` needed (config.toml [mount_aliases] hf_cache repoints this
+# site-wide).
+#
+# Boot markers: "Model loading took ~92.7 GiB" → "[quantprobe] …
+# algo=MXFP8" (MTP/modelopt fix active in this image; algo=None means the
+# image lacks the mod — report and fall back to our lab image) →
+# "GPU KV cache size" → "Application startup complete".
+#
+# Chat needs "chat_template_kwargs":{"reasoning_effort":"low"} — omitted or
+# other values select MAX reasoning effort (empty content on short
+# max_tokens).
+
+engine = "vllm"
+# timothystewart6 image has ENTRYPOINT [] + CMD bash — declare it explicitly.
+entrypoint = ["vllm", "serve"]
+name = "GLM-5.3-Flash W4A16-AutoRound Intel (vLLM, 2x Spark)"
+# timothystewart6/vllm-gb10 v0.28.0-gb10.2 — GB10 daily-build image (upstream
+# vLLM 0.28.0, torch 2.11, arm64 sm_121a; has vllm-rs frontend). Was suggested
+# for the Intel W4A16 AutoRound lane; also covers glm5_next arch coverage risk.
+docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.28.0-gb10.2"
+model = "Intel/GLM-5.3-Flash-W4A16-AutoRound"
+required_nodes = 2
+tensor_parallel_size = 2
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+args = [
+    # Stable served name — quant-agnostic (shared across glm53 quants)
+    "--served-model-name", "glm53",
+    "--trust-remote-code",
+    # AutoRound int4 pack → GPTQ-style safetensors; explicit quant route (the
+    # timothystewart6 upstream-stack build knows quant_method auto-round).
+    "--quantization", "gptq_marlin",
+    "--distributed-executor-backend", "mp",
+    # GMU is ONLY a startup guardrail (KV pool is pinned below): 0.90
+    # flaked boots on GB10 ("Free memory ... less than desired") because
+    # driver reserve + page cache leave ~106-109 GiB free of 121.63.
+    # 0.82 = 99.7 GiB, below any realistic dip; changes nothing allocated.
+    "--gpu-memory-utilization", "0.82",
+    "--kv-cache-memory-bytes", "10737418240",
+    # DEVIATION from lab: plain fp8 KV (this image's mla cache kernel asserts
+    # "pe_dim must be 64 for fp8_ds_mla" — its NoPE handling differs from the
+    # lab image's rope-pad mod; plain fp8 is what the vendor GLM-5.3 recipe
+    # uses and doesn't assert pe layout).
+    "--kv-cache-dtype", "fp8",
+    "--block-size", "256",
+    "--max-model-len", "524288",
+    "--max-num-seqs", "8",
+    "--max-num-batched-tokens", "4096",
+    "--enable-chunked-prefill",
+    "--async-scheduling",
+    "--enable-prefix-caching",
+    "--disable-custom-all-reduce",
+    # Pin the MoE backend: the lab image's `auto` resolves to MARLIN for
+    # this quant; this image's `auto` resolves to a TileLang/deepgemm JIT
+    # path that compiles 97 CUDA kernels AT BOOT (all cores, evicted the
+    # kernel's own worker via OOM — dmesg 08:30, cicc/cudafe++ storms).
+    # MARLIN = proven lab path, no JIT at boot.
+    "--moe-backend", "marlin",
+    # Proven day-0 boot: no flashinfer autotune, no CuTe-DSL warmup
+    "--kernel-config", """
+'{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
+    # DEVIATION from lab recipe: no MTP here. This image lacks the lab
+    # image's FuijitsuPolycom modelopt-MTP namespace fix (boot dies at
+    # layers.45.mtp_block.mlp.experts.routed_experts.w2_weight_scale —
+    # verified 2026-08-30 on ds4). Spec layers are skipped entirely when
+    # no spec config is set, so the base model loads clean.
+    "--language-model-only",
+    # Day-0 guard: on the lab image, graphs+MTP regressed tool quality. Keep
+    # while unverified on this build; A/B if you want graphs.
+    "--enforce-eager",
+    "--tool-call-parser", "glm47",
+    "--enable-auto-tool-choice",
+    "--reasoning-parser", "glm45",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+# NoPE-MLA rope-pad runtime gate (baked into our lab image; no-op if this
+# image gates it differently — boot marker will tell)
+VLLM_MLA_NOPE_PAD_ROPE = "1"
+VLLM_ENGINE_READY_TIMEOUT_S = "3600"
+VLLM_NO_USAGE_STATS = "1"
+# Upstream launch-script env (memory + compile knobs)
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+VLLM_USE_AOT_COMPILE = "0"
+TORCH_USE_RTLD_GLOBAL = "1"
+
+# Architecture (aarch64 / sm_121a DGX Spark)
+TORCH_CUDA_ARCH_LIST = "12.1a"
+FLASHINFER_CUDA_ARCH_LIST = "12.1f"
+CUTE_DSL_ARCH = "sm_121a"
+CMAKE_CUDA_ARCHITECTURES = "121"
+OMP_NUM_THREADS = "4"
+
+# NAS-backed HF cache (read-only use; HF writes nothing in offline mode).
+# Direct host path — NAS is mounted at /mnt/nas on every node; use
+# [mount_aliases] in config.toml when paths differ per site.
+[[extra_mounts]]
+host = "/mnt/nas/ai/huggingface"
+container = "/mnt/nas/ai/huggingface"
+env = "HF_HOME"
+
+# The image ships a CUDA 13 runtime WITHOUT dev headers (no nvrtc.h —
+# 105 headers total) and flashinfer's deepgemm JIT compiles at warmup with
+# nvcc -> "nvrtc.h: No such file or directory" (verified 2026-08-30 on
+# ds4). Bind the host's nvrtc.h through; it is an API-decl header, ABI-
+# independent of the 13.x minor in this image.
+[[extra_mounts]]
+host = "/usr/local/cuda/include/nvrtc.h"
+container = "/usr/local/cuda/include/nvrtc.h"
+read_only = true
+

recipes/glm53-flash-nvfp4-spark/recipe.toml

@@ -0,0 +1,139 @@
+# GLM-5.3-Flash NVFP4 — local-inference-lab quant (vLLM) — 2× DGX Spark, TP=2.
+#
+# local-inference-lab mixed quant: NVFP4 routed experts (layers 3–44,
+# group-16) + MXFP8 MTP experts (layer 45) + W4A4 input scales. Best agent
+# quant so far: tool-eval-bench 89/100 (vs 85 LibertAIDAI, 90 official FP8).
+# Model: https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4
+#        (~186 GB → 92.72 GiB weights/node; KV pinned at 10 GiB, see below)
+#
+# Image: glm53-flash-nvfp4-lab:local (docker-images/glm53-flash-nvfp4-lab/ —
+# day-0 arm64 image + NoPE-MLA mod + the four FujitsuPolycom lab-checkpoint
+# patches: model naming shim, modelopt MTP namespace fix, persistent_topk
+# CC-12.x guards). Their launch config is mirrored below.
+# PROVENANCE: https://github.com/FujitsuPolycom/glm53-flash-tp2-spark
+#             (Apache-2.0; see patches/UPSTREAM.md) + forum 381350 #41.
+#
+# Deviations from their launch script (intentional):
+#   * no --kda-prefill-backend flashkda (needs their base-image patch — skipped)
+#   * --kv-cache-dtype fp8_ds_mla instead of `fp8` (same packed layout; this
+#     spelling is what the NoPE-MLA mod booted with on the LibertAIDAI stack)
+#   * --language-model-only + no mm-renderer patch/env (text-only)
+#   * no bundled NCCL/LD_PRELOAD, no NCCL_* fabric block (our switched fabric
+#     is configured via [nccl.ds4] + launcher GID auto-detect)
+#   * no --moe-backend marlin: their proven stack uses the default backend for
+#     THIS quant and reports clean evals. Sanity-check output for token loops
+#     anyway (the silent FP4 MoE corruption gotcha) before trusting a new node.
+#   * flashinfer autotune disabled via --kernel-config (their proven boot);
+#     VLLM_ENGINE_READY_TIMEOUT_S stays high regardless.
+#
+# Boot success markers (engine log, in order):
+#   "Model loading took 92.72 GiB" → "[quantprobe] ... algo=MXFP8" (MTP fix
+#   active; algo=None means broken) → "GPU KV cache size: 1,164,369 tokens" →
+#   "Application startup complete". ~700 s sequential weight load.
+#
+# Test request: chat completions need "chat_template_kwargs":{"reasoning_effort"
+# :"low"} — omitted/other values select MAX reasoning effort (empty content on
+# short max_tokens).
+#
+# Optional: instanttensor loader (~30 s weight load vs ~700 s):
+#   FROM glm53-flash-nvfp4-lab:local + pip install --no-deps --no-build-isolation
+#   instanttensor==0.1.9, then add --load-format instanttensor (--no-deps is
+#   required: resolution would downgrade the image's NCCL).
+#
+# PREREQUISITES: weights cached on both nodes:
+# GLM-5.3-Flash NVFP4-Spark — local-inference-lab quant with the speculative
+# layers embedded — 2× DGX Spark, TP=2.
+#
+# local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (rev 06972cdee7280cec849aedf05a02062b3e870246):
+# NVFP4 routed experts (layers 3+, group-16) + NVFP4 MTP expert layer in the
+# separate model-mtp-0000x.safetensors shards (mtp_tokens: 5 per their
+# lil.yaml), amax/nonexpert/inputscales side shards, Glm5NextForConditionalGeneration.
+# THE "DSpark speculative embedded" checkpoint — no separate DFlash2 download
+# needed; lil.yaml: default_speculator mtp, dflash2_tokens 7 (r12 doc fr: their
+# standalone-Q: MTP:3 254 tok/s CC1, 333 Sieve, 94 steps/s).
+#
+# Image: glm53-flash-nvfp4-lab:local (same proven day-0 arm64 stack + NoPE-MLA
+# mod + modelopt MTP namespace patch used by the 92/100 lab lane). The MTP
+# layer here is NVFP4 (not MXFP8) — the quantprobe marker may print
+# algo=NVFP4/- different; algo=None = patch missed (fall back to 4p67 image).
+#
+# Deviations from lab (unchanged mechanics): same fabric/env/GMU/KV pinning.
+# lil.yaml launch mirrors: block 256, disable_flashinfer_autotune, mm data
+# mode — our text-only flags identical to glm53-flash-nvfp4-lab.
+#
+# Boot markers: "Model loading took ~XX GiB" → "[quantprobe] ..." →
+# "GPU KV cache size" → "Application startup complete".
+#
+# PREREQUISITES: weights on the NAS hub (175 GiB): hf download local-inference-lab/GLM-5.3-Flash-NVFP4-Spark
+
+engine = "vllm"
+name = "GLM-5.3-Flash NVFP4-Spark (vLLM, 2x Spark, MTP-5)"
+docker_image = "glm53-flash-nvfp4-lab:local"
+docker_image_dir = "glm53-flash-nvfp4-lab"
+model = "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark"
+required_nodes = 2
+tensor_parallel_size = 2
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+args = [
+    # Stable served name — quant-agnostic (shared across glm53 quants)
+    "--served-model-name", "glm53",
+    "--trust-remote-code",
+    "--distributed-executor-backend", "mp",
+    # GMU is ONLY a startup guardrail here (KV pool is pinned below): 0.90
+    # demands 109.46 GiB free on GB10's 121.63 GiB, but driver reserve +
+    # page cache routinely leave ~108.5 -> flaky
+    # "Free memory ... less than desired" boot deaths (this was the class
+    # of the old "54-second death"), and post-crash reruns see ~106.6 GiB.
+    # 0.82 = 99.7 GiB, below any realistic dip; does NOT change what gets
+    # allocated (weights ~93 + pinned KV 10 + activations).
+    "--gpu-memory-utilization", "0.82",
+    # Pinned 10 GiB KV (≈1.16M tokens) — GMU sizing has no headroom at
+    # 92.72 GiB weights/node; pinning also keeps warmup off the swap path
+    "--kv-cache-memory-bytes", "10737418240",
+    "--kv-cache-dtype", "fp8_ds_mla",
+    "--block-size", "256",
+    "--max-model-len", "524288",
+    "--max-num-seqs", "8",
+    "--max-num-batched-tokens", "4096",
+    "--enable-chunked-prefill",
+    "--async-scheduling",
+    "--enable-prefix-caching",
+    "--disable-custom-all-reduce",
+    # Their proven boot: no flashinfer autotune, no CuTe-DSL warmup
+    "--kernel-config", """
+'{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
+    # MTP-5 draft from the embedded NVFP4 MTP shards (lil.yaml mtp_tokens: 5)
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":3}'""",
+    "--language-model-only",
+    # MTP + CUDA graphs degrades OUTPUT QUALITY on the glm53 day-0 dev build
+    # (forum #381350/#44, CONFIRMED by A/B bench 2026-08-28: same seed/flags,
+    # graphs on = 89/100, eager = 92/100; decode unchanged 28.4 t/s @ c1,
+    # c2 -7%, TTFT +1.6s). The b12x glm52 stack does NOT need this (its
+    # graphs+MTP combo benches clean at 100/100) — bug is build-specific.
+    "--enforce-eager",
+    "--tool-call-parser", "glm47",
+    "--enable-auto-tool-choice",
+    "--reasoning-parser", "glm45",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+# NoPE-MLA rope-pad runtime gate (mod baked into the image)
+VLLM_MLA_NOPE_PAD_ROPE = "1"
+VLLM_ENGINE_READY_TIMEOUT_S = "3600"
+# Upstream launch-script env (memory + compile knobs)
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+VLLM_USE_AOT_COMPILE = "0"
+TORCH_USE_RTLD_GLOBAL = "1"
+VLLM_NO_USAGE_STATS = "1"
+
+# Architecture (aarch64 / sm_121a DGX Spark)
+TORCH_CUDA_ARCH_LIST = "12.1a"
+FLASHINFER_CUDA_ARCH_LIST = "12.1f"
+CUTE_DSL_ARCH = "sm_121a"
+CMAKE_CUDA_ARCHITECTURES = "121"
+OMP_NUM_THREADS = "4"

specs/tech-architecture/glm53-recipe-state.md

@@ -302,3 +302,38 @@ capability is the raw 1M-context draw. 3 safety-critical failures (TC-42 extra-p
 TC-43 empty required param, TC-60 sleeper injection) — the day-0 0731-style tool
 formatting is the gap vs the GLM lanes (thinking-pair JSON fencing was already a
 known day-0 weakness). Quality 81 < sparkrun 89 < lab 92.
+
+## New 2026-09-01 checkpoints: Intel W4A16-AutoRound + lab NVFP4-Spark
+
+Checkpoints on the NAS hub (hf download'd):
+- `Intel/GLM-5.3-Flash-W4A16-AutoRound` rev 5eee1846f0321058ed73745f9aa16f2aaf0fc0a0 —
+  AutoRound 0.15 W4A16 sym int4 g168, 99.84% of BF16 (GSM8K .9712/MMLU .8620),
+  self_attn/hc_/indexer/embed lm_head FP16 by design (their ignore_layers),
+  pure GPTQ packing (qweight/qzeros/scales — qweight count 37152).
+- `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` rev 06972cdee7280cec849aedf05a02062b3e870246 —
+  NVFP4 W4A4 *whole network* (in_proj_qkvbfg_a.weight_scale etc.) + NVFP4 MTP
+  expert layer in separate model-mtp-0000x shards (lil.yaml: mtp_tokens 5,
+  dflash2_tokens 7, default_speculator mtp, block 256, disable flashinfer
+  autotune, INSTANTTENSOR env sizes; arch Glm5NextForConditionalGeneration +
+  vision blocks in the index). This is the rtx6kpro r12-stack checkpoint
+  (their doc: MTP:3 254 tok/s CC1 @ TP4, Sieve 333-403).
+
+Recipes added: `recipes/glm53-flash-intel-w4a16` (timothystewart6/vllm-gb10
+v0.28.0-gb10.2 arm64, gptq_marlin, entrypoint declared — tini AssertionError
+ taught us the image has ENTRYPOINT []+CMD bash), `recipes/glm53-flash-nvfp4-spark`
+(lab image). Boot outcomes:
+- Intel lane died in the fork's Transformers-backend weight loader:
+  "no module or parameter named 'layers.0.self_attn.q_conv1d'" — the
+  tjstewart image has NO native glm5_next arch (vllm archs list lacks it;
+  transformers 5.16.1 has Glm5NextForConditionalGeneration) → needs the
+  local-inference-lab fork (native glm5next) or a transformers name-shim mod.
+- NVFP4-Spark on our lab fork: KeyError on `in_proj_qkvbfg_a.weight_scale` at
+  80% shard load — whole-network NVFP4 W4A4 is beyond the R8-era lab fork
+  loader (r12 stack territory).
+
+⇒ THE unifying next step (user directive): build our own arm64/sm_121a image
+of local-inference-lab's fork — source contract from models/glm-5.3-flash.md
+(local-inference-lab/vllm dev/jovian-judgement + b12x master + 6/5 PRs; kk3
+build rules apply: CUDA 13.0.88, sm_121f gencode rule for block-scaled FP4).
+RTX docs archive: /tmp/lil-docs (ds4-flash-v6, ds4dspark-v20-r33, infernal
+r21, pcie-autotune) for the DeepSeek quality track.