# GLM-5.3-Flash NVFP4 — three quant variants measured on 2× DGX Spark (ds4) Measured 2026-08-27/28 on cluster `ds4` (dgx3+dgx4, switched fabric), all three served by sm121 through the launcher, same benchmark invocation: ``` tool-eval-bench run --base-url http://:30000/v1 --model glm53 \ --seed 42 --hardmode --perf --concurrency 1,2 ``` **Tool version pin (2026-09-05, verified against the installed tool):** tool-eval-bench 2.6.1.dev45+gcf54b4bfe (uv tool; git SeraphimSerapis/tool-eval-bench @ cf54b4bfe). Current pool semantics: `--hardmode` = 69 standard (A–O) + 19 Hard Mode (P) = **88 scenarios** (≈26 min, dry-run verified); `--hardmode-only` = 19 (≈6 min). The `/176` scores above (155/176 anchor, lab ref 156/176) are pinned to the OLDER 176-point pool — scores do NOT transfer across tool versions. Every campaign now re-baselines the anchor lane with the CURRENT tool first (night queue 1a), then gates new lanes against that new baseline. All three share served name **`glm53`** — clients never changed between runs. ## Results | recipe | quant | quality | points | decode tg t/s (c1) | tg c2 | TTFT | deployability | MTP | |---|---|---|---|---|---|---|---|---| | `glm53-flash-nvfp4-lab` | local-inference-lab mixed NVFP4/MXFP8-MTP | **92** (eager) / 89 (graphs) | 162/176 | **28.4** | 40.5 | 5.6 s | **72** | ✅ k=3, accept len ~2.9–3.5, ~63–83 % | | `glm53-flash-nvfp4-redhat` | RedHatAI W4A4 compressed-tensors | **90** | 159/176 | 13.9 | 13.8 | 2.2 s | 67 | ❌ (no MTP weights in checkpoint) | | `glm53-flash-nvfp4` | LibertAIDAI NVFP4-A16 | **88** | 154/176 | 26.2 | 22.7 | 6.3 s | 69 | ✅ k=5 | | `glm53-flash-nvfp4-4p67` | 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 | **87** | 153/176 | 24.4 | 39.2 | 1.4 s | 67 | ✅ accept ~90% | | `glm53-flash-aiden-4x` | aiden image production-1.1 (DFlash2 overlay), NVFP4, **incoai DFlash2 k=7**, eager, 16 GiB KV pin, 4× Spark TP4 (his production shape) | **89** | 156/176 | 37.6 (d0 c1; flat to d8192, 2,185 pp t/s — matches his ~2,100) | 50.7 (c2) / 63.5 (c4) | 0.9 s | **80** | ✅ DFlash2 k=7 | | `glm53-flash-aiden-2x` | aiden image **production-1.1** (DFlash2 overlay), NVFP4, **incoai DFlash2 k=7**, eager, 4 GiB KV pin (2x profile) | **89** | 157/176 | 22.3 (d0 c1; d8192 c4 degrades — 4 GiB pin) | — | — | 66 | ✅ DFlash2 k=7 (drafter block 8−1) | | `glm53-flash-sparkrun` | community sparkrun image (aidendle94), NVFP4, eager, NO MTP (image lacks the fix) | **89** | 154/176 | 14.3 | 23.8 | 1.4 s | 66 | ❌ (n/a in image) | | `glm53-flash-exl3` | brandonmusic EXL3/TR3 4bpw (Entrpi image + instanttensor), DFlash2-7, graphs | **88** | 155/176 | 23.1 | 24.1 | 2.6 s d0 | **77** | ✅ accept 48–81 % | (lab scored /174 — TC-53 was excluded from its run as an infrastructure failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s total — harness drops timed-out scenarios from the denominator; had it scored like redhat's 2/2, lab would be 157/176 = 89 — no change). Quality % is comparable. LibertAIDAI is the only variant whose MTP *regresses* under c2 (26.2→22.7) — lab's k=3 draft scales, LibertAIDAI's k=5 does not. 4x note (2026-09-04): the 4x aiden run had one safety-critical failure — TC-43 called `web_search` with an empty query (required-parameter violation); not seen in the 2x aiden run.) Prefill (pp2048) 0.3–1.7 K t/s depending on depth/concurrency — noisy, all three in the same band (NAS load-influenced); decode is the stable discriminator. ## Read of the numbers - **lab (eager) wins outright**: 92/100 best quality AND 2× decode (28–40 t/s vs ~14 baseline). Acceptance lengths matched upstream's 2.55–3.5 range. - **redhat** (90/100 with graphs) is a **baseline-decode dead-end on this box: its checkpoint ships zero MTP tensors** (verified index: 146,745 tensors, none under layer 45), so it is stuck at ~14 t/s — no longer the quality pick either. - **lab --enforce-eager = day-driver; redhat = fallback; LibertAIDAI =** proven first but k=5 MTP regresses at c2 (eager might lift it too — untested). ## Images & patches | image dir | base | patches | |---|---|---| | `glm53-flash-nvfp4` | vllm/vllm-openai:glm53-flash-arm64-cu130 | kingjones30 NoPE-MLA mod (4-part, anchor-asserted) | | `glm53-flash-nvfp4-lab` | same | + 4 FujitsuPolycom patches (Apache-2.0, see patches/UPSTREAM.md): model.py naming shim, modelopt MTP namespace fix, persistent_topk CC-12.x guards ×2 | | redhat recipe reuses `glm53-flash-nvfp4:local` (mod is quant-independent) | | | Boot markers `[quantprobe] layer=RoutedExperts prefix=model.layers.45.mlp.experts algo=MXFP8` confirm the MTP fix is live; `algo=None` there = broken load. ## MTP + CUDA graphs quality bug (forum #381350/#44, CONFIRMED 2026-08-28) `0rand`: “mtp or dflash plus cuda graphs seriously hit quality — try disabling cuda graph capture”. A/B with identical seed/flags on lab: | arm | quality | points | tg c1 | tg c2 | TTFT | |---|---|---|---|---|---| | graphs (CUDA graphs ON) | 89 | 155/174 | 28.4 | 43.4 | 3.9 s | | **`--enforce-eager`** | **92** | **162/176** | 28.4 | 40.5 | 5.6 s | +3 quality points for −7 % c2 decode — eager is now the lab default. Accept metrics were healthy in BOTH arms (71.6 %, len 3.15 eager), so this is not an acceptance collapse — graph capture corrupts sampled output silently. Corroborating: randomllama (DFlash2 recipe) already flagged “not claiming bit-identical greedy outputs” with graphs on. The b12x glm52 stack runs graphs + MTP k=4 FULL_AND_PIECEWISE benches clean (100/100) — build-specific bug in the glm53 day-0 dev build, do NOT propagate --enforce-eager to glm52. Untested follow-ups: LibertAIDAI + eager (k=5; expect ~88→9x), redhat is graph-neutral (no MTP, stays 90 with graphs). ## Operational notes (measured) ### redhat @ 524288 ctx (2026-08-28) 262144 booted on the base image; **512K needed three changes** (each verified by a real boot that died without it): 1. `--kernel-config {"enable_flashinfer_autotune":false,"enable_cutedsl_warmup":false}` + `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` — autotune scratch at 512K shapes OOMs the driver (`NVRM NV_ERR_NO_MEMORY _memdescAllocInternal`); the sparse-MLA kernels' non-autotuned fallback is fine (lab proved it). 2. `--max-num-batched-tokens 4096` + `--block-size 256` (lab's memory shape). 3. **The lab image** — the killer was `launch_persistent_topk ... total_ctas=62 > num_sms*occupancy=48 ... fallback requires >=128KB smem (have 101376)`, i.e. exactly what the FujitsuPolycom `sparse_attn_indexer(_kpool)` CC-12.0 guards fix. Indexer CTAs scale with ctx: 262K fit 48 SMs, 512K needs 62. The patches are arch-level → safe for the compressed-tensors checkpoint (quantprobe is a no-op without modelopt; output verified correct). Result: KV pool **1,334,265 tokens / 2.54× @ 524288** (10 GiB pin; lab gets 1,164,369 from the same pin — its MTP speculator reserves memory before KV sizing), ~17 min clean boot, correct greedy output. Note: an orphaned **rank-1 headless worker survives rank-0 death** and keeps the node occupied — always `docker rm -f` label-filtered leftovers before a retry serve (or the launcher refuses with "only 1 nodes"). - lab weight load: 92.19 GiB / 700 s sequential from the NAS HF cache; `--kv-cache-memory-bytes 10737418240` → KV 1,164,369 tokens / 2.22× @ 512K (matches upstream to the digit — do NOT switch back to GMU sizing). - First launcher serve of lab died on both ranks at ~54 s (exit 1, logs lost to `--rm`); re-run was clean twice. Suspected transient (cold imports + simultaneous launch). If it recurs: re-launch without `--rm` and keep logs. - `sm121 serve` immediately after `sm121 stop` can fail with "cluster has only 1 nodes" — occupancy race, nodes not yet released; wait ~10 s and retry. - LibertAIDAI/redhat recipes still run FlashInfer autotune (60-min possible boot, `VLLM_ENGINE_READY_TIMEOUT_S=3600` set); the lab recipe disables it via `--kernel-config` per upstream's proven boot. - Chat-template caveat: `reasoning_effort:"low"` still emits a reasoning trace (budget ≠ 0); `max_tokens:48` can yield `content: null, finish: length` — use ≥256 for smoke tests. ## Follow-ups - eugr B12X variant recipe `glm53-flash-nvfp4-eugr` (2026-09-05): port of eugr/spark-vllm-docker @ 94b3c30 recipes/glm-5.3-flash.yaml on eugr/spark-vllm-b12x:nightly-20260904 (their dev/jovian-judgement 09-04 tip + lukealonso/b12x master; no ENTRYPOINT in image → recipe carries ["vllm","serve"]). A/B cell vs the r18 fork build: full b12x cluster (linear-b12x = r17 toxin retest) + RoCE one-shot all-reduce + MTP k=5. Model: lab GLM-5.3-Flash-NVFP4-Spark (cache on both nodes first). Unmeasured — dry-run verified only. - Vendor instanttensor loader into the lab image (~30 s vs 700 s weight load). - Consider adding `--kernel-config` autotune-off to the LibertAIDAI recipe (boot minutes shorter; redhat now has it, see 512K section — verify before trusting on LibertAIDAI). - MTP for redhat: RESOLVED (2026-09-05) — no variant repo needed: the main `RedHatAI/GLM-5.3-Flash-NVFP4` repo gained `model_mtp.safetensors` (7.6 GB) and the card an "Enable Speculative Decoding" section (k=5, plain `--speculative-config {"method":"mtp",...}` — same model path, no swap). Recipe `glm53-flash-nvfp4-redhat` updated to MTP k=5 (watch the +7.6 GB/node memory cost vs the 10 GiB KV pin; k=3 fallback). The lab's 5.2 pattern (`-mtp-FP8` variant repos) did not apply; the cross-graft MXFP8-MTP idea is moot. ## 4p67: --enforce-eager vs CUDA graphs (2026-08-29, same seed/invocation) Same recipe both arms (MTP k=3 included), only `--enforce-eager` removed; graphs arm verified to capture (FULL 5/5 + PIECEWISE 11/11), KV identical (1,164,369 tok pinned). | metric | eager | graphs | |---|---|---| | quality | **87** (153/176) | **88** (154/176) | | deployability | 67 | 68 | | tg t/s c1 (d0/d4096/d8192) | 21.1 / 23.2 / **24.4** | **23.6** / 22.6 / 22.9 | | tg t/s c2 (d0/d4096) | **39.2** / 22.9 | 33.1 / **24.0** | | TTFT (d0 / d8192) | 1.45 s / 6.17 s | 1.41 s / 6.10 s | | MTP draft accept | 90.0–92.2 % | 89.0–97.0 % | Findings: * **No graphs+MTP quality regression on 4p67** (88 vs 87 = 1 point, run-to-run noise; the failure sets differ only by 5 shuffled partials). The lab 92→89 graphs regression (forum #44) does NOT reproduce on this quant — so the bug is build/quant-specific, not universal. * **Graphs buy nothing on decode here**: every cell within ±5%, and the headline c2 number is *worse* with graphs (39.2→33.1). MTP already eats the launch-gap that graphs would recover. * Prefill rows are prefix-cache-polluted (recipe enables prefix caching and the perf probe repeats prompts); cold prefill ~1.4–1.5 K t/s both arms. * **Decision: `--enforce-eager` stays in `glm53-flash-nvfp4-4p67`** — it is measurably free here and stays immune to the lab graph bug class. Reports: `/tmp/runs/2026/08/2026-08-29T09-09-14…--(eager)`, `…10-27-35.818420Z_181bbd18.md` (graphs). ## sparkrun (community image) A/B (2026-08-30, seed 42) aidendle94/sparkrun-vllm-glm53-gb10:production-1.0 (same day-0 vLLM dev base as our lab image, arm64-only) serving the same lab quant: quality 89 (154/176) but decode tg 14.3/23.8 t/s (c1/c2) vs lab-eager 24.4/39.2 and responsiveness 12/100 (median turn 11.7 s) — works, not performant: - MTP impossible in that image (no modelopt-MTP namespace fix). - --moe-backend marlin must be pinned (their auto = TileLang JIT storm, worker OOM-killed at boot), nvrtc.h bind-mount needed, plain fp8 KV (their mla kernel asserts pe_dim==64 for fp8_ds_mla). Report: /tmp/runs/2026/08/2026-08-30T09-41-57.562243Z_108a798b.md. ## EXL3 lane (Entrpi community image) — boot diagnosis (2026-08-30) ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2 + brandonmusic GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. VERIFIED SERVING 2026-08-30 evening: the loader-side OOM chain (`--load-format instanttensor` overlay) resolved — stock loader balloons ~80 GiB CPU-anon streaming the checkpoint; instanttensor streams ~175 MB/s with tiny buffers. ~22.8 tok/s with DFlash2 k=7 (accept 48-81% by content); math smoke exact. DFlash2 drafter switched to the upstream `incoai/GLM-5.3-Flash-DFlash2` (downloaded to NAS at operator request); MTP3-language-only + v84 image lanes remain SM120-only upstream. Fixed in-recipe: NGC entrypoint (needs explicit `vllm serve`), drafter swap to the MXFP8 mirror (incoai raw drafter absent from the NAS hub), GMU 0.86. BLOCKED: global OOM kills the worker at 75-95% of shard load on BOTH streamed-from- NAS nodes (repro 2×; dmesg oom-killer 12:22:25 / 13:12:24). Upstream's "validated NFS topology" does not hold on our fabric — the mirror topology (per-node local NVMe) is the next step; dgx4 is 98% full (candidate cleanup targets ~1.3T of re-downloadable zai BF16 repos there) — awaiting operator call. ## EXL3 lane bench (2026-08-30 evening, seed 42, same invocation) `glm53-flash-exl3` (brandonmusic GLM-5.3-Flash-tr3-4bpw 4bpw, Entrpi v1-dflash2 image + instanttensor overlay) — quality 88 (155/176, the failure set is the usual 16 shuffled partials/fails; nothing new flags), decode tg 23.1/24.1 t/s (DFlash2 k=7 drafting keeps C1≈C2 — the flattest concurrency profile of any glm53 lane), TTFT 2.6 s @d0, pp ~817-1,004 (noisy, NAS-adjacent). The headline: **deployability 77/100 — the best of the four lanes** (lab 72, 4p67 67, sparkrun 66), driven by responsiveness 50/100 with a 3.0 s median turn. Quality still trails lab-eager-with-MTP3 (92) by ~4 points — the 4bpw price. Report: /tmp/runs/2026/08/2026-08-30T17-43-04.770487Z_63900bf7.md. ## 0rand winning-config trial on the lab quant (2026-08-30 evening) Forum 381350 #117/#119 report 93/100 (164/176) for the lab quant with GMU 0.89, MNBT 2048, KV auto-sized (1.05M pool), MTP-3. Reproduced on ds4 as `glm53-flash-nvfp4-lab-0rand` (variant in tree): KV auto-size lands at 1,167,586 tokens — same capacity as our pinned 10 GiB, so the pin is unnecessary; smoke exact. Bench (--seed 42 --hardmode --perf --concurrency 1,2,4): quality 90 (158/176), tg 27.1/34.6/**37.2** c1/c2/c4, TTFT d0 1.3 s, responsiveness 25 (6.2 s median), deployability 70. vs our lab baseline (92 = 162/176; c1 28.4 / c2 40.5; resp 21, deploy 72): quality −2, c2 decode −15%, c1 ≈, but c4 decode 37.2 in an arm the baseline never ran, and responsiveness +4. On OUR stack the 0rand combo is not a strict win; keep the baseline as the daily driver, keep the variant in tree for concurrency-hungry serving. Possible confounder: their run posts a single 14.8 s median turn at responsiveness 8 — an earlier bench build. Report: /tmp/runs/2026/08/2026-08-30T19-33-22.444571Z_4616f476.md. ## MNBT 1024-vs-2048 assertion on the lab quant (2026-08-30/31 night) Forum #124 ("1024 = quality-first") asserted. Our c1,2,4 run with MNBT 1024 + seqs 16 + GMU 0.89 + auto-KV (pool 1,051,792): quality 88, denominator 146 (harness dropped ~30 pts of scenarios), tg 27.6/27.0/30.4, TTFT d0 8.6 s, resp 24 (6.4 s). vs the same-protocol MNBT-2048 run one hour earlier: quality 90 (158/176, full denominator), tg 27.1/34.6/37.2, resp 25 (6.2 s). Verdict: at our protocol (reasoning_effort-low, temp 0, seed 42) **2048 wins** on both quality percentage and decode; the 1024 advantage was measured under their thinking-max + temp-0.1 + trials-2 protocol. 0rand's #123 already noted "2048 batch + MTP4 ... loses quality (89 vs 93)" for THEIR protocol — different regime, same lesson: batch tuning is protocol-dependent. Variant stays in tree; record both. Reports: 2048 = ...19-33-22Z_4616f476.md, 1024 = ...22-04-48Z_4a70e6be.md. ## DeepSeek-V4-Flash-Vision-Exp lane — campaign state (2026-08-31 night) Day-0 port landed (sfxnz plugin @7a958128b84a vendored under docker-images/dsv4-flash-vision/, bases digest-pinned; sm121 build works, 35-40 min roundtrip). Lane verified TEXT-serving on ds4 (+NVMe/NAS weights): - eugr b12x base: 1024-token gen in 29.4 s = 34.9 tok/s with DSpark-6 (accept 0.714, length 2.55), text math exact, plugin loads ViT+aligner weights (whole checkpoint present, 82.28 GiB/node). - fixing their gaps so far: plugin __init__.py must ship the general- plugins `register` (raw.githubusercontent flaked on re-fetch — fixed via the contents API); wrapper class needs `supports_multimodal = True` (OpenAI layer refuses image_url otherwise — interfaces.supports_multimodal is a getattr check). - OPEN: OpenAI-image path raises "Model class ... has no registered multimodal processor" — the day-0 plugin never registers a _processor_factory (their shipped tests exercise the loader directly, not the API). Two paths: implement the BaseMultiModalProcessor inside our plugin (grid-tokens-per-image machinery exists in dsv4_vision.image_processor; fork interface = MultimodalRegistry. register_processor factory in multimodal/registry.py), or wait for the upstream repo to ship it. aiden-fork A/B build launched. ## dsv4vision MM-into-OpenAI campaign — 2026-09-01 checkpoint Built the full OpenAI multimodal stack in our plugin (processor trio, grid/sentinel tokens, embed_multimodal + embed_input_ids merge; verified in -image: embedding rows byte-exact vs aligner(ViT(patches))[perm]) and got the engine through profiling+warmup with mm registered. Two hard walls remain (before the first image_url token): 1. The fork's V2 "always-embed" mm arm passes `input_batch.input_ids[:n]` — a wrong-rank slice that arrives as [B, T]-shaped 2-D ids (observed (17,17) for a 17-token prompt, (7,7) on DSpark-6 verify steps) — and downstream logits degenerate (all-`<｜begin▁of▁sentence｜>`). 2. The AOT/dynamo trace of inputs_embeds freezes the dummy's 2-D shape and then asserts "wrong number of dimensions3 for op: input" on the native 3-D stream (tilelang mhc kernels are rank-strict both ways). State: the whole mm branch is env-gated (`VLLM_DSV4_MM=1`), default OFF = the verified text-only lane — restored EXACT on the new per-node NVMe weights (~/dsv4vision-hf on dgx3+dgx4, 167 GiB each, recipe HF_HOME flipped from the NAS hub; shard load 7 min → <1 min, boot ~6 min end-to-end). Bridges done tonight for the next attempt: encoder-runner + speculator buffer shape-follow patches (patch_mm_hc_streams.py), placeholder `<｜deepseek_image｜>` = in-vocab id 129264, `get_placeholder_str` chat integration, AOT=0 knob for mm experiments. nid: fork A/B (aiden base) still pending its pip-layout patch; dspark_gate patch now auto-finds the fork's venv root. ### dsv4vision tool-eval-bench (c1, 2026-09-01) `tool-eval-bench run --base-url http://dgx3.lan:30000/v1 --model deepseek-v4-flash-vision --seed 42 --hardmode --perf --concurrency 1` → report `/private/tmp/runs/2026/09/2026-09-01T17-46-28.764372Z_89b18250.md`: | recipe | quality | points | decode tg t/s c1 (d0/d4096/d8192) | pp2048 t/s | TTFT | deployability | |---|---|---|---|---|---|---| | `dsv4-flash-vision` (text lane, local-NVMe, DSpark-6) | **81** | 143/176 | — | — | — | — | perf block: tg 34.1 / 31.0 / 29.1, pp2048 3,907 / 2,714 / 2,449, TTFT 1.9-4.9 s (spec), scenario-median turn 2.4 s, deployability **74** (best of any ds4 lane) — long-context capability is the raw 1M-context draw. 3 safety-critical failures (TC-42 extra-param, TC-43 empty required param, TC-60 sleeper injection) — the day-0 0731-style tool formatting is the gap vs the GLM lanes (thinking-pair JSON fencing was already a known day-0 weakness). Quality 81 < sparkrun 89 < lab 92. ## New 2026-09-01 checkpoints: Intel W4A16-AutoRound + lab NVFP4-Spark Checkpoints on the NAS hub (hf download'd): - `Intel/GLM-5.3-Flash-W4A16-AutoRound` rev 5eee1846f0321058ed73745f9aa16f2aaf0fc0a0 — AutoRound 0.15 W4A16 sym int4 g168, 99.84% of BF16 (GSM8K .9712/MMLU .8620), self_attn/hc_/indexer/embed lm_head FP16 by design (their ignore_layers), pure GPTQ packing (qweight/qzeros/scales — qweight count 37152). - `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` rev 06972cdee7280cec849aedf05a02062b3e870246 — NVFP4 W4A4 *whole network* (in_proj_qkvbfg_a.weight_scale etc.) + NVFP4 MTP expert layer in separate model-mtp-0000x shards (lil.yaml: mtp_tokens 5, dflash2_tokens 7, default_speculator mtp, block 256, disable flashinfer autotune, INSTANTTENSOR env sizes; arch Glm5NextForConditionalGeneration + vision blocks in the index). This is the rtx6kpro r12-stack checkpoint (their doc: MTP:3 254 tok/s CC1 @ TP4, Sieve 333-403). Recipes added: `recipes/glm53-flash-intel-w4a16` (timothystewart6/vllm-gb10 v0.28.0-gb10.2 arm64, gptq_marlin, entrypoint declared — tini AssertionError taught us the image has ENTRYPOINT []+CMD bash), `recipes/glm53-flash-nvfp4-spark` (lab image). Boot outcomes: - Intel lane died in the fork's Transformers-backend weight loader: "no module or parameter named 'layers.0.self_attn.q_conv1d'" — the tjstewart image has NO native glm5_next arch (vllm archs list lacks it; transformers 5.16.1 has Glm5NextForConditionalGeneration) → needs the local-inference-lab fork (native glm5next) or a transformers name-shim mod. - NVFP4-Spark on our lab fork: KeyError on `in_proj_qkvbfg_a.weight_scale` at 80% shard load — whole-network NVFP4 W4A4 is beyond the R8-era lab fork loader (r12 stack territory). ⇒ THE unifying next step (user directive): build our own arm64/sm_121a image of local-inference-lab's fork — source contract from models/glm-5.3-flash.md (local-inference-lab/vllm dev/jovian-judgement + b12x master + 6/5 PRs; kk3 build rules apply: CUDA 13.0.88, sm_121f gencode rule for block-scaled FP4). RTX docs archive: /tmp/lil-docs (ds4-flash-v6, ds4dspark-v20-r33, infernal r21, pcie-autotune) for the DeepSeek quality track. ## 2026-09-02: r12 fork built + NVFP4-Spark serving on it (the day's win) Built `glm53-jovian-r12:local` in-docker on dgx3 (docker-images/glm53-jovian-r12/): vllm 0.1.dev20489+gcc161e220 (local-inference-lab/vllm base 54f6e9826c + PR heads 550/552/553/554/555/557 merged in order — the doc's "composition commit" 7d8a09c is not a public ref) + b12x 1.3.0 (master 139e0405 + PR heads 260-270), on the day-0 `vllm/vllm-openai:glm53-flash-arm64-cu130@905c02…` base. Build fixes applied along the way: apt git, setuptools-rust, ninja/cmake, cu13 wheel include-header symlinks (cusparse.h), unversioned libnvrtc.so symlink for the deepgemm_C link step. NVFP4-Spark boot on it (recipe glm53-flash-nvfp4-spark, retargeted): - whole-network NVFP4 W4A4 (incl. attention in_proj_qkvbfg scales) now LOADS — the r12 fork's modelopt handles it (our lab fork: KeyError). - Attention wall solved: the sm12x auto-priority list only offers [TRITON_MLA (no sparse), FLASHINFER_MLA_SPARSE_SM120 (head 576 only)] — neither fits GLM head-512. Fix = `--attention-backend B12X` (model hook in glm5next/nvidia/attention.py: _select_sparse_backend) for the main model, and `"attention_backend":"B12X"` INSIDE --speculative-config for the MTP draft (eagle/utils.py: _make_eagle_draft_vllm_config takes the draft's backend from the speculative config, not the global flag — user's hint). - Served: exact "391" verified, MTP-3 Glm5NextMTPModel resolved, KV fp8_ds_mla default, 2-node TP2. c1 bench running. - Intel W4A16 recipe retargeted to the same image (gptq_marlin + B12X attn). - patch_cuda_sm12x.py (B12X into sm12x priorities) staged in the image dir for the next rebuild — the image-baked version of the same fix. ## 2026-09-02 (cont): bench + clean rebuild per operator direction - NVFP4-Spark (r12 fork image, day-0 base) c1 bench: **88/100** (120/136, default 69-scenario suite; deployability 67, median turn 8.1s — TP2 GB10 is slower per-turn than the eugr vision lane's 2.2s on the same suite, also 88). Same-suite A/B vs vavrusa lane: 88 = 88. The lab's 92 was the 88-scenario/176-point suite — that invocation's re-run is pending. - Operator: docker_image_dir in the spark recipe pointed at the lab dir — fixed to glm53-jovian-r12 (intel recipe same). - Operator: "start with cuda image, solar-open2 pattern but r12-specific". Dockerfile/versions.env rewritten: raw nvidia/cuda:13.2.0-devel-ubuntu24.04 (a5b6256e…, solar-pinned) → apt stage (solar locks) → torch 2.13.0 cu130 + flashinfer 0.6.17 (whl index) → vllm wheel (fork base 54f6e9826c + PR-head merges 550-557, use_existing_torch, ccache) → b12x wheel (master + PR heads 260-270, pure-python) → runner installs both wheels + deps, bakes patch_cuda_sm12x.py, GB10 gate (vllm/b12x/_flashkda_C imports, B12X enum, patch presence). No day-0 vllm contamination; ENTRYPOINT [] CMD bash. Build running on dgx3 (tmux r12build → /tmp/r12build2.log), same tag glm53-jovian-r12:local (running serve keeps the old image; re-verify lane after build). ## 2026-09-02 (cont 2): lil — the lab's own launcher, read local-inference-lab/lil = standalone Go launcher (their sm121 equivalent): reads each HF model repo's lil.yaml at runtime + embedded family policy + discovered topology (local or spark-rdma via SSH), renders/validates/runs a typed vLLM invocation. `lil render ` prints the exact env + command. GLM family policy (configs/models/_bases.yaml) — the lab's canonical launch for these checkpoints, vs our spark recipe today: - --attention-backend B12X, --linear-backend b12x, --moe-backend b12x (we have only the global attention flag; linear/moe backends MISSING) - spec config: {"method":"mtp","num_speculative_tokens":5 (lil.yaml mtp_tokens), "moe_backend":, "attention_backend":"B12X"} — we use k=3 and no moe_backend. MTP:3 is the r12 x86-qualified mode; the checkpoint's manifest default is 5. - --load-format instanttensor (+INSTANTTENSOR_BUFFER_SIZE/CHUNK_SIZE env) - --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096, --max-model-len auto - --reasoning-parser glm45, --tool-call-parser glm47 - DFlash2 (the GLM-side second speculator, method "dflash") needs the separate draft checkpoint local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8. TODO: after the clean image lands, align the spark recipe to the lil policy (moe/linear backend b12x, spec moe_backend, instanttensor, mamba-cache-mode, parsers) and A/B k=3 vs k=5. ## Clean r12 build: fix log (builds 2-9) 1. flashinfer: whl-index path dead → flashinfer-python 0.6.17 from PyPI (fork's setup.py skips flashinfer-cubin at build; runtime fetches cubins). 2. git protocol v2 flake in-container → GIT_PROTOCOL=0 + protocol.version 0. 3. setuptools: fork build needs setuptools-rust + setuptools-scm (uv hint). 4. DeepGEMM FetchContent: pinned SHA on non-default nv_dev branch → GitHub 404s raw-SHA fetch → DEEPGEMM_SRC_DIR pre-clone stage (cmake escape hatch). 5. PR 550 closed (unmerged) → refs/pull/550/head deleted; all PR heads now pinned by commit SHA (vllm 550-557, b12x 260-270). Raw-SHA fetches verified working. 6. Operator rule: no build while a serve container runs (OOM risk on 128 GB unified) — spark lane stopped for the build window. ## 2026-09-02 (cont 3): clean r12 image SHIPPED Image glm53-jovian-r12:local rebuilt from raw nvidia/cuda:13.2.0-devel (solar pattern, r12-specific content): 16.1 GB (vs 37.6 GB day-0-based), gates green (vllm 0.1.dev20489+g06f922207, b12x, _flashkda_C, B12X enum, sm121 patch), synced dgx3→dgx4. Boot fixes found on first clean boot: - tini "[FATAL] exec local-inference-lab/… failed" → clean image is ENTRYPOINT [] (solar contract), so the RECIPE now declares entrypoint = ["vllm","serve"] (the old day-0 image had it baked in). - FlashInfer JIT OOM: 12 parallel nvcc jobs (mxfp8_gemm_cutlass_sm120 etc.) get OOM-killed (code 137) while 93 GB weights + 10 GB KV are resident on 121 GB unified → MAX_JOBS=2 in recipe [env]; JIT (~15 min) then survives. Boot total ~35 min (NFS shards 16 min + JIT 15 min). c1 bench (default 69-scenario suite, same as the vavrusa run): **90/100** (124/138; quality 90, deployability 69, median 7.9 s) — vs 88 on the day-0-based image, same suite. Clean image is now the reference GLM lane. Remaining lab-alignment (next boot, one change-set at a time): --linear-backend b12x + --moe-backend b12x, spec moe_backend:b12x, A/B k=3 vs k=5 (lil default), --load-format instanttensor (+env), --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096, --reasoning-parser glm45, --tool-call-parser glm47. Then: Intel W4A16 boot on the clean image (queued, needs cluster). ## 2026-09-02 (cont 4): lab-stack perf push — eugr build pipeline adopted Goal (user): 30 tok/s in --perf. Found the missing pieces: 1. **lil render reference** (built lil from local-inference-lab/lil @ cb58d54, rendered with testdata manifest GLM-5.3-Flash-NVFP4-Spark + spark.yaml): our recipe flags now 100% match the family policy (_bases.yaml glm: b12x linear/moe, B12X attention, instanttensor, block 256, seqs 8, batched 4096, glm45/glm47, mamba align) + spec {mtp:5, moe_backend b12x, B12X} + env VLLM_USE_AOT_COMPILE=1 / V2_MODEL_RUNNER=1 / FLASHINFER_SAMPLER=1 / BREAKABLE_CUDAGRAPH=0. Boot 35m -> ~7m (instanttensor loader, user: keep). Perf after engine switch: d0 c1 24.2 / c4 31.8 tok/s (was 10.3/16.5). Quality dropped with full lil set (16/22 at TC-22) -> bisection in progress; user: instanttensor is fine, problem elsewhere. 2. **eugr/spark-vllm-docker** (the lab's actual build repo, main): 804-line Dockerfile, CUDA 13.0.2 base, 12 build-time vllm patches (docker/patch_vllm_*.py), b12x from source (lukealonso/b12x; lab: local-inference-lab/b12x master), nightly DockerHub images eugr/spark-vllm[-b12x]:latest (stale 08-23, user: dockerhub is the wrong source — build from source). Ported all 12 patches into docker-images/glm53-jovian-r12/patches-eugr/ + Dockerfile (VLLM_PRESERVE_SM12X_TARGET=1, VLLM_PATCH_B12X_C128A_ALIGNMENT=1; patches are idempotent, several no-op on our r12 ref). Build blocker: node github.com access flaky ("could not read Username" on FetchContent clones) -> pre-fetched all 9 sub-deps on the Mac into deps-src.tar.gz (triton_kernels@v3.5.1, MSA, FlashMLA, FlashKDA, flash-attention, qutlass@e74319e, tml-fa4@b206834, cutlass@v4.4.2) + the SRC_DIR env escapes each cmake documents. Build 14 compiling. ## r17 (20260902-r17 card) adoption + quality bisect (2026-09-03) Latest lab card: https://github.com/local-inference-lab/rtx6kpro/blob/master/models/glm-5.3-flash.md (names the qualified serving contract per artifact and SHA; TP4 on RTX PRO, not Spark — the Spark lil.yaml testdata render remains our TP2 adapter). 1. **Public r17 composition** built from source: `/tmp/vllmr12` = local-inference-lab/vllm `dev/jovian-judgement@9c4dd054` + open PRs **579/580/581/582/586** (branch supersedes the r12 PR set 550–557 — same symbols, evolved, see below) + staged-cutlass cmake patch; `/tmp/b12xb17` = local-inference-lab/b12x `master@2bbab479` + PRs **290/284**. Pre-staged as vllm-src.tar.gz / b12x-src.tar.gz on the build node (WAF blocks node→github git). DO NOT re-merge 550-557: creates conflict-marker SyntaxErrors (pr550's pooled_indexer.py conflicts; branch tip already contains the evolved implementation). 2. **Important correction to the 2026-08-28 eager verdict**: on the public r12/r17 refs, graphs+MTP is no longer a quality bug — the lab's private "packed GLM-5.3 package" (card SHAs 47f86087/a02841bc, not public) was the presumed fix carrier, but our public-tree composition measured: - r17 + graphs + 90/100-flags (no b12x/align): **88/100 hardmode (155/176) == lab-quant's own day-0 reference 156/176** (r17graphs01/…perf) - r17 + eager same-flags: 67/100, median turn 11.4 s, 2 scenarios timed out and excluded (r12eager01) — eager is now the WORST mode. So: graphs stay ON; enforce-eager is gone. 3. **Execution-env knobs are quality-clean**: GMU 0.90 (0.95 boot-deaths on GB10 — "free memory 111.96 < desired"), max-num-seqs 16, OMP 2, VLLM_USE_STANDALONE_COMPILE/FLASHINFER_SAMPLER/BREAKABLE envs: 87/100 (r17knobsonly). AOT=1+V2 runner: NEUTRAL for quality (47 either way when backends were on) and NEUTRAL for the d0 grid — dropped (AOT=0, no V2). 4. **THE QUALITY TOXIN = the b12x backend-flag cluster** (on the PUBLIC r17/r12 compositions, with our b12x 1.3.0/1.2.6 builds): - k=5 full cluster: 74/100 (r12 img), ~74 → killed-to-16/22 (r12 img) - k=5 full cluster (r17 img): ~70-ish band (r17cluster01, stopped early) - k=3 full cluster: **46/100** (r17clusterk3) - k=3 cluster minus AOT/V2: **47/100** (r17noaotk3) - k=3 cluster minus backends (knobs only): **87/100** (r17knobsonly) The lab's own stack runs these backends qualified — but their composition carries the private packed GLM-5.3 package (RTX-PRO target); the public branch tip alone does NOT make the b12x W4A4/linear/align paths quality-safe. Also: b12x cluster wrecked deep-context speed (d4096 c1 12.6 @k5) while helping d0 (33-34 c1) — nets negative anyway. 5. **SHIP ANCHOR (current recipe)** = r17 image (94ac0882a8b7) + 90/100 args (attention B12X, block 256, seqs 16, batched 4096, LM-only, glm47/glm45 parsers, instanttensor load, k=3 spec {mtp,attn B12X} with moe_backend key) + graphs (no eager) + knobs env (GMU 0.90, seqs 16, OMP 2, AOT=0, STANDALONE=1, SAMPLER=1, BREAKABLE=0) + JIT-cache mounts: **87/100 hardmode (153/176; lab-quant ref 156/176), default-suite 88 class**; perf d0 c1 24.1 / c2 30.2 / c4 44.7; d4096 c1 23.9; d8192 c1 23.8; pp ~1.6-1.8k. Boot 8 min (JIT warm) / ~25 min cold (nvcc re-JIT). 6. Boot-time JIT: changing the rw flashinfer cache invalidates per-config kernel sets; a config's first boot silently nvcc-compiles (MAX_JOBS=2) ~15-25 min inside "No available shared memory" log-silence; later boots of the same config are fast. Expect this on every NEW kernel path. 7. KV-dtype probes: fp8_ds_mla on r12-img = 75/100 default-suite (net zero, TC-45 flipped PASS but others flipped off); graphs here. r17 anchor keeps default dtype. Not a quality lever. 8. Open perf gap: c1 24 vs manifest-cluster 33-34 (which is quality-dead). If the lab ever publishes the packed package (or a public branch commit makes b12x-backends clean), the 30+ c1 target lands with manifest flags. Alternative to probe later: one b12x backend flag at a time (a single toxin member may be free). ## One-backend bisect (2026-09-03) — SOLVED (hardmode, r17 img 94ac0882, k=3 graphs) - knobs-only: 87 (153/176); d0 c1 24.1, d4096 23.9, d8192 23.8 (r17knobsonly) - + `--moe-backend b12x` (spec default MTP experts): **88 (154/176)**; d0 23.5, d4096 22.5, d8192 **26.6** (r17moeonly) — quality-clean, keep - + `--linear-backend b12x`: **44 (77/176)**, Parameter Precision 0%, TC-21 0/5 validations — THE toxin. B12X linear kernel path destroys numerics on the public composition (r17linearon) - + `--mamba-cache-mode align`: 88 (154/176) but d0 c1 18.6 (−4.9), no quality gain (r17alignon) — perf-negative, dropped - Full-cluster combos (44-74) were dominated by linear; spec's moe_backend b12x key and the knobs env family are all exonerated individually. **SHIP = r17 img + 90/100 args + graphs + knobs env + `--moe-backend b12x` only** (recipe @ 2d379f9): 88/100, d0 23.5 / c4 43.9, depth 22.5/26.6. The 30 c1 target is only reachable with linear-b12x (29.6 d0) which is quality-dead until the lab publishes the packed GLM-5.3 package. ## Checkpoint A/B on ds4 (2026-09-03, hardmode 176pt): Spark vs NVFP4 lab checkpoint User-flagged must-have: our lane served the Spark-variant checkpoint with quality losses vs lab references; A/B the checkpoint under matched stacks. | Cell | Stack | Checkpoint | Hardmode | d0 c1 | d4096 c1 | d8192 c1 | notes | |---|---|---|---|---|---|---|---| | r17ablabimg | lab img + lab recipe (eager, fp8_ds_mla, k3, default MoE, day-0 fork 0.1.dev20051+g487ecf187) | NVFP4 | **93 (164/176)** ★★★★★ | 24.4 | 26.5 | 22.8 | 78 pass / 2 fail; TC-45 PASS; matches lab's own 92 ref | | r17ablabck | **shipped r17 img** + ship flags (graphs, knobs, k3, default MoE — moe-b12x unsupported on this ckpt: layer-45 is MXFP8) | NVFP4 | **88 (154/176)** | **27.5** | **29.6** | 26.6 | 46.4/46.7 c2/c4 at d0 | | r17moeonly (ship) | shipped r17 img | Spark | 88 (154/176) | 23.5 | 22.5 | 26.6 | — | Verdict: 1. **Checkpoint is not the quality factor on the r17 stack** (88 = 88 both checkpoints) — but the NVFP4 lab checkpoint is significantly FASTER (+4 tok/s d0, +7 d4096) and lighter rules (no --moe-backend b12x). 2. **The lab day-0 stack retains +5 quality (93 vs 88) regardless of checkpoint** — its eager + fp8_ds_mla + default-everything runtime is what the lab's own 92-hardmode reference was measured on. Enforce-eager costs prefill pp at c2/c4 (0.9-1.1k vs 1.5-1.6k) but TG is equal-or-better. 3. Recipes: `glm53-flash-nvfp4-r17abi` (ship flags + NVFP4, moe flag removed) is the fastest quality-parity lane; `glm53-flash-nvfp4-lab` remains the quality champion (93). ## Night-shift queue (2026-09-05) Prepared lanes + builds queued for the night (cluster is serving the 4x aiden lane during the day — stop it first, then all 4 nodes are free). Order = value/duration; each item lists prereqs + expected verdict. 1. **r18 fork build** (docker-images/glm53-jovian-r12, ~1-2 h build). Stage on the build node: /tmp/vllm-src.tar.gz (94 MB) + /tmp/b12x-src.tar.gz (29 MB) — pins per versions.env (vllm a8c796f3+pr582+pr646 → 343661fe; b12x b58f34ea+pr284 → 1adb8a38), then `TAG=glm53-jovian-r12:local bash build.sh`. Test campaign: (a) **Re-baseline the quality anchor first** (tool version changed — see the bench header): run the anchor lane glm53-flash-nvfp4-r17abi (2x, known-good at 88/100 on the old pool) with the current tool: `tool-eval-bench run --base-url http://:30000/v1 --model glm53 --seed 42 --hardmode --perf --concurrency 1,2` (88 scenarios, ≈26 min), record the new baseline score in this file, save the report to recipes/glm53-flash-nvfp4-r17abi/reports/; gate ALL new lanes (r18, eugr 2x/4x, redhat-MTP, dsv4-vision) against the NEW baseline — never against the old /176 numbers. Campaign reports always go to recipes//reports/_