Trang chủ Khám phá Trợ giúp
Đăng nhập
sm121
/
launcher
2
0
Fork 0
Các tập tin Các vấn đề 0 Yêu cầu kéo về 0 Wiki
Tree: 3e63d19947
Branches Tags
launcher
/
specs
/
tech-architecture
/
glm53-recipe-state.md

glm53-recipe-state.md 53 KB
Lịch sử Raw

GLM-5.3-Flash NVFP4 — three quant variants measured on 2× DGX Spark (ds4)

Measured 2026-08-27/28 on cluster ds4 (dgx3+dgx4, switched fabric), all three served by sm121 through the launcher, same benchmark invocation:

tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53 \
  --seed 42 --hardmode --perf --concurrency 1,2

Tool version pin (2026-09-05, verified against the installed tool): tool-eval-bench 2.6.1.dev45+gcf54b4bfe (uv tool; git SeraphimSerapis/tool-eval-bench @ cf54b4bfe). Current pool semantics: --hardmode = 69 standard (A–O) + 19 Hard Mode (P) = 88 scenarios (≈26 min, dry-run verified); --hardmode-only = 19 (≈6 min). The /176 scores above (155/176 anchor, lab ref 156/176) are pinned to the OLDER 176-point pool — scores do NOT transfer across tool versions. Every campaign now re-baselines the anchor lane with the CURRENT tool first (night queue 1a), then gates new lanes against that new baseline.

All three share served name glm53 — clients never changed between runs.

Results

recipe quant quality points decode tg t/s (c1) tg c2 TTFT deployability MTP
glm53-flash-nvfp4-lab local-inference-lab mixed NVFP4/MXFP8-MTP 92 (eager) / 89 (graphs) 162/176 28.4 40.5 5.6 s 72 ✅ k=3, accept len ~2.9–3.5, ~63–83 %
glm53-flash-nvfp4-redhat RedHatAI W4A4 compressed-tensors 90 159/176 13.9 13.8 2.2 s 67 ❌ (no MTP weights in checkpoint)
glm53-flash-nvfp4 LibertAIDAI NVFP4-A16 88 154/176 26.2 22.7 6.3 s 69 ✅ k=5
glm53-flash-nvfp4-4p67 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 87 153/176 24.4 39.2 1.4 s 67 ✅ accept ~90%
glm53-flash-aiden-4x aiden image production-1.1 (DFlash2 overlay), NVFP4, incoai DFlash2 k=7, eager, 16 GiB KV pin, 4× Spark TP4 (his production shape) 89 156/176 37.6 (d0 c1; flat to d8192, 2,185 pp t/s — matches his ~2,100) 50.7 (c2) / 63.5 (c4) 0.9 s 80 ✅ DFlash2 k=7
glm53-flash-aiden-2x aiden image production-1.1 (DFlash2 overlay), NVFP4, incoai DFlash2 k=7, eager, 4 GiB KV pin (2x profile) 89 157/176 22.3 (d0 c1; d8192 c4 degrades — 4 GiB pin) — — 66 ✅ DFlash2 k=7 (drafter block 8−1)
glm53-flash-sparkrun community sparkrun image (aidendle94), NVFP4, eager, NO MTP (image lacks the fix) 89 154/176 14.3 23.8 1.4 s 66 ❌ (n/a in image)
glm53-flash-exl3 brandonmusic EXL3/TR3 4bpw (Entrpi image + instanttensor), DFlash2-7, graphs 88 155/176 23.1 24.1 2.6 s d0 77 ✅ accept 48–81 %

(lab scored /174 — TC-53 was excluded from its run as an infrastructure failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s total — harness drops timed-out scenarios from the denominator; had it scored like redhat's 2/2, lab would be 157/176 = 89 — no change). Quality % is comparable. LibertAIDAI is the only variant whose MTP regresses under c2 (26.2→22.7) — lab's k=3 draft scales, LibertAIDAI's k=5 does not.

4x note (2026-09-04): the 4x aiden run had one safety-critical failure — TC-43 called web_search with an empty query (required-parameter violation); not seen in the 2x aiden run.)

Prefill (pp2048) 0.3–1.7 K t/s depending on depth/concurrency — noisy, all three in the same band (NAS load-influenced); decode is the stable discriminator.

Read of the numbers

  • lab (eager) wins outright: 92/100 best quality AND 2× decode (28–40 t/s vs ~14 baseline). Acceptance lengths matched upstream's 2.55–3.5 range.
  • redhat (90/100 with graphs) is a baseline-decode dead-end on this box: its checkpoint ships zero MTP tensors (verified index: 146,745 tensors, none under layer 45), so it is stuck at ~14 t/s — no longer the quality pick either.
  • lab --enforce-eager = day-driver; redhat = fallback; LibertAIDAI = proven first but k=5 MTP regresses at c2 (eager might lift it too — untested).

Images & patches

image dir base patches
glm53-flash-nvfp4 vllm/vllm-openai:glm53-flash-arm64-cu130 kingjones30 NoPE-MLA mod (4-part, anchor-asserted)
glm53-flash-nvfp4-lab same + 4 FujitsuPolycom patches (Apache-2.0, see patches/UPSTREAM.md): model.py naming shim, modelopt MTP namespace fix, persistent_topk CC-12.x guards ×2
redhat recipe reuses glm53-flash-nvfp4:local (mod is quant-independent)

Boot markers [quantprobe] layer=RoutedExperts prefix=model.layers.45.mlp.experts algo=MXFP8 confirm the MTP fix is live; algo=None there = broken load.

MTP + CUDA graphs quality bug (forum #381350/#44, CONFIRMED 2026-08-28)

0rand: “mtp or dflash plus cuda graphs seriously hit quality — try disabling cuda graph capture”. A/B with identical seed/flags on lab:

arm quality points tg c1 tg c2 TTFT
graphs (CUDA graphs ON) 89 155/174 28.4 43.4 3.9 s
--enforce-eager 92 162/176 28.4 40.5 5.6 s

+3 quality points for −7 % c2 decode — eager is now the lab default. Accept metrics were healthy in BOTH arms (71.6 %, len 3.15 eager), so this is not an acceptance collapse — graph capture corrupts sampled output silently. Corroborating: randomllama (DFlash2 recipe) already flagged “not claiming bit-identical greedy outputs” with graphs on. The b12x glm52 stack runs graphs + MTP k=4 FULL_AND_PIECEWISE benches clean (100/100) — build-specific bug in the glm53 day-0 dev build, do NOT propagate --enforce-eager to glm52. Untested follow-ups: LibertAIDAI + eager (k=5; expect ~88→9x), redhat is graph-neutral (no MTP, stays 90 with graphs).

Operational notes (measured)

redhat @ 524288 ctx (2026-08-28)

262144 booted on the base image; 512K needed three changes (each verified by a real boot that died without it):

  1. --kernel-config {"enable_flashinfer_autotune":false,"enable_cutedsl_warmup":false}
    • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — autotune scratch at 512K shapes OOMs the driver (NVRM NV_ERR_NO_MEMORY _memdescAllocInternal); the sparse-MLA kernels' non-autotuned fallback is fine (lab proved it).
  2. --max-num-batched-tokens 4096 + --block-size 256 (lab's memory shape).
  3. The lab image — the killer was launch_persistent_topk ... total_ctas=62 > num_sms*occupancy=48 ... fallback requires >=128KB smem (have 101376), i.e. exactly what the FujitsuPolycom sparse_attn_indexer(_kpool) CC-12.0 guards fix. Indexer CTAs scale with ctx: 262K fit 48 SMs, 512K needs 62. The patches are arch-level → safe for the compressed-tensors checkpoint (quantprobe is a no-op without modelopt; output verified correct).

Result: KV pool 1,334,265 tokens / 2.54× @ 524288 (10 GiB pin; lab gets 1,164,369 from the same pin — its MTP speculator reserves memory before KV sizing), ~17 min clean boot, correct greedy output.

Note: an orphaned rank-1 headless worker survives rank-0 death and keeps the node occupied — always docker rm -f label-filtered leftovers before a retry serve (or the launcher refuses with "only 1 nodes").

  • lab weight load: 92.19 GiB / 700 s sequential from the NAS HF cache; --kv-cache-memory-bytes 10737418240 → KV 1,164,369 tokens / 2.22× @ 512K (matches upstream to the digit — do NOT switch back to GMU sizing).
  • First launcher serve of lab died on both ranks at ~54 s (exit 1, logs lost to --rm); re-run was clean twice. Suspected transient (cold imports + simultaneous launch). If it recurs: re-launch without --rm and keep logs.
  • sm121 serve immediately after sm121 stop can fail with "cluster has only 1 nodes" — occupancy race, nodes not yet released; wait ~10 s and retry.
  • LibertAIDAI/redhat recipes still run FlashInfer autotune (60-min possible boot, VLLM_ENGINE_READY_TIMEOUT_S=3600 set); the lab recipe disables it via --kernel-config per upstream's proven boot.
  • Chat-template caveat: reasoning_effort:"low" still emits a reasoning trace (budget ≠ 0); max_tokens:48 can yield content: null, finish: length — use ≥256 for smoke tests.

Follow-ups

  • eugr B12X variant recipe glm53-flash-nvfp4-eugr (2026-09-05): port of eugr/spark-vllm-docker @ 94b3c30 recipes/glm-5.3-flash.yaml on eugr/spark-vllm-b12x:nightly-20260904 (their dev/jovian-judgement 09-04 tip

    • lukealonso/b12x master; no ENTRYPOINT in image → recipe carries ["vllm","serve"]). A/B cell vs the r18 fork build: full b12x cluster (linear-b12x = r17 toxin retest) + RoCE one-shot all-reduce + MTP k=5. Model: lab GLM-5.3-Flash-NVFP4-Spark (cache on both nodes first). Unmeasured — dry-run verified only.

  • Vendor instanttensor loader into the lab image (~30 s vs 700 s weight load).

  • Consider adding --kernel-config autotune-off to the LibertAIDAI recipe (boot minutes shorter; redhat now has it, see 512K section — verify before trusting on LibertAIDAI).

  • MTP for redhat: RESOLVED (2026-09-05) — no variant repo needed: the main RedHatAI/GLM-5.3-Flash-NVFP4 repo gained model_mtp.safetensors (7.6 GB) and the card an "Enable Speculative Decoding" section (k=5, plain --speculative-config {"method":"mtp",...} — same model path, no swap). Recipe glm53-flash-nvfp4-redhat updated to MTP k=5 (watch the +7.6 GB/node memory cost vs the 10 GiB KV pin; k=3 fallback). The lab's 5.2 pattern (-mtp-FP8 variant repos) did not apply; the cross-graft MXFP8-MTP idea is moot.

4p67: --enforce-eager vs CUDA graphs (2026-08-29, same seed/invocation)

Same recipe both arms (MTP k=3 included), only --enforce-eager removed; graphs arm verified to capture (FULL 5/5 + PIECEWISE 11/11), KV identical (1,164,369 tok pinned).

metric eager graphs
quality 87 (153/176) 88 (154/176)
deployability 67 68
tg t/s c1 (d0/d4096/d8192) 21.1 / 23.2 / 24.4 23.6 / 22.6 / 22.9
tg t/s c2 (d0/d4096) 39.2 / 22.9 33.1 / 24.0
TTFT (d0 / d8192) 1.45 s / 6.17 s 1.41 s / 6.10 s
MTP draft accept 90.0–92.2 % 89.0–97.0 %

Findings:

  • No graphs+MTP quality regression on 4p67 (88 vs 87 = 1 point, run-to-run noise; the failure sets differ only by 5 shuffled partials). The lab 92→89 graphs regression (forum #44) does NOT reproduce on this quant — so the bug is build/quant-specific, not universal.
  • Graphs buy nothing on decode here: every cell within ±5%, and the headline c2 number is worse with graphs (39.2→33.1). MTP already eats the launch-gap that graphs would recover.
  • Prefill rows are prefix-cache-polluted (recipe enables prefix caching and the perf probe repeats prompts); cold prefill ~1.4–1.5 K t/s both arms.
  • Decision: --enforce-eager stays in glm53-flash-nvfp4-4p67 — it is measurably free here and stays immune to the lab graph bug class.

Reports: /tmp/runs/2026/08/2026-08-29T09-09-14…--(eager), …10-27-35.818420Z_181bbd18.md (graphs).

sparkrun (community image) A/B (2026-08-30, seed 42)

aidendle94/sparkrun-vllm-glm53-gb10:production-1.0 (same day-0 vLLM dev base as our lab image, arm64-only) serving the same lab quant: quality 89 (154/176) but decode tg 14.3/23.8 t/s (c1/c2) vs lab-eager 24.4/39.2 and responsiveness 12/100 (median turn 11.7 s) — works, not performant:

  • MTP impossible in that image (no modelopt-MTP namespace fix).
  • --moe-backend marlin must be pinned (their auto = TileLang JIT storm, worker OOM-killed at boot), nvrtc.h bind-mount needed, plain fp8 KV (their mla kernel asserts pe_dim==64 for fp8_ds_mla). Report: /tmp/runs/2026/08/2026-08-30T09-41-57.562243Z_108a798b.md.

EXL3 lane (Entrpi community image) — boot diagnosis (2026-08-30)

ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2 + brandonmusic GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. VERIFIED SERVING 2026-08-30 evening: the loader-side OOM chain (--load-format instanttensor overlay) resolved — stock loader balloons ~80 GiB CPU-anon streaming the checkpoint; instanttensor streams ~175 MB/s with tiny buffers. ~22.8 tok/s with DFlash2 k=7 (accept 48-81% by content); math smoke exact. DFlash2 drafter switched to the upstream incoai/GLM-5.3-Flash-DFlash2 (downloaded to NAS at operator request); MTP3-language-only + v84 image lanes remain SM120-only upstream. Fixed in-recipe: NGC entrypoint (needs explicit vllm serve), drafter swap to the MXFP8 mirror (incoai raw drafter absent from the NAS hub), GMU 0.86. BLOCKED: global OOM kills the worker at 75-95% of shard load on BOTH streamed-from- NAS nodes (repro 2×; dmesg oom-killer 12:22:25 / 13:12:24). Upstream's "validated NFS topology" does not hold on our fabric — the mirror topology (per-node local NVMe) is the next step; dgx4 is 98% full (candidate cleanup targets ~1.3T of re-downloadable zai BF16 repos there) — awaiting operator call.

EXL3 lane bench (2026-08-30 evening, seed 42, same invocation)

glm53-flash-exl3 (brandonmusic GLM-5.3-Flash-tr3-4bpw 4bpw, Entrpi v1-dflash2 image + instanttensor overlay) — quality 88 (155/176, the failure set is the usual 16 shuffled partials/fails; nothing new flags), decode tg 23.1/24.1 t/s (DFlash2 k=7 drafting keeps C1≈C2 — the flattest concurrency profile of any glm53 lane), TTFT 2.6 s @d0, pp ~817-1,004 (noisy, NAS-adjacent). The headline: deployability 77/100 — the best of the four lanes (lab 72, 4p67 67, sparkrun 66), driven by responsiveness 50/100 with a 3.0 s median turn. Quality still trails lab-eager-with-MTP3 (92) by ~4 points — the 4bpw price. Report: /tmp/runs/2026/08/2026-08-30T17-43-04.770487Z_63900bf7.md.

0rand winning-config trial on the lab quant (2026-08-30 evening)

Forum 381350 #117/#119 report 93/100 (164/176) for the lab quant with GMU 0.89, MNBT 2048, KV auto-sized (1.05M pool), MTP-3. Reproduced on ds4 as glm53-flash-nvfp4-lab-0rand (variant in tree): KV auto-size lands at 1,167,586 tokens — same capacity as our pinned 10 GiB, so the pin is unnecessary; smoke exact. Bench (--seed 42 --hardmode --perf --concurrency 1,2,4): quality 90 (158/176), tg 27.1/34.6/37.2 c1/c2/c4, TTFT d0 1.3 s, responsiveness 25 (6.2 s median), deployability 70. vs our lab baseline (92 = 162/176; c1 28.4 / c2 40.5; resp 21, deploy 72): quality −2, c2 decode −15%, c1 ≈, but c4 decode 37.2 in an arm the baseline never ran, and responsiveness +4. On OUR stack the 0rand combo is not a strict win; keep the baseline as the daily driver, keep the variant in tree for concurrency-hungry serving. Possible confounder: their run posts a single 14.8 s median turn at responsiveness 8 — an earlier bench build. Report: /tmp/runs/2026/08/2026-08-30T19-33-22.444571Z_4616f476.md.

MNBT 1024-vs-2048 assertion on the lab quant (2026-08-30/31 night)

Forum #124 ("1024 = quality-first") asserted. Our c1,2,4 run with MNBT 1024 + seqs 16 + GMU 0.89 + auto-KV (pool 1,051,792): quality 88, denominator 146 (harness dropped ~30 pts of scenarios), tg 27.6/27.0/30.4, TTFT d0 8.6 s, resp 24 (6.4 s). vs the same-protocol MNBT-2048 run one hour earlier: quality 90 (158/176, full denominator), tg 27.1/34.6/37.2, resp 25 (6.2 s). Verdict: at our protocol (reasoning_effort-low, temp 0, seed 42) 2048 wins on both quality percentage and decode; the 1024 advantage was measured under their thinking-max + temp-0.1 + trials-2 protocol. 0rand's #123 already noted "2048 batch + MTP4 ... loses quality (89 vs 93)" for THEIR protocol — different regime, same lesson: batch tuning is protocol-dependent. Variant stays in tree; record both. Reports: 2048 = ...19-33-22Z_4616f476.md, 1024 = ...22-04-48Z_4a70e6be.md.

DeepSeek-V4-Flash-Vision-Exp lane — campaign state (2026-08-31 night)

Day-0 port landed (sfxnz plugin 7a958128b8">@7a958128b8 vendored under docker-images/dsv4-flash-vision/, bases digest-pinned; sm121 build works, 35-40 min roundtrip). Lane verified TEXT-serving on ds4 (+NVMe/NAS weights):

  • eugr b12x base: 1024-token gen in 29.4 s = 34.9 tok/s with DSpark-6 (accept 0.714, length 2.55), text math exact, plugin loads ViT+aligner weights (whole checkpoint present, 82.28 GiB/node).
  • fixing their gaps so far: plugin init.py must ship the general- plugins register (raw.githubusercontent flaked on re-fetch — fixed via the contents API); wrapper class needs supports_multimodal = True (OpenAI layer refuses image_url otherwise — interfaces.supports_multimodal is a getattr check).
  • OPEN: OpenAI-image path raises "Model class ... has no registered multimodal processor" — the day-0 plugin never registers a _processor_factory (their shipped tests exercise the loader directly, not the API). Two paths: implement the BaseMultiModalProcessor inside our plugin (grid-tokens-per-image machinery exists in dsv4_vision.image_processor; fork interface = MultimodalRegistry. register_processor factory in multimodal/registry.py), or wait for the upstream repo to ship it. aiden-fork A/B build launched.

dsv4vision MM-into-OpenAI campaign — 2026-09-01 checkpoint

Built the full OpenAI multimodal stack in our plugin (processor trio, grid/sentinel tokens, embed_multimodal + embed_input_ids merge; verified in -image: embedding rows byte-exact vs aligner(ViT(patches))[perm]) and got the engine through profiling+warmup with mm registered. Two hard walls remain (before the first image_url token):

  1. The fork's V2 "always-embed" mm arm passes input_batch.input_ids[:n] — a wrong-rank slice that arrives as [B, T]-shaped 2-D ids (observed (17,17) for a 17-token prompt, (7,7) on DSpark-6 verify steps) — and downstream logits degenerate (all-<｜begin▁of▁sentence｜>).
  2. The AOT/dynamo trace of inputs_embeds freezes the dummy's 2-D shape and then asserts "wrong number of dimensions3 for op: input" on the native 3-D stream (tilelang mhc kernels are rank-strict both ways).

State: the whole mm branch is env-gated (VLLM_DSV4_MM=1), default OFF = the verified text-only lane — restored EXACT on the new per-node NVMe weights (~/dsv4vision-hf on dgx3+dgx4, 167 GiB each, recipe HF_HOME flipped from the NAS hub; shard load 7 min → <1 min, boot ~6 min end-to-end). Bridges done tonight for the next attempt: encoder-runner + speculator buffer shape-follow patches (patch_mm_hc_streams.py), placeholder `<｜deepseek_image｜>= in-vocab id 129264,get_placeholder_str` chat integration, AOT=0 knob for mm experiments. nid: fork A/B (aiden base) still pending its pip-layout patch; dspark_gate patch now auto-finds the fork's venv root.

dsv4vision tool-eval-bench (c1, 2026-09-01)

tool-eval-bench run --base-url http://dgx3.lan:30000/v1 --model deepseek-v4-flash-vision --seed 42 --hardmode --perf --concurrency 1 → report /private/tmp/runs/2026/09/2026-09-01T17-46-28.764372Z_89b18250.md:

recipe quality points decode tg t/s c1 (d0/d4096/d8192) pp2048 t/s TTFT deployability
dsv4-flash-vision (text lane, local-NVMe, DSpark-6) 81 143/176 — — — —

perf block: tg 34.1 / 31.0 / 29.1, pp2048 3,907 / 2,714 / 2,449, TTFT 1.9-4.9 s (spec), scenario-median turn 2.4 s, deployability 74 (best of any ds4 lane) — long-context capability is the raw 1M-context draw. 3 safety-critical failures (TC-42 extra-param, TC-43 empty required param, TC-60 sleeper injection) — the day-0 0731-style tool formatting is the gap vs the GLM lanes (thinking-pair JSON fencing was already a known day-0 weakness). Quality 81 < sparkrun 89 < lab 92.

New 2026-09-01 checkpoints: Intel W4A16-AutoRound + lab NVFP4-Spark

Checkpoints on the NAS hub (hf download'd):

  • Intel/GLM-5.3-Flash-W4A16-AutoRound rev 5eee1846f0 — AutoRound 0.15 W4A16 sym int4 g168, 99.84% of BF16 (GSM8K .9712/MMLU .8620), selfattn/hc/indexer/embed lm_head FP16 by design (their ignore_layers), pure GPTQ packing (qweight/qzeros/scales — qweight count 37152).
  • local-inference-lab/GLM-5.3-Flash-NVFP4-Spark rev 06972cdee7 — NVFP4 W4A4 whole network (in_proj_qkvbfg_a.weight_scale etc.) + NVFP4 MTP expert layer in separate model-mtp-0000x shards (lil.yaml: mtp_tokens 5, dflash2_tokens 7, default_speculator mtp, block 256, disable flashinfer autotune, INSTANTTENSOR env sizes; arch Glm5NextForConditionalGeneration + vision blocks in the index). This is the rtx6kpro r12-stack checkpoint (their doc: MTP:3 254 tok/s CC1 @ TP4, Sieve 333-403).

Recipes added: recipes/glm53-flash-intel-w4a16 (timothystewart6/vllm-gb10 v0.28.0-gb10.2 arm64, gptq_marlin, entrypoint declared — tini AssertionError taught us the image has ENTRYPOINT []+CMD bash), recipes/glm53-flash-nvfp4-spark (lab image). Boot outcomes:

  • Intel lane died in the fork's Transformers-backend weight loader: "no module or parameter named 'layers.0.self_attn.q_conv1d'" — the tjstewart image has NO native glm5_next arch (vllm archs list lacks it; transformers 5.16.1 has Glm5NextForConditionalGeneration) → needs the local-inference-lab fork (native glm5next) or a transformers name-shim mod.
  • NVFP4-Spark on our lab fork: KeyError on in_proj_qkvbfg_a.weight_scale at 80% shard load — whole-network NVFP4 W4A4 is beyond the R8-era lab fork loader (r12 stack territory).

⇒ THE unifying next step (user directive): build our own arm64/sm_121a image of local-inference-lab's fork — source contract from models/glm-5.3-flash.md (local-inference-lab/vllm dev/jovian-judgement + b12x master + 6/5 PRs; kk3 build rules apply: CUDA 13.0.88, sm_121f gencode rule for block-scaled FP4). RTX docs archive: /tmp/lil-docs (ds4-flash-v6, ds4dspark-v20-r33, infernal r21, pcie-autotune) for the DeepSeek quality track.

2026-09-02: r12 fork built + NVFP4-Spark serving on it (the day's win)

Built glm53-jovian-r12:local in-docker on dgx3 (docker-images/glm53-jovian-r12/): vllm 0.1.dev20489+gcc161e220 (local-inference-lab/vllm base 54f6e9826c + PR heads 550/552/553/554/555/557 merged in order — the doc's "composition commit" 7d8a09c is not a public ref) + b12x 1.3.0 (master 139e0405 + PR heads 260-270), on the day-0 vllm/vllm-openai:glm53-flash-arm64-cu130@905c02… base. Build fixes applied along the way: apt git, setuptools-rust, ninja/cmake, cu13 wheel include-header symlinks (cusparse.h), unversioned libnvrtc.so symlink for the deepgemm_C link step.

NVFP4-Spark boot on it (recipe glm53-flash-nvfp4-spark, retargeted):

  • whole-network NVFP4 W4A4 (incl. attention in_proj_qkvbfg scales) now LOADS — the r12 fork's modelopt handles it (our lab fork: KeyError).
  • Attention wall solved: the sm12x auto-priority list only offers [TRITON_MLA (no sparse), FLASHINFER_MLA_SPARSE_SM120 (head 576 only)] — neither fits GLM head-512. Fix = --attention-backend B12X (model hook in glm5next/nvidia/attention.py: _select_sparse_backend) for the main model, and "attention_backend":"B12X" INSIDE --speculative-config for the MTP draft (eagle/utils.py: _make_eagle_draft_vllm_config takes the draft's backend from the speculative config, not the global flag — user's hint).
  • Served: exact "391" verified, MTP-3 Glm5NextMTPModel resolved, KV fp8_ds_mla default, 2-node TP2. c1 bench running.
  • Intel W4A16 recipe retargeted to the same image (gptq_marlin + B12X attn).
  • patch_cuda_sm12x.py (B12X into sm12x priorities) staged in the image dir for the next rebuild — the image-baked version of the same fix.

2026-09-02 (cont): bench + clean rebuild per operator direction

  • NVFP4-Spark (r12 fork image, day-0 base) c1 bench: 88/100 (120/136, default 69-scenario suite; deployability 67, median turn 8.1s — TP2 GB10 is slower per-turn than the eugr vision lane's 2.2s on the same suite, also 88). Same-suite A/B vs vavrusa lane: 88 = 88. The lab's 92 was the 88-scenario/176-point suite — that invocation's re-run is pending.
  • Operator: docker_image_dir in the spark recipe pointed at the lab dir — fixed to glm53-jovian-r12 (intel recipe same).
  • Operator: "start with cuda image, solar-open2 pattern but r12-specific". Dockerfile/versions.env rewritten: raw nvidia/cuda:13.2.0-devel-ubuntu24.04 (a5b6256e…, solar-pinned) → apt stage (solar locks) → torch 2.13.0 cu130 + flashinfer 0.6.17 (whl index) → vllm wheel (fork base 54f6e9826c + PR-head merges 550-557, use_existing_torch, ccache) → b12x wheel (master + PR heads 260-270, pure-python) → runner installs both wheels + deps, bakes patch_cuda_sm12x.py, GB10 gate (vllm/b12x/_flashkda_C imports, B12X enum, patch presence). No day-0 vllm contamination; ENTRYPOINT [] CMD bash. Build running on dgx3 (tmux r12build → /tmp/r12build2.log), same tag glm53-jovian-r12:local (running serve keeps the old image; re-verify lane after build).

2026-09-02 (cont 2): lil — the lab's own launcher, read

local-inference-lab/lil = standalone Go launcher (their sm121 equivalent): reads each HF model repo's lil.yaml at runtime + embedded family policy + discovered topology (local or spark-rdma via SSH), renders/validates/runs a typed vLLM invocation. lil render <model> prints the exact env + command.

GLM family policy (configs/models/_bases.yaml) — the lab's canonical launch for these checkpoints, vs our spark recipe today:

  • --attention-backend B12X, --linear-backend b12x, --moe-backend b12x (we have only the global attention flag; linear/moe backends MISSING)
  • spec config: {"method":"mtp","num_speculative_tokens":5 (lil.yaml mtp_tokens), "moe_backend":, "attention_backend":"B12X"} — we use k=3 and no moe_backend. MTP:3 is the r12 x86-qualified mode; the checkpoint's manifest default is 5.
  • --load-format instanttensor (+INSTANTTENSOR_BUFFER_SIZE/CHUNK_SIZE env)
  • --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096, --max-model-len auto
  • --reasoning-parser glm45, --tool-call-parser glm47
  • DFlash2 (the GLM-side second speculator, method "dflash") needs the separate draft checkpoint local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8.

    • TODO: after the clean image lands, align the spark recipe to the lil policy (moe/linear backend b12x, spec moe_backend, instanttensor, mamba-cache-mode, parsers) and A/B k=3 vs k=5.

    Clean r12 build: fix log (builds 2-9)

    1. flashinfer: whl-index path dead → flashinfer-python 0.6.17 from PyPI (fork's setup.py skips flashinfer-cubin at build; runtime fetches cubins).
    2. git protocol v2 flake in-container → GIT_PROTOCOL=0 + protocol.version 0.
    3. setuptools: fork build needs setuptools-rust + setuptools-scm (uv hint).
    4. DeepGEMM FetchContent: pinned SHA on non-default nv_dev branch → GitHub 404s raw-SHA fetch → DEEPGEMM_SRC_DIR pre-clone stage (cmake escape hatch).
    5. PR 550 closed (unmerged) → refs/pull/550/head deleted; all PR heads now pinned by commit SHA (vllm 550-557, b12x 260-270). Raw-SHA fetches verified working.
    6. Operator rule: no build while a serve container runs (OOM risk on 128 GB unified) — spark lane stopped for the build window.

    2026-09-02 (cont 3): clean r12 image SHIPPED

    Image glm53-jovian-r12:local rebuilt from raw nvidia/cuda:13.2.0-devel (solar pattern, r12-specific content): 16.1 GB (vs 37.6 GB day-0-based), gates green (vllm 0.1.dev20489+g06f922207, b12x, _flashkda_C, B12X enum, sm121 patch), synced dgx3→dgx4. Boot fixes found on first clean boot:

    • tini "[FATAL] exec local-inference-lab/… failed" → clean image is ENTRYPOINT [] (solar contract), so the RECIPE now declares entrypoint = "vllm","serve".
    • FlashInfer JIT OOM: 12 parallel nvcc jobs (mxfp8_gemm_cutlass_sm120 etc.) get OOM-killed (code 137) while 93 GB weights + 10 GB KV are resident on 121 GB unified → MAX_JOBS=2 in recipe [env]; JIT (~15 min) then survives. Boot total ~35 min (NFS shards 16 min + JIT 15 min). c1 bench (default 69-scenario suite, same as the vavrusa run): 90/100 (124/138; quality 90, deployability 69, median 7.9 s) — vs 88 on the day-0-based image, same suite. Clean image is now the reference GLM lane.

    Remaining lab-alignment (next boot, one change-set at a time): --linear-backend b12x + --moe-backend b12x, spec moe_backend:b12x, A/B k=3 vs k=5 (lil default), --load-format instanttensor (+env), --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096, --reasoning-parser glm45, --tool-call-parser glm47. Then: Intel W4A16 boot on the clean image (queued, needs cluster).

    2026-09-02 (cont 4): lab-stack perf push — eugr build pipeline adopted

    Goal (user): 30 tok/s in --perf. Found the missing pieces:

    1. lil render reference (built lil from local-inference-lab/lil @ cb58d54, rendered with testdata manifest GLM-5.3-Flash-NVFP4-Spark + spark.yaml): our recipe flags now 100% match the family policy (_bases.yaml glm: b12x linear/moe, B12X attention, instanttensor, block 256, seqs 8, batched 4096, glm45/glm47, mamba align) + spec {mtp:5, moe_backend b12x, B12X} + env VLLM_USE_AOT_COMPILE=1 / V2_MODEL_RUNNER=1 / FLASHINFER_SAMPLER=1 / BREAKABLE_CUDAGRAPH=0. Boot 35m -> ~7m (instanttensor loader, user: keep). Perf after engine switch: d0 c1 24.2 / c4 31.8 tok/s (was 10.3/16.5). Quality dropped with full lil set (16/22 at TC-22) -> bisection in progress; user: instanttensor is fine, problem elsewhere.
    2. eugr/spark-vllm-docker (the lab's actual build repo, main): 804-line Dockerfile, CUDA 13.0.2 base, 12 build-time vllm patches (docker/patchvllm*.py), b12x from source (lukealonso/b12x; lab: local-inference-lab/b12x master), nightly DockerHub images eugr/spark-vllm[-b12x]:latest (stale 08-23, user: dockerhub is the wrong source — build from source). Ported all 12 patches into docker-images/glm53-jovian-r12/patches-eugr/
      • Dockerfile (VLLM_PRESERVE_SM12X_TARGET=1, VLLM_PATCH_B12X_C128A_ALIGNMENT=1; patches are idempotent, several no-op on our r12 ref). Build blocker: node github.com access flaky ("could not read Username" on FetchContent clones) -> pre-fetched all 9 sub-deps on the Mac into deps-src.tar.gz (triton_kernels@v3.5.1, MSA, FlashMLA, FlashKDA, flash-attention, qutlass@e74319e, tml-fa4@b206834, cutlass@v4.4.2) + the SRC_DIR env escapes each cmake documents. Build 14 compiling.

    r17 (20260902-r17 card) adoption + quality bisect (2026-09-03)

    Latest lab card: https://github.com/local-inference-lab/rtx6kpro/blob/master/models/glm-5.3-flash.md (names the qualified serving contract per artifact and SHA; TP4 on RTX PRO, not Spark — the Spark lil.yaml testdata render remains our TP2 adapter).

    1. Public r17 composition built from source: /tmp/vllmr12 = local-inference-lab/vllm dev/jovian-judgement@9c4dd054

      • open PRs 579/580/581/582/586 (branch supersedes the r12 PR set 550–557 — same symbols, evolved, see below) + staged-cutlass cmake patch; /tmp/b12xb17 = local-inference-lab/b12x master@2bbab479 + PRs 290/284. Pre-staged as vllm-src.tar.gz / b12x-src.tar.gz on the build node (WAF blocks node→github git). DO NOT re-merge 550-557: creates conflict-marker SyntaxErrors (pr550's pooled_indexer.py conflicts; branch tip already contains the evolved implementation).

    2. Important correction to the 2026-08-28 eager verdict: on the public r12/r17 refs, graphs+MTP is no longer a quality bug — the lab's private "packed GLM-5.3 package" (card SHAs 47f86087/a02841bc, not public) was the presumed fix carrier, but our public-tree composition measured:

      • r17 + graphs + 90/100-flags (no b12x/align): 88/100 hardmode (155/176) == lab-quant's own day-0 reference 156/176 (r17graphs01/…perf)
      • r17 + eager same-flags: 67/100, median turn 11.4 s, 2 scenarios timed out and excluded (r12eager01) — eager is now the WORST mode. So: graphs stay ON; enforce-eager is gone.

    3. Execution-env knobs are quality-clean: GMU 0.90 (0.95 boot-deaths on GB10 — "free memory 111.96 < desired"), max-num-seqs 16, OMP 2, VLLM_USE_STANDALONE_COMPILE/FLASHINFER_SAMPLER/BREAKABLE envs: 87/100 (r17knobsonly). AOT=1+V2 runner: NEUTRAL for quality (47 either way when backends were on) and NEUTRAL for the d0 grid — dropped (AOT=0, no V2).

    4. THE QUALITY TOXIN = the b12x backend-flag cluster (on the PUBLIC r17/r12 compositions, with our b12x 1.3.0/1.2.6 builds):

      • k=5 full cluster: 74/100 (r12 img), ~74 → killed-to-16/22 (r12 img)
      • k=5 full cluster (r17 img): ~70-ish band (r17cluster01, stopped early)
      • k=3 full cluster: 46/100 (r17clusterk3)
      • k=3 cluster minus AOT/V2: 47/100 (r17noaotk3)
      • k=3 cluster minus backends (knobs only): 87/100 (r17knobsonly) The lab's own stack runs these backends qualified — but their composition carries the private packed GLM-5.3 package (RTX-PRO target); the public branch tip alone does NOT make the b12x W4A4/linear/align paths quality-safe. Also: b12x cluster wrecked deep-context speed (d4096 c1 12.6 @k5) while helping d0 (33-34 c1) — nets negative anyway.

    5. SHIP ANCHOR (current recipe) = r17 image (94ac0882a8) + 90/100 args (attention B12X, block 256, seqs 16, batched 4096, LM-only, glm47/glm45 parsers, instanttensor load, k=3 spec {mtp,attn B12X} with moe_backend key) + graphs (no eager) + knobs env (GMU 0.90, seqs 16, OMP 2, AOT=0, STANDALONE=1, SAMPLER=1, BREAKABLE=0) + JIT-cache mounts: 87/100 hardmode (153/176; lab-quant ref 156/176), default-suite 88 class; perf d0 c1 24.1 / c2 30.2 / c4 44.7; d4096 c1 23.9; d8192 c1 23.8; pp ~1.6-1.8k. Boot 8 min (JIT warm) / ~25 min cold (nvcc re-JIT).

    6. Boot-time JIT: changing the rw flashinfer cache invalidates per-config kernel sets; a config's first boot silently nvcc-compiles (MAX_JOBS=2) ~15-25 min inside "No available shared memory" log-silence; later boots of the same config are fast. Expect this on every NEW kernel path.

    7. KV-dtype probes: fp8_ds_mla on r12-img = 75/100 default-suite (net zero, TC-45 flipped PASS but others flipped off); graphs here. r17 anchor keeps default dtype. Not a quality lever.

    8. Open perf gap: c1 24 vs manifest-cluster 33-34 (which is quality-dead). If the lab ever publishes the packed package (or a public branch commit makes b12x-backends clean), the 30+ c1 target lands with manifest flags. Alternative to probe later: one b12x backend flag at a time (a single toxin member may be free).

    One-backend bisect (2026-09-03) — SOLVED (hardmode, r17 img 94ac0882, k=3 graphs)

    • knobs-only: 87 (153/176); d0 c1 24.1, d4096 23.9, d8192 23.8 (r17knobsonly)
    • + --moe-backend b12x (spec default MTP experts): 88 (154/176); d0 23.5, d4096 22.5, d8192 26.6 (r17moeonly) — quality-clean, keep
    • + --linear-backend b12x: 44 (77/176), Parameter Precision 0%, TC-21 0/5 validations — THE toxin. B12X linear kernel path destroys numerics on the public composition (r17linearon)
    • + --mamba-cache-mode align: 88 (154/176) but d0 c1 18.6 (−4.9), no quality gain (r17alignon) — perf-negative, dropped
    • Full-cluster combos (44-74) were dominated by linear; spec's moe_backend b12x key and the knobs env family are all exonerated individually. SHIP = r17 img + 90/100 args + graphs + knobs env + --moe-backend b12x only (recipe @ 2d379f9): 88/100, d0 23.5 / c4 43.9, depth 22.5/26.6. The 30 c1 target is only reachable with linear-b12x (29.6 d0) which is quality-dead until the lab publishes the packed GLM-5.3 package.

    Checkpoint A/B on ds4 (2026-09-03, hardmode 176pt): Spark vs NVFP4 lab checkpoint

    User-flagged must-have: our lane served the Spark-variant checkpoint with quality losses vs lab references; A/B the checkpoint under matched stacks.

    Cell Stack Checkpoint Hardmode d0 c1 d4096 c1 d8192 c1 notes
    r17ablabimg lab img + lab recipe (eager, fp8_ds_mla, k3, default MoE, day-0 fork 0.1.dev20051+g487ecf187) NVFP4 93 (164/176) ★★★★★ 24.4 26.5 22.8 78 pass / 2 fail; TC-45 PASS; matches lab's own 92 ref
    r17ablabck shipped r17 img + ship flags (graphs, knobs, k3, default MoE — moe-b12x unsupported on this ckpt: layer-45 is MXFP8) NVFP4 88 (154/176) 27.5 29.6 26.6 46.4/46.7 c2/c4 at d0
    r17moeonly (ship) shipped r17 img Spark 88 (154/176) 23.5 22.5 26.6 —

    Verdict:

    1. Checkpoint is not the quality factor on the r17 stack (88 = 88 both checkpoints) — but the NVFP4 lab checkpoint is significantly FASTER (+4 tok/s d0, +7 d4096) and lighter rules (no --moe-backend b12x).
    2. The lab day-0 stack retains +5 quality (93 vs 88) regardless of checkpoint — its eager + fp8_ds_mla + default-everything runtime is what the lab's own 92-hardmode reference was measured on. Enforce-eager costs prefill pp at c2/c4 (0.9-1.1k vs 1.5-1.6k) but TG is equal-or-better.
    3. Recipes: glm53-flash-nvfp4-r17abi (ship flags + NVFP4, moe flag removed) is the fastest quality-parity lane; glm53-flash-nvfp4-lab remains the quality champion (93).

    Night-shift queue (2026-09-05)

    Prepared lanes + builds queued for the night (cluster is serving the 4x aiden lane during the day — stop it first, then all 4 nodes are free). Order = value/duration; each item lists prereqs + expected verdict.

    1. r18 fork build (docker-images/glm53-jovian-r12, ~1-2 h build). Stage on the build node: /tmp/vllm-src.tar.gz (94 MB) + /tmp/b12x-src.tar.gz (29 MB) — pins per versions.env (vllm a8c796f3+pr582+pr646 → 343661fe; b12x b58f34ea+pr284 → 1adb8a38), then TAG=glm53-jovian-r12:local bash build.sh. Test campaign: (a) Re-baseline the quality anchor first — DONE 2026-09-06: ran glm53-flash-nvfp4-r17abi (2x, r17 image) on 2nodes-dac with the current tool: NEW BASELINE = 153/176 (87/100, deployability 69) — report in recipes/glm53-flash-nvfp4-r17abi/reports/2026-09-06_r18-rebaseline-r17abi-2x.md (2 pts below the historical 155/176 anchor: DAC fabric + NAS weights; scoring confirmed current-tool-compatible: 88 scenarios × 2 pts = 176). Gate ALL new lanes against 153/176 (≥151 = parity, <151 = regression); (b) **re-bisect `--linear-backend b12x`** — new b12x A16 precision selection may clear the 44/100 toxin (29.6 c1 unlock if clean), (c) RoCE one-shot on the 4x lane (VLLM_ENABLE_ROCE_ALLREDUCE=1; expect −5–19% decode per-step per their #597 numbers), (d) `--kv-cache-dtype nvfp4_ds_mla` probe (new B12X KV path), (e) #646 geometry check: 2048/256/256 split blocks selected + repeat-request prefix resume at the fine boundary (their regression scenario: retention 0, states 32/16, resume at 32) — if broken, drop the 646 pin from versions.env and re-cut (rest of r18 is independent). 1b. **DCP4 probe** (4x lane, no build dependency — run on the current r17abi image first, re-run on r18 if positive). Add `--decode-context-parallel-size 4` (stock vLLM flag, verified in the r18 tree: vllm/config/parallel.py; DCP>1 requires MLA — GLM53 qualifies). Reference (sparkring MTP3 mesh profile, 4 Sparks, DCP4 + 24 GiB fp8 KV/rank): ~4.32M logical KV tokens (vs our ~1.25M at 12 GiB DCP1), 231.3 aggregate decode tok/s peak, prefill ~2.7k flat 8K→128K. Verdict: KV-token capacity per node at our pin, decode C1/C8/C16 vs baseline, prefill 8K/64K. Their fabric is a direct HW-forwarded mesh (cheaper all-reduce than our switch) — measure, do not extrapolate.
    2. RedHatAI MTP lane (2x, ~30 min). Re-run huggingface-cli download RedHatAI/GLM-5.3-Flash-NVFP4 on both nodes (model_mtp.safetensors 7.6 GB is new since the initial pulls; HF_HUB_OFFLINE=1). Serve glm53-flash-nvfp4-redhat (MTP k=5 per vendor card) → compare vs the 90/100 no-MTP baseline + decode speed; memory watch +7.6 GB/node (KV pin 10→8 GiB if boot dies; k=3 fallback).
    3. eugr b12x image verify + GLM lane (2x, ~1 h). Pull eugr/spark-vllm-b12x:nightly-20260904 (10.9 GB) on the 2 nodes, read /workspace/build-metadata.yaml + /workspace/b12x-source-commit (record exact pin in the recipe header). huggingface-cli download local-inference-lab/GLM-5.3-Flash-NVFP4-Spark on both nodes. Serve glm53-flash-nvfp4-eugr → A/B vs r18: full b12x cluster quality (linear-b12x toxin retest on their stack) + RoCE 2x decode numbers. 4-node cell: glm53-flash-nvfp4-eugr-4x (needs all 4 nodes free) vs the 2x lane — 2x stays the production default unless 4x is dramatic. 3b. DFlash2 spec A/B (same eugr image, after 3 boots). Swap the spec to "method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2", "num_speculative_tokens":7,"draft_tensor_parallel_size":2, "kv_cache_dtype":"auto" (separate drafter repo — huggingface-cli download on both nodes first; exists on HF, single safetensors). VS the recipe's MTP k=5: decode at C1/C4 + quality spot-check (forum numbers are perf-only: ~60 c1 / ~73 c4 tok/s on 4x — their 4x/TP4 config, so 2x numbers will differ).
    4. DSV4F-Vision eugr lane (2x, prefer dgx3/dgx4 — weight reuse). Reuse the per-node ~/dsv4vision-hf copies (verify the HF_HOME mapping vs the recipe's default HF cache) or re-download Vision-Exp (168 GB). Serve dsv4-flash-vision-eugr → compare vs the fork lane (dsv4-flash-vision, DSpark-6): quality (text + vision smoke) + decode
      • TTFT; spec A/B: k=3 adaptive (vendor card) vs k=6 (fork lane's proven value).

    Night-shift results (2026-09-05/06, all seed 42, hardmode, c1,2)

    Re-baseline (queue 1a): r17abi 2x/2nodes-dac = 153/176 (87/100, dep 69) with tool-eval-bench 2.6.1.dev45+gcf54b4bfe (88 scenarios). This is the new canonical anchor — all lanes below gate against it. Report: recipes/glm53-flash-nvfp4-r17abi/reports/.

    r18 build (5 attempts on dgx4, log r18build*.log): three root causes — (1) transient snapshot.ubuntu.com 503s; (2) patch_vllm_spark_kv_cache_cleanup.py anchor gone (upstream replaced the profiling flow with MemorySnapshot/request_memory which already cleans before the snapshot — patch now detects the new flow and exits 0); (3) staged-deps submodule stubs: both flash-attention/csrc/cutlass AND MSA/python/fmha_sm100/cutlass were empty dirs in deps-src.tar.gz (the r18 tree dropped implicit cutlass include wiring / MSA's install needs the dir) — populated with each project's own pin (FA: nvidia/cutlass 62750a2b, MSA: nvidia/cutlass eb61c911) and purged 23k macOS AppleDouble ._* files. Image: tag glm53-jovian-r12:local, tree 67e7e6e (base a8c796f3 + #582 + #646), 16.1 GB, on all 4 nodes.

    Lane results (gate = 153/176 baseline; quality is the gate, dep = deployability):

    Lane /176 /100 dep KV tokens verdict
    r17abi 2x (baseline) 153 87 69 ~1.25M anchor
    r18 2x (ds4) 157 89 71 — PASS (+4)
    RedHat MTP k=5 2x 156 89 70 — PASS (MTP works, +3)
    eugr b12x MTP k=5 2x 143 81 65 — FAIL (−10)
    r18 4x RoCE TP4 156 89 76 1.47M PASS
    r18 4x DCP4 (TP4+--decode-context-parallel-size 4+--cp-kv-cache-interleave-size 4) 161 91 75 4.84M BEST
    r18 4x nvfp4_ds_mla KV 161 91 77 2.35M PASS (best dep)

    --perf results (llama-benchy, pp2048/tg128, runs=3, from each report's Throughput Metrics table; c1 = single, c2 = 2 concurrent; t/s = tokens/s):

    Lane (img) pp c1 tg c1 TTFT c1 pp c2 tg c2 TTFT c2 tg c2 @8k-ctx
    r17abi 2x (baseline, r17) 1,577 29.0 1,362ms 1,595 41.7 2,093ms 29.7
    r18 2x (r18) 1,473 30.7 1,481ms 1,374 40.2 2,268ms 30.6
    RedHat MTP k=5 2x (lab) 1,340 26.8 1,638ms 460 22.4 4,667ms 14.7
    eugr MTP k=5 2x (eugr) 1,292 31.1 2,397ms 1,388 41.6 2,751ms 17.7
    r18 4x RoCE (r18) 2,904 47.7 762ms 2,413 70.9 1,399ms 35.9
    r18 4x DCP4 (r18) 1,639 43.1 1,521ms 1,756 51.1 1,637ms 30.0
    r18 4x nvfp4 KV (r18) 2,614 45.3 873ms 2,250 62.6 1,368ms 35.6

    Perf readings:

    • 4x RoCE is the throughput king: 2× prefill of 2x (2.9k vs 1.6k), 1.7× decode (70.9 c2 vs 41.7), half the TTFT (762ms). 4x wins on both axes vs 2x — the case for the 4x production cell.
    • DCP4 pays a prefill tax: 1.6k pp (vs 2.9k RoCE) — the DCP ring exchange costs prefill; decode is comparable (43–51). Its edge is the 4.84M KV capacity + quality, not raw throughput.
    • nvfp4 KV ≈ RoCE (2.6k pp, 62.6 c2): the KV dtype costs nothing measurable — combine candidates: DCP4 + nvfp4 KV.
    • RedHat MTP k=5 collapses at c2: pp 460 t/s (vs ~1.4–1.6k everywhere else), TTFT 4.7s — quality is fine (156) but the lab image's MTP-k=5 path is perf-weak under concurrency. r18 2x dominates it on both axes.
    • eugr 2x: c1/c2 short-ctx throughput matches the baseline, but decode degrades at depth (15.8–17.7 t/s @4k/8k-ctx vs 28–30) — consistent with the quality failure (81/100).

    Key findings:

    • DCP4 works on our switch fabric (sparkring steal confirmed): 4.84M KV tokens at the SAME 10 GiB/node pin — 3.3× plain 4x, and beats their 4.32M @ 24 GiB/rank (their HW mesh gave them 231 t/s aggregate; our RoCE switch numbers are in the report). Best quality of the night (91). B12X is NOT DCP-compatible (engine auto-falls the DCP cell to FLASH_ATTN attention; that's what benched).
    • nvfp4_ds_mla KV dtype (r18-tree value, vllm/config/cache.py): 1.59× the fp8 capacity at the same pin, tied quality, best deployability. Combine with DCP4 next (projected ~7.7M KV).
    • #646 geometry live (queue 1e): serve logs show "Setting attention block size to 2304" = the 2048+256 split-block geometry; no retention/states anomalies seen, no fallback needed.
    • eugr image lane fails the quality gate (143 vs 153) on 2x — the linear-b12x toxin persists on their full b12x cluster; eugr-4x cell skipped (2x already decides it).

    Blocked (documented, not solvable from recipes):

    • DFlash2 drafter lane (queue 3b): the drafter inherits the engine's FP8 KV dtype; on SM121 B12X is rejected for it (kv_cache_dtype constraint) and FLASH_ATTN too ("FP8 KV cache requires FA3 on SM90 or FA4 on SM100"). With drafter kv_cache_dtype: "auto" it fails the FORK's own assertion: ValueError: GLM MLA parent-page stride must be an exact number of C4 pages. Fork-side fix needed (drafter KV geometry under C4 paging) before the k=7 A/B can run.
    • dsv4-flash-vision-eugr (queue 4): the eugr 0904 image's DeepseekV4ForCausalLM (vllm/models/deepseek_v4/nvidia/model.py) contains NO aligner module — the image predates V4-Vision support; the Vision-Exp checkpoint's vision weights have nowhere to land. Re-check on a newer eugr image; fork lane (dsv4-flash-vision) remains the only V4-Vision path.

    Open decisions: (1) production 4x candidate = r18 + DCP4 (quality 91, 4.84M KV) — needs a production recipe + operator sign-off; (2) r18 2x adoption as the new 2x default (157 vs 153 baseline); (3) DCP4+nvfp4 KV combined cell; (4) eugr lanes stay demoted.

    r12/r18 image separation (2026-09-06, operator direction)

    The glm53-jovian-r12:local tag pointing at r18 content was wrong — separate now, no mixing:

    • docker-images/glm53-jovian-r18/ — self-contained r18 build dir (r18 versions.env pins 582+646, fixed kv-cache-cleanup patch, re-staged deps-src with FA+MSA cutlass, source tarballs vllm 67e7e6e / b12x b58f34ea). Image tag: glm53-jovian-r18:local (renamed on all 4 nodes from the r12 tag; content unchanged, built during the night shift).
    • docker-images/glm53-jovian-r12/ — restored via git checkout 582343c to the genuine r17 state (pins 550–557, original kv-cache-cleanup patch). Image tag glm53-jovian-r12:local currently points at NOTHING (removed from nodes) — an r17 rebuild is possible but NOT done: the r17-era deps-src staging is unrecoverable (untracked + overwritten), so a rebuild today would use the r18 deps staging (FA/MSA cutlass populated) and be a subtly different image than the original r17 anchor. If the r17 A/B anchor is ever needed again, re-cut the r17-era deps first.
    • Recipes: glm53-flash-nvfp4-r18{,-4x,-4x-dcp4,-4x-nvfp4kv,-dflash2} (new) → r18 tag/dir; glm53-flash-nvfp4-r17abi unchanged → r12 tag/dir.
    • Source tarballs (vllm-src/b12x-src) are local build artifacts — now gitignored; deps-src already was.

    2x nvfp4-KV + Spark checkpoint extension (2026-09-06, continued)

    Operator direction: nvfp4_ds_mla in the 2x setup (parity with the 4x cell) and a Spark-checkpoint variant of each nvfp4kv cell; 4x KV pin 10 -> 16 GiB (NVFP4 4x has the headroom: 47.1 GiB weights/node vs ~108 usable).

    Lane (r18 img) /176 /100 dep KV @ pin verdict
    2x nvfp4 KV (non-Spark) 154 88 70 2.14M @ 10G (1.7x fp8) PASS, no quality regression
    2x nvfp4 KV + Spark ckpt 158 90 70 2.14M @ 10G BEST 2x of the campaign
    4x nvfp4 KV + Spark ckpt — — — (16G) BLOCKED: mm_mxfp8 TP4
    • The Spark checkpoint (DSpark-embedded quant) beats the non-Spark lab quant by +1/176 at 2x (158 vs 157 r18 2x) and by +5 over the anchor — candidate for the 2x production default alongside r18.
    • 4x + Spark is blocked: ValueError: Problem size is not supported for mm_mxfp8 — the Spark quant's MXFP8 MTP-expert layer does not tile at TP4 (non-Spark is fine at 4x). Spark checkpoint => 2x lane only.
    • 16 GiB 4x pins: recipes bumped; expected KV: RoCE 2.35M, DCP4 7.7M, nvfp4 3.76M. No re-bench needed (capacity-only change vs the 10G benches).
    • JIT persistence first measurement: cold serve ~45 min -> 9 min warm (per-node flashinfer/torchinductor/vllm caches; see next section).

    JIT cache persistence (2026-09-06, operator direction)

    Every fresh container re-pays ~20 min of flashinfer/torch JIT compilation (cold /root/.cache per container — seen on every lane all night). All glm53 lanes now persist the JIT caches per node (local FS, not NAS) via plain [[extra_mounts]]:

    • ~/sm121-cache/jit-121f/flashinfer → /root/.cache/flashinfer
    • ~/sm121-cache/jit-121f/torchinductor → /root/.cache/torchinductor
    • ~/sm121-cache/jit-121f/vllm → /root/.cache/vllm Host dirs are SHARED between lanes of the same arch — each cache self-namespaces by its engine-version key internally (flashinfer keys by pkg version + arch; torchinductor by torch version; vllm by compile input hash), so mixing r17/r18/lab/eugr in one dir is safe. The pre-existing glm53-jit-121f/b12x mount is untouched (keeps its accumulated cache). First serve of each image pays the compile once per node; later serves should start ~20 min faster.

    Ops layout from 2026-09-06 afternoon (operator direction)

    • 2nodes-dac (dgx1+dgx2) = PRODUCTION: serves the operator's own agent on glm53-flash-nvfp4-r18-2x-nvfp4kv-spark (158/176). Do NOT stop/re-serve/ benchmark against it without explicit operator confirmation — treat like a live cluster (AGENTS.md warning applies).
    • ds4 (dgx3+dgx4) = DEV lane: identical recipe serves here as the A/B reference for fork/tree improvement work ("improve our 2 node builds"). Next 4x cells (which need all 4 nodes) require stopping the production lane — operator sign-off first.

    Image build v2: clone-at-pin (2026-09-06, operator direction)

    Operator verdict: regenerating tarballs on some machine is nonsense (the sources existed only on that machine — a fresh repo clone could not build). The r27 build has NO context tarballs:

    • docker-images/glm53-jovian-r27/ — clone-at-pin build: the Dockerfile clones vLLM/b12x/LMCache at pinned commits (versions.env *_PIN = SHA, *_TREE = git tree hash, gates assert BOTH), builds sub-deps from the pins the tree itself declares (fetch-deps.sh, grep-gated against the tree's cmake), builds LMCache + cuMem interposer in-container, and keeps the proven GB10 runtime (CUDA 13.2 aarch64 / torch 2.13.0 / flashinfer 0.6.17 — the same versions the r27 tree's requirements/cuda.txt asks for).
    • Fresh-clone reproducibility: git clone this repo, upload the 215 KB context, TAG=... bash build.sh — everything else comes from GitHub at pinned SHAs (verified locally end to end; b12x/lmcache fetch-by-SHA works, branch-tip gates catch drift — LMCache's lock branch was already renamed upstream, so pins are SHAs, branches are comments only).
    • Source contract = rtx6kpro r27 source.lock: vLLM integration/glm53-r27-release-20260906 @ 63a82f8d (tree f54cd9ca), B12X @ e8ad299b (f3cd8a9e), LMCache @ 822330db (008ac3e0, 0.5.5.dev0+glm53cache.engine.822330db), FlashKDA @ 3b225bf2, cutlass v4.4.2 + FA gitlink 62750a2b + MSA gitlink eb61c911, FlashMLA a8f794d1, qutlass e74319e3, tml-fa4 b2068346, triton v3.5.1, DeepGEMM 8b1392b. Wheels get deterministic versions via SETUPTOOLS_SCM_PRETEND_VERSION.
    • Where r27 lands vs our r18: new in the tree are LMCache (engine-driven cache transfers), qualified DFlash2 support (dflash2 depth 7 via env), 2048-token GPU pages with public block 256 (the #646 geometry, now qualified), recurrent-boundary prefix reuse, prefill compute sharing (fairness engine), DCP full-cache prefill; eugr patch set retained (idempotent, skips patterns that moved upstream).
    • Legacy: r12/r18 build dirs are staging-era, not rebuildable (tarballs were context-only artifacts). Cleanup candidate once r27 qualifies.