Главная Обзор Помощь
Вход
sm121
/
launcher
2
0
Ответвить 0
Файлы Задачи 0 Запросы на слияние 0 Вики
Дерево: fe070bebdd
Ветки Метки
launcher
/
specs
/
tech-architecture
/
glm53-recipe-state.md

glm53-recipe-state.md 45 KB
История Исходник

GLM-5.3-Flash NVFP4 — three quant variants measured on 2× DGX Spark (ds4)

Measured 2026-08-27/28 on cluster ds4 (dgx3+dgx4, switched fabric), all three served by sm121 through the launcher, same benchmark invocation:

tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53 \
  --seed 42 --hardmode --perf --concurrency 1,2

Tool version pin (2026-09-05, verified against the installed tool): tool-eval-bench 2.6.1.dev45+gcf54b4bfe (uv tool; git SeraphimSerapis/tool-eval-bench @ cf54b4bfe). Current pool semantics: --hardmode = 69 standard (A–O) + 19 Hard Mode (P) = 88 scenarios (≈26 min, dry-run verified); --hardmode-only = 19 (≈6 min). The /176 scores above (155/176 anchor, lab ref 156/176) are pinned to the OLDER 176-point pool — scores do NOT transfer across tool versions. Every campaign now re-baselines the anchor lane with the CURRENT tool first (night queue 1a), then gates new lanes against that new baseline.

All three share served name glm53 — clients never changed between runs.

Results

recipe quant quality points decode tg t/s (c1) tg c2 TTFT deployability MTP
glm53-flash-nvfp4-lab local-inference-lab mixed NVFP4/MXFP8-MTP 92 (eager) / 89 (graphs) 162/176 28.4 40.5 5.6 s 72 ✅ k=3, accept len ~2.9–3.5, ~63–83 %
glm53-flash-nvfp4-redhat RedHatAI W4A4 compressed-tensors 90 159/176 13.9 13.8 2.2 s 67 ❌ (no MTP weights in checkpoint)
glm53-flash-nvfp4 LibertAIDAI NVFP4-A16 88 154/176 26.2 22.7 6.3 s 69 ✅ k=5
glm53-flash-nvfp4-4p67 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 87 153/176 24.4 39.2 1.4 s 67 ✅ accept ~90%
glm53-flash-aiden-4x aiden image production-1.1 (DFlash2 overlay), NVFP4, incoai DFlash2 k=7, eager, 16 GiB KV pin, 4× Spark TP4 (his production shape) 89 156/176 37.6 (d0 c1; flat to d8192, 2,185 pp t/s — matches his ~2,100) 50.7 (c2) / 63.5 (c4) 0.9 s 80 ✅ DFlash2 k=7
glm53-flash-aiden-2x aiden image production-1.1 (DFlash2 overlay), NVFP4, incoai DFlash2 k=7, eager, 4 GiB KV pin (2x profile) 89 157/176 22.3 (d0 c1; d8192 c4 degrades — 4 GiB pin) — — 66 ✅ DFlash2 k=7 (drafter block 8−1)
glm53-flash-sparkrun community sparkrun image (aidendle94), NVFP4, eager, NO MTP (image lacks the fix) 89 154/176 14.3 23.8 1.4 s 66 ❌ (n/a in image)
glm53-flash-exl3 brandonmusic EXL3/TR3 4bpw (Entrpi image + instanttensor), DFlash2-7, graphs 88 155/176 23.1 24.1 2.6 s d0 77 ✅ accept 48–81 %

(lab scored /174 — TC-53 was excluded from its run as an infrastructure failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s total — harness drops timed-out scenarios from the denominator; had it scored like redhat's 2/2, lab would be 157/176 = 89 — no change). Quality % is comparable. LibertAIDAI is the only variant whose MTP regresses under c2 (26.2→22.7) — lab's k=3 draft scales, LibertAIDAI's k=5 does not.

4x note (2026-09-04): the 4x aiden run had one safety-critical failure — TC-43 called web_search with an empty query (required-parameter violation); not seen in the 2x aiden run.)

Prefill (pp2048) 0.3–1.7 K t/s depending on depth/concurrency — noisy, all three in the same band (NAS load-influenced); decode is the stable discriminator.

Read of the numbers

  • lab (eager) wins outright: 92/100 best quality AND 2× decode (28–40 t/s vs ~14 baseline). Acceptance lengths matched upstream's 2.55–3.5 range.
  • redhat (90/100 with graphs) is a baseline-decode dead-end on this box: its checkpoint ships zero MTP tensors (verified index: 146,745 tensors, none under layer 45), so it is stuck at ~14 t/s — no longer the quality pick either.
  • lab --enforce-eager = day-driver; redhat = fallback; LibertAIDAI = proven first but k=5 MTP regresses at c2 (eager might lift it too — untested).

Images & patches

image dir base patches
glm53-flash-nvfp4 vllm/vllm-openai:glm53-flash-arm64-cu130 kingjones30 NoPE-MLA mod (4-part, anchor-asserted)
glm53-flash-nvfp4-lab same + 4 FujitsuPolycom patches (Apache-2.0, see patches/UPSTREAM.md): model.py naming shim, modelopt MTP namespace fix, persistent_topk CC-12.x guards ×2
redhat recipe reuses glm53-flash-nvfp4:local (mod is quant-independent)

Boot markers [quantprobe] layer=RoutedExperts prefix=model.layers.45.mlp.experts algo=MXFP8 confirm the MTP fix is live; algo=None there = broken load.

MTP + CUDA graphs quality bug (forum #381350/#44, CONFIRMED 2026-08-28)

0rand: “mtp or dflash plus cuda graphs seriously hit quality — try disabling cuda graph capture”. A/B with identical seed/flags on lab:

arm quality points tg c1 tg c2 TTFT
graphs (CUDA graphs ON) 89 155/174 28.4 43.4 3.9 s
--enforce-eager 92 162/176 28.4 40.5 5.6 s

+3 quality points for −7 % c2 decode — eager is now the lab default. Accept metrics were healthy in BOTH arms (71.6 %, len 3.15 eager), so this is not an acceptance collapse — graph capture corrupts sampled output silently. Corroborating: randomllama (DFlash2 recipe) already flagged “not claiming bit-identical greedy outputs” with graphs on. The b12x glm52 stack runs graphs + MTP k=4 FULL_AND_PIECEWISE benches clean (100/100) — build-specific bug in the glm53 day-0 dev build, do NOT propagate --enforce-eager to glm52. Untested follow-ups: LibertAIDAI + eager (k=5; expect ~88→9x), redhat is graph-neutral (no MTP, stays 90 with graphs).

Operational notes (measured)

redhat @ 524288 ctx (2026-08-28)

262144 booted on the base image; 512K needed three changes (each verified by a real boot that died without it):

  1. --kernel-config {"enable_flashinfer_autotune":false,"enable_cutedsl_warmup":false}
    • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — autotune scratch at 512K shapes OOMs the driver (NVRM NV_ERR_NO_MEMORY _memdescAllocInternal); the sparse-MLA kernels' non-autotuned fallback is fine (lab proved it).
  2. --max-num-batched-tokens 4096 + --block-size 256 (lab's memory shape).
  3. The lab image — the killer was launch_persistent_topk ... total_ctas=62 > num_sms*occupancy=48 ... fallback requires >=128KB smem (have 101376), i.e. exactly what the FujitsuPolycom sparse_attn_indexer(_kpool) CC-12.0 guards fix. Indexer CTAs scale with ctx: 262K fit 48 SMs, 512K needs 62. The patches are arch-level → safe for the compressed-tensors checkpoint (quantprobe is a no-op without modelopt; output verified correct).

Result: KV pool 1,334,265 tokens / 2.54× @ 524288 (10 GiB pin; lab gets 1,164,369 from the same pin — its MTP speculator reserves memory before KV sizing), ~17 min clean boot, correct greedy output.

Note: an orphaned rank-1 headless worker survives rank-0 death and keeps the node occupied — always docker rm -f label-filtered leftovers before a retry serve (or the launcher refuses with "only 1 nodes").

  • lab weight load: 92.19 GiB / 700 s sequential from the NAS HF cache; --kv-cache-memory-bytes 10737418240 → KV 1,164,369 tokens / 2.22× @ 512K (matches upstream to the digit — do NOT switch back to GMU sizing).
  • First launcher serve of lab died on both ranks at ~54 s (exit 1, logs lost to --rm); re-run was clean twice. Suspected transient (cold imports + simultaneous launch). If it recurs: re-launch without --rm and keep logs.
  • sm121 serve immediately after sm121 stop can fail with "cluster has only 1 nodes" — occupancy race, nodes not yet released; wait ~10 s and retry.
  • LibertAIDAI/redhat recipes still run FlashInfer autotune (60-min possible boot, VLLM_ENGINE_READY_TIMEOUT_S=3600 set); the lab recipe disables it via --kernel-config per upstream's proven boot.
  • Chat-template caveat: reasoning_effort:"low" still emits a reasoning trace (budget ≠ 0); max_tokens:48 can yield content: null, finish: length — use ≥256 for smoke tests.

Follow-ups

  • eugr B12X variant recipe glm53-flash-nvfp4-eugr (2026-09-05): port of eugr/spark-vllm-docker @ 94b3c30 recipes/glm-5.3-flash.yaml on eugr/spark-vllm-b12x:nightly-20260904 (their dev/jovian-judgement 09-04 tip

    • lukealonso/b12x master; no ENTRYPOINT in image → recipe carries ["vllm","serve"]). A/B cell vs the r18 fork build: full b12x cluster (linear-b12x = r17 toxin retest) + RoCE one-shot all-reduce + MTP k=5. Model: lab GLM-5.3-Flash-NVFP4-Spark (cache on both nodes first). Unmeasured — dry-run verified only.

  • Vendor instanttensor loader into the lab image (~30 s vs 700 s weight load).

  • Consider adding --kernel-config autotune-off to the LibertAIDAI recipe (boot minutes shorter; redhat now has it, see 512K section — verify before trusting on LibertAIDAI).

  • MTP for redhat: RESOLVED (2026-09-05) — no variant repo needed: the main RedHatAI/GLM-5.3-Flash-NVFP4 repo gained model_mtp.safetensors (7.6 GB) and the card an "Enable Speculative Decoding" section (k=5, plain --speculative-config {"method":"mtp",...} — same model path, no swap). Recipe glm53-flash-nvfp4-redhat updated to MTP k=5 (watch the +7.6 GB/node memory cost vs the 10 GiB KV pin; k=3 fallback). The lab's 5.2 pattern (-mtp-FP8 variant repos) did not apply; the cross-graft MXFP8-MTP idea is moot.

4p67: --enforce-eager vs CUDA graphs (2026-08-29, same seed/invocation)

Same recipe both arms (MTP k=3 included), only --enforce-eager removed; graphs arm verified to capture (FULL 5/5 + PIECEWISE 11/11), KV identical (1,164,369 tok pinned).

metric eager graphs
quality 87 (153/176) 88 (154/176)
deployability 67 68
tg t/s c1 (d0/d4096/d8192) 21.1 / 23.2 / 24.4 23.6 / 22.6 / 22.9
tg t/s c2 (d0/d4096) 39.2 / 22.9 33.1 / 24.0
TTFT (d0 / d8192) 1.45 s / 6.17 s 1.41 s / 6.10 s
MTP draft accept 90.0–92.2 % 89.0–97.0 %

Findings:

  • No graphs+MTP quality regression on 4p67 (88 vs 87 = 1 point, run-to-run noise; the failure sets differ only by 5 shuffled partials). The lab 92→89 graphs regression (forum #44) does NOT reproduce on this quant — so the bug is build/quant-specific, not universal.
  • Graphs buy nothing on decode here: every cell within ±5%, and the headline c2 number is worse with graphs (39.2→33.1). MTP already eats the launch-gap that graphs would recover.
  • Prefill rows are prefix-cache-polluted (recipe enables prefix caching and the perf probe repeats prompts); cold prefill ~1.4–1.5 K t/s both arms.
  • Decision: --enforce-eager stays in glm53-flash-nvfp4-4p67 — it is measurably free here and stays immune to the lab graph bug class.

Reports: /tmp/runs/2026/08/2026-08-29T09-09-14…--(eager), …10-27-35.818420Z_181bbd18.md (graphs).

sparkrun (community image) A/B (2026-08-30, seed 42)

aidendle94/sparkrun-vllm-glm53-gb10:production-1.0 (same day-0 vLLM dev base as our lab image, arm64-only) serving the same lab quant: quality 89 (154/176) but decode tg 14.3/23.8 t/s (c1/c2) vs lab-eager 24.4/39.2 and responsiveness 12/100 (median turn 11.7 s) — works, not performant:

  • MTP impossible in that image (no modelopt-MTP namespace fix).
  • --moe-backend marlin must be pinned (their auto = TileLang JIT storm, worker OOM-killed at boot), nvrtc.h bind-mount needed, plain fp8 KV (their mla kernel asserts pe_dim==64 for fp8_ds_mla). Report: /tmp/runs/2026/08/2026-08-30T09-41-57.562243Z_108a798b.md.

EXL3 lane (Entrpi community image) — boot diagnosis (2026-08-30)

ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2 + brandonmusic GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. VERIFIED SERVING 2026-08-30 evening: the loader-side OOM chain (--load-format instanttensor overlay) resolved — stock loader balloons ~80 GiB CPU-anon streaming the checkpoint; instanttensor streams ~175 MB/s with tiny buffers. ~22.8 tok/s with DFlash2 k=7 (accept 48-81% by content); math smoke exact. DFlash2 drafter switched to the upstream incoai/GLM-5.3-Flash-DFlash2 (downloaded to NAS at operator request); MTP3-language-only + v84 image lanes remain SM120-only upstream. Fixed in-recipe: NGC entrypoint (needs explicit vllm serve), drafter swap to the MXFP8 mirror (incoai raw drafter absent from the NAS hub), GMU 0.86. BLOCKED: global OOM kills the worker at 75-95% of shard load on BOTH streamed-from- NAS nodes (repro 2×; dmesg oom-killer 12:22:25 / 13:12:24). Upstream's "validated NFS topology" does not hold on our fabric — the mirror topology (per-node local NVMe) is the next step; dgx4 is 98% full (candidate cleanup targets ~1.3T of re-downloadable zai BF16 repos there) — awaiting operator call.

EXL3 lane bench (2026-08-30 evening, seed 42, same invocation)

glm53-flash-exl3 (brandonmusic GLM-5.3-Flash-tr3-4bpw 4bpw, Entrpi v1-dflash2 image + instanttensor overlay) — quality 88 (155/176, the failure set is the usual 16 shuffled partials/fails; nothing new flags), decode tg 23.1/24.1 t/s (DFlash2 k=7 drafting keeps C1≈C2 — the flattest concurrency profile of any glm53 lane), TTFT 2.6 s @d0, pp ~817-1,004 (noisy, NAS-adjacent). The headline: deployability 77/100 — the best of the four lanes (lab 72, 4p67 67, sparkrun 66), driven by responsiveness 50/100 with a 3.0 s median turn. Quality still trails lab-eager-with-MTP3 (92) by ~4 points — the 4bpw price. Report: /tmp/runs/2026/08/2026-08-30T17-43-04.770487Z_63900bf7.md.

0rand winning-config trial on the lab quant (2026-08-30 evening)

Forum 381350 #117/#119 report 93/100 (164/176) for the lab quant with GMU 0.89, MNBT 2048, KV auto-sized (1.05M pool), MTP-3. Reproduced on ds4 as glm53-flash-nvfp4-lab-0rand (variant in tree): KV auto-size lands at 1,167,586 tokens — same capacity as our pinned 10 GiB, so the pin is unnecessary; smoke exact. Bench (--seed 42 --hardmode --perf --concurrency 1,2,4): quality 90 (158/176), tg 27.1/34.6/37.2 c1/c2/c4, TTFT d0 1.3 s, responsiveness 25 (6.2 s median), deployability 70. vs our lab baseline (92 = 162/176; c1 28.4 / c2 40.5; resp 21, deploy 72): quality −2, c2 decode −15%, c1 ≈, but c4 decode 37.2 in an arm the baseline never ran, and responsiveness +4. On OUR stack the 0rand combo is not a strict win; keep the baseline as the daily driver, keep the variant in tree for concurrency-hungry serving. Possible confounder: their run posts a single 14.8 s median turn at responsiveness 8 — an earlier bench build. Report: /tmp/runs/2026/08/2026-08-30T19-33-22.444571Z_4616f476.md.

MNBT 1024-vs-2048 assertion on the lab quant (2026-08-30/31 night)

Forum #124 ("1024 = quality-first") asserted. Our c1,2,4 run with MNBT 1024 + seqs 16 + GMU 0.89 + auto-KV (pool 1,051,792): quality 88, denominator 146 (harness dropped ~30 pts of scenarios), tg 27.6/27.0/30.4, TTFT d0 8.6 s, resp 24 (6.4 s). vs the same-protocol MNBT-2048 run one hour earlier: quality 90 (158/176, full denominator), tg 27.1/34.6/37.2, resp 25 (6.2 s). Verdict: at our protocol (reasoning_effort-low, temp 0, seed 42) 2048 wins on both quality percentage and decode; the 1024 advantage was measured under their thinking-max + temp-0.1 + trials-2 protocol. 0rand's #123 already noted "2048 batch + MTP4 ... loses quality (89 vs 93)" for THEIR protocol — different regime, same lesson: batch tuning is protocol-dependent. Variant stays in tree; record both. Reports: 2048 = ...19-33-22Z_4616f476.md, 1024 = ...22-04-48Z_4a70e6be.md.

DeepSeek-V4-Flash-Vision-Exp lane — campaign state (2026-08-31 night)

Day-0 port landed (sfxnz plugin 7a958128b8">@7a958128b8 vendored under docker-images/dsv4-flash-vision/, bases digest-pinned; sm121 build works, 35-40 min roundtrip). Lane verified TEXT-serving on ds4 (+NVMe/NAS weights):

  • eugr b12x base: 1024-token gen in 29.4 s = 34.9 tok/s with DSpark-6 (accept 0.714, length 2.55), text math exact, plugin loads ViT+aligner weights (whole checkpoint present, 82.28 GiB/node).
  • fixing their gaps so far: plugin init.py must ship the general- plugins register (raw.githubusercontent flaked on re-fetch — fixed via the contents API); wrapper class needs supports_multimodal = True (OpenAI layer refuses image_url otherwise — interfaces.supports_multimodal is a getattr check).
  • OPEN: OpenAI-image path raises "Model class ... has no registered multimodal processor" — the day-0 plugin never registers a _processor_factory (their shipped tests exercise the loader directly, not the API). Two paths: implement the BaseMultiModalProcessor inside our plugin (grid-tokens-per-image machinery exists in dsv4_vision.image_processor; fork interface = MultimodalRegistry. register_processor factory in multimodal/registry.py), or wait for the upstream repo to ship it. aiden-fork A/B build launched.

dsv4vision MM-into-OpenAI campaign — 2026-09-01 checkpoint

Built the full OpenAI multimodal stack in our plugin (processor trio, grid/sentinel tokens, embed_multimodal + embed_input_ids merge; verified in -image: embedding rows byte-exact vs aligner(ViT(patches))[perm]) and got the engine through profiling+warmup with mm registered. Two hard walls remain (before the first image_url token):

  1. The fork's V2 "always-embed" mm arm passes input_batch.input_ids[:n] — a wrong-rank slice that arrives as [B, T]-shaped 2-D ids (observed (17,17) for a 17-token prompt, (7,7) on DSpark-6 verify steps) — and downstream logits degenerate (all-<｜begin▁of▁sentence｜>).
  2. The AOT/dynamo trace of inputs_embeds freezes the dummy's 2-D shape and then asserts "wrong number of dimensions3 for op: input" on the native 3-D stream (tilelang mhc kernels are rank-strict both ways).

State: the whole mm branch is env-gated (VLLM_DSV4_MM=1), default OFF = the verified text-only lane — restored EXACT on the new per-node NVMe weights (~/dsv4vision-hf on dgx3+dgx4, 167 GiB each, recipe HF_HOME flipped from the NAS hub; shard load 7 min → <1 min, boot ~6 min end-to-end). Bridges done tonight for the next attempt: encoder-runner + speculator buffer shape-follow patches (patch_mm_hc_streams.py), placeholder `<｜deepseek_image｜>= in-vocab id 129264,get_placeholder_str` chat integration, AOT=0 knob for mm experiments. nid: fork A/B (aiden base) still pending its pip-layout patch; dspark_gate patch now auto-finds the fork's venv root.

dsv4vision tool-eval-bench (c1, 2026-09-01)

tool-eval-bench run --base-url http://dgx3.lan:30000/v1 --model deepseek-v4-flash-vision --seed 42 --hardmode --perf --concurrency 1 → report /private/tmp/runs/2026/09/2026-09-01T17-46-28.764372Z_89b18250.md:

recipe quality points decode tg t/s c1 (d0/d4096/d8192) pp2048 t/s TTFT deployability
dsv4-flash-vision (text lane, local-NVMe, DSpark-6) 81 143/176 — — — —

perf block: tg 34.1 / 31.0 / 29.1, pp2048 3,907 / 2,714 / 2,449, TTFT 1.9-4.9 s (spec), scenario-median turn 2.4 s, deployability 74 (best of any ds4 lane) — long-context capability is the raw 1M-context draw. 3 safety-critical failures (TC-42 extra-param, TC-43 empty required param, TC-60 sleeper injection) — the day-0 0731-style tool formatting is the gap vs the GLM lanes (thinking-pair JSON fencing was already a known day-0 weakness). Quality 81 < sparkrun 89 < lab 92.

New 2026-09-01 checkpoints: Intel W4A16-AutoRound + lab NVFP4-Spark

Checkpoints on the NAS hub (hf download'd):

  • Intel/GLM-5.3-Flash-W4A16-AutoRound rev 5eee1846f0 — AutoRound 0.15 W4A16 sym int4 g168, 99.84% of BF16 (GSM8K .9712/MMLU .8620), selfattn/hc/indexer/embed lm_head FP16 by design (their ignore_layers), pure GPTQ packing (qweight/qzeros/scales — qweight count 37152).
  • local-inference-lab/GLM-5.3-Flash-NVFP4-Spark rev 06972cdee7 — NVFP4 W4A4 whole network (in_proj_qkvbfg_a.weight_scale etc.) + NVFP4 MTP expert layer in separate model-mtp-0000x shards (lil.yaml: mtp_tokens 5, dflash2_tokens 7, default_speculator mtp, block 256, disable flashinfer autotune, INSTANTTENSOR env sizes; arch Glm5NextForConditionalGeneration + vision blocks in the index). This is the rtx6kpro r12-stack checkpoint (their doc: MTP:3 254 tok/s CC1 @ TP4, Sieve 333-403).

Recipes added: recipes/glm53-flash-intel-w4a16 (timothystewart6/vllm-gb10 v0.28.0-gb10.2 arm64, gptq_marlin, entrypoint declared — tini AssertionError taught us the image has ENTRYPOINT []+CMD bash), recipes/glm53-flash-nvfp4-spark (lab image). Boot outcomes:

  • Intel lane died in the fork's Transformers-backend weight loader: "no module or parameter named 'layers.0.self_attn.q_conv1d'" — the tjstewart image has NO native glm5_next arch (vllm archs list lacks it; transformers 5.16.1 has Glm5NextForConditionalGeneration) → needs the local-inference-lab fork (native glm5next) or a transformers name-shim mod.
  • NVFP4-Spark on our lab fork: KeyError on in_proj_qkvbfg_a.weight_scale at 80% shard load — whole-network NVFP4 W4A4 is beyond the R8-era lab fork loader (r12 stack territory).

⇒ THE unifying next step (user directive): build our own arm64/sm_121a image of local-inference-lab's fork — source contract from models/glm-5.3-flash.md (local-inference-lab/vllm dev/jovian-judgement + b12x master + 6/5 PRs; kk3 build rules apply: CUDA 13.0.88, sm_121f gencode rule for block-scaled FP4). RTX docs archive: /tmp/lil-docs (ds4-flash-v6, ds4dspark-v20-r33, infernal r21, pcie-autotune) for the DeepSeek quality track.

2026-09-02: r12 fork built + NVFP4-Spark serving on it (the day's win)

Built glm53-jovian-r12:local in-docker on dgx3 (docker-images/glm53-jovian-r12/): vllm 0.1.dev20489+gcc161e220 (local-inference-lab/vllm base 54f6e9826c + PR heads 550/552/553/554/555/557 merged in order — the doc's "composition commit" 7d8a09c is not a public ref) + b12x 1.3.0 (master 139e0405 + PR heads 260-270), on the day-0 vllm/vllm-openai:glm53-flash-arm64-cu130@905c02… base. Build fixes applied along the way: apt git, setuptools-rust, ninja/cmake, cu13 wheel include-header symlinks (cusparse.h), unversioned libnvrtc.so symlink for the deepgemm_C link step.

NVFP4-Spark boot on it (recipe glm53-flash-nvfp4-spark, retargeted):

  • whole-network NVFP4 W4A4 (incl. attention in_proj_qkvbfg scales) now LOADS — the r12 fork's modelopt handles it (our lab fork: KeyError).
  • Attention wall solved: the sm12x auto-priority list only offers [TRITON_MLA (no sparse), FLASHINFER_MLA_SPARSE_SM120 (head 576 only)] — neither fits GLM head-512. Fix = --attention-backend B12X (model hook in glm5next/nvidia/attention.py: _select_sparse_backend) for the main model, and "attention_backend":"B12X" INSIDE --speculative-config for the MTP draft (eagle/utils.py: _make_eagle_draft_vllm_config takes the draft's backend from the speculative config, not the global flag — user's hint).
  • Served: exact "391" verified, MTP-3 Glm5NextMTPModel resolved, KV fp8_ds_mla default, 2-node TP2. c1 bench running.
  • Intel W4A16 recipe retargeted to the same image (gptq_marlin + B12X attn).
  • patch_cuda_sm12x.py (B12X into sm12x priorities) staged in the image dir for the next rebuild — the image-baked version of the same fix.

2026-09-02 (cont): bench + clean rebuild per operator direction

  • NVFP4-Spark (r12 fork image, day-0 base) c1 bench: 88/100 (120/136, default 69-scenario suite; deployability 67, median turn 8.1s — TP2 GB10 is slower per-turn than the eugr vision lane's 2.2s on the same suite, also 88). Same-suite A/B vs vavrusa lane: 88 = 88. The lab's 92 was the 88-scenario/176-point suite — that invocation's re-run is pending.
  • Operator: docker_image_dir in the spark recipe pointed at the lab dir — fixed to glm53-jovian-r12 (intel recipe same).
  • Operator: "start with cuda image, solar-open2 pattern but r12-specific". Dockerfile/versions.env rewritten: raw nvidia/cuda:13.2.0-devel-ubuntu24.04 (a5b6256e…, solar-pinned) → apt stage (solar locks) → torch 2.13.0 cu130 + flashinfer 0.6.17 (whl index) → vllm wheel (fork base 54f6e9826c + PR-head merges 550-557, use_existing_torch, ccache) → b12x wheel (master + PR heads 260-270, pure-python) → runner installs both wheels + deps, bakes patch_cuda_sm12x.py, GB10 gate (vllm/b12x/_flashkda_C imports, B12X enum, patch presence). No day-0 vllm contamination; ENTRYPOINT [] CMD bash. Build running on dgx3 (tmux r12build → /tmp/r12build2.log), same tag glm53-jovian-r12:local (running serve keeps the old image; re-verify lane after build).

2026-09-02 (cont 2): lil — the lab's own launcher, read

local-inference-lab/lil = standalone Go launcher (their sm121 equivalent): reads each HF model repo's lil.yaml at runtime + embedded family policy + discovered topology (local or spark-rdma via SSH), renders/validates/runs a typed vLLM invocation. lil render <model> prints the exact env + command.

GLM family policy (configs/models/_bases.yaml) — the lab's canonical launch for these checkpoints, vs our spark recipe today:

  • --attention-backend B12X, --linear-backend b12x, --moe-backend b12x (we have only the global attention flag; linear/moe backends MISSING)
  • spec config: {"method":"mtp","num_speculative_tokens":5 (lil.yaml mtp_tokens), "moe_backend":, "attention_backend":"B12X"} — we use k=3 and no moe_backend. MTP:3 is the r12 x86-qualified mode; the checkpoint's manifest default is 5.
  • --load-format instanttensor (+INSTANTTENSOR_BUFFER_SIZE/CHUNK_SIZE env)
  • --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096, --max-model-len auto
  • --reasoning-parser glm45, --tool-call-parser glm47
  • DFlash2 (the GLM-side second speculator, method "dflash") needs the separate draft checkpoint local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8.

    • TODO: after the clean image lands, align the spark recipe to the lil policy (moe/linear backend b12x, spec moe_backend, instanttensor, mamba-cache-mode, parsers) and A/B k=3 vs k=5.

    Clean r12 build: fix log (builds 2-9)

    1. flashinfer: whl-index path dead → flashinfer-python 0.6.17 from PyPI (fork's setup.py skips flashinfer-cubin at build; runtime fetches cubins).
    2. git protocol v2 flake in-container → GIT_PROTOCOL=0 + protocol.version 0.
    3. setuptools: fork build needs setuptools-rust + setuptools-scm (uv hint).
    4. DeepGEMM FetchContent: pinned SHA on non-default nv_dev branch → GitHub 404s raw-SHA fetch → DEEPGEMM_SRC_DIR pre-clone stage (cmake escape hatch).
    5. PR 550 closed (unmerged) → refs/pull/550/head deleted; all PR heads now pinned by commit SHA (vllm 550-557, b12x 260-270). Raw-SHA fetches verified working.
    6. Operator rule: no build while a serve container runs (OOM risk on 128 GB unified) — spark lane stopped for the build window.

    2026-09-02 (cont 3): clean r12 image SHIPPED

    Image glm53-jovian-r12:local rebuilt from raw nvidia/cuda:13.2.0-devel (solar pattern, r12-specific content): 16.1 GB (vs 37.6 GB day-0-based), gates green (vllm 0.1.dev20489+g06f922207, b12x, _flashkda_C, B12X enum, sm121 patch), synced dgx3→dgx4. Boot fixes found on first clean boot:

    • tini "[FATAL] exec local-inference-lab/… failed" → clean image is ENTRYPOINT [] (solar contract), so the RECIPE now declares entrypoint = "vllm","serve".
    • FlashInfer JIT OOM: 12 parallel nvcc jobs (mxfp8_gemm_cutlass_sm120 etc.) get OOM-killed (code 137) while 93 GB weights + 10 GB KV are resident on 121 GB unified → MAX_JOBS=2 in recipe [env]; JIT (~15 min) then survives. Boot total ~35 min (NFS shards 16 min + JIT 15 min). c1 bench (default 69-scenario suite, same as the vavrusa run): 90/100 (124/138; quality 90, deployability 69, median 7.9 s) — vs 88 on the day-0-based image, same suite. Clean image is now the reference GLM lane.

    Remaining lab-alignment (next boot, one change-set at a time): --linear-backend b12x + --moe-backend b12x, spec moe_backend:b12x, A/B k=3 vs k=5 (lil default), --load-format instanttensor (+env), --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096, --reasoning-parser glm45, --tool-call-parser glm47. Then: Intel W4A16 boot on the clean image (queued, needs cluster).

    2026-09-02 (cont 4): lab-stack perf push — eugr build pipeline adopted

    Goal (user): 30 tok/s in --perf. Found the missing pieces:

    1. lil render reference (built lil from local-inference-lab/lil @ cb58d54, rendered with testdata manifest GLM-5.3-Flash-NVFP4-Spark + spark.yaml): our recipe flags now 100% match the family policy (_bases.yaml glm: b12x linear/moe, B12X attention, instanttensor, block 256, seqs 8, batched 4096, glm45/glm47, mamba align) + spec {mtp:5, moe_backend b12x, B12X} + env VLLM_USE_AOT_COMPILE=1 / V2_MODEL_RUNNER=1 / FLASHINFER_SAMPLER=1 / BREAKABLE_CUDAGRAPH=0. Boot 35m -> ~7m (instanttensor loader, user: keep). Perf after engine switch: d0 c1 24.2 / c4 31.8 tok/s (was 10.3/16.5). Quality dropped with full lil set (16/22 at TC-22) -> bisection in progress; user: instanttensor is fine, problem elsewhere.
    2. eugr/spark-vllm-docker (the lab's actual build repo, main): 804-line Dockerfile, CUDA 13.0.2 base, 12 build-time vllm patches (docker/patchvllm*.py), b12x from source (lukealonso/b12x; lab: local-inference-lab/b12x master), nightly DockerHub images eugr/spark-vllm[-b12x]:latest (stale 08-23, user: dockerhub is the wrong source — build from source). Ported all 12 patches into docker-images/glm53-jovian-r12/patches-eugr/
      • Dockerfile (VLLM_PRESERVE_SM12X_TARGET=1, VLLM_PATCH_B12X_C128A_ALIGNMENT=1; patches are idempotent, several no-op on our r12 ref). Build blocker: node github.com access flaky ("could not read Username" on FetchContent clones) -> pre-fetched all 9 sub-deps on the Mac into deps-src.tar.gz (triton_kernels@v3.5.1, MSA, FlashMLA, FlashKDA, flash-attention, qutlass@e74319e, tml-fa4@b206834, cutlass@v4.4.2) + the SRC_DIR env escapes each cmake documents. Build 14 compiling.

    r17 (20260902-r17 card) adoption + quality bisect (2026-09-03)

    Latest lab card: https://github.com/local-inference-lab/rtx6kpro/blob/master/models/glm-5.3-flash.md (names the qualified serving contract per artifact and SHA; TP4 on RTX PRO, not Spark — the Spark lil.yaml testdata render remains our TP2 adapter).

    1. Public r17 composition built from source: /tmp/vllmr12 = local-inference-lab/vllm dev/jovian-judgement@9c4dd054

      • open PRs 579/580/581/582/586 (branch supersedes the r12 PR set 550–557 — same symbols, evolved, see below) + staged-cutlass cmake patch; /tmp/b12xb17 = local-inference-lab/b12x master@2bbab479 + PRs 290/284. Pre-staged as vllm-src.tar.gz / b12x-src.tar.gz on the build node (WAF blocks node→github git). DO NOT re-merge 550-557: creates conflict-marker SyntaxErrors (pr550's pooled_indexer.py conflicts; branch tip already contains the evolved implementation).

    2. Important correction to the 2026-08-28 eager verdict: on the public r12/r17 refs, graphs+MTP is no longer a quality bug — the lab's private "packed GLM-5.3 package" (card SHAs 47f86087/a02841bc, not public) was the presumed fix carrier, but our public-tree composition measured:

      • r17 + graphs + 90/100-flags (no b12x/align): 88/100 hardmode (155/176) == lab-quant's own day-0 reference 156/176 (r17graphs01/…perf)
      • r17 + eager same-flags: 67/100, median turn 11.4 s, 2 scenarios timed out and excluded (r12eager01) — eager is now the WORST mode. So: graphs stay ON; enforce-eager is gone.

    3. Execution-env knobs are quality-clean: GMU 0.90 (0.95 boot-deaths on GB10 — "free memory 111.96 < desired"), max-num-seqs 16, OMP 2, VLLM_USE_STANDALONE_COMPILE/FLASHINFER_SAMPLER/BREAKABLE envs: 87/100 (r17knobsonly). AOT=1+V2 runner: NEUTRAL for quality (47 either way when backends were on) and NEUTRAL for the d0 grid — dropped (AOT=0, no V2).

    4. THE QUALITY TOXIN = the b12x backend-flag cluster (on the PUBLIC r17/r12 compositions, with our b12x 1.3.0/1.2.6 builds):

      • k=5 full cluster: 74/100 (r12 img), ~74 → killed-to-16/22 (r12 img)
      • k=5 full cluster (r17 img): ~70-ish band (r17cluster01, stopped early)
      • k=3 full cluster: 46/100 (r17clusterk3)
      • k=3 cluster minus AOT/V2: 47/100 (r17noaotk3)
      • k=3 cluster minus backends (knobs only): 87/100 (r17knobsonly) The lab's own stack runs these backends qualified — but their composition carries the private packed GLM-5.3 package (RTX-PRO target); the public branch tip alone does NOT make the b12x W4A4/linear/align paths quality-safe. Also: b12x cluster wrecked deep-context speed (d4096 c1 12.6 @k5) while helping d0 (33-34 c1) — nets negative anyway.

    5. SHIP ANCHOR (current recipe) = r17 image (94ac0882a8) + 90/100 args (attention B12X, block 256, seqs 16, batched 4096, LM-only, glm47/glm45 parsers, instanttensor load, k=3 spec {mtp,attn B12X} with moe_backend key) + graphs (no eager) + knobs env (GMU 0.90, seqs 16, OMP 2, AOT=0, STANDALONE=1, SAMPLER=1, BREAKABLE=0) + JIT-cache mounts: 87/100 hardmode (153/176; lab-quant ref 156/176), default-suite 88 class; perf d0 c1 24.1 / c2 30.2 / c4 44.7; d4096 c1 23.9; d8192 c1 23.8; pp ~1.6-1.8k. Boot 8 min (JIT warm) / ~25 min cold (nvcc re-JIT).

    6. Boot-time JIT: changing the rw flashinfer cache invalidates per-config kernel sets; a config's first boot silently nvcc-compiles (MAX_JOBS=2) ~15-25 min inside "No available shared memory" log-silence; later boots of the same config are fast. Expect this on every NEW kernel path.

    7. KV-dtype probes: fp8_ds_mla on r12-img = 75/100 default-suite (net zero, TC-45 flipped PASS but others flipped off); graphs here. r17 anchor keeps default dtype. Not a quality lever.

    8. Open perf gap: c1 24 vs manifest-cluster 33-34 (which is quality-dead). If the lab ever publishes the packed package (or a public branch commit makes b12x-backends clean), the 30+ c1 target lands with manifest flags. Alternative to probe later: one b12x backend flag at a time (a single toxin member may be free).

    One-backend bisect (2026-09-03) — SOLVED (hardmode, r17 img 94ac0882, k=3 graphs)

    • knobs-only: 87 (153/176); d0 c1 24.1, d4096 23.9, d8192 23.8 (r17knobsonly)
    • + --moe-backend b12x (spec default MTP experts): 88 (154/176); d0 23.5, d4096 22.5, d8192 26.6 (r17moeonly) — quality-clean, keep
    • + --linear-backend b12x: 44 (77/176), Parameter Precision 0%, TC-21 0/5 validations — THE toxin. B12X linear kernel path destroys numerics on the public composition (r17linearon)
    • + --mamba-cache-mode align: 88 (154/176) but d0 c1 18.6 (−4.9), no quality gain (r17alignon) — perf-negative, dropped
    • Full-cluster combos (44-74) were dominated by linear; spec's moe_backend b12x key and the knobs env family are all exonerated individually. SHIP = r17 img + 90/100 args + graphs + knobs env + --moe-backend b12x only (recipe @ 2d379f9): 88/100, d0 23.5 / c4 43.9, depth 22.5/26.6. The 30 c1 target is only reachable with linear-b12x (29.6 d0) which is quality-dead until the lab publishes the packed GLM-5.3 package.

    Checkpoint A/B on ds4 (2026-09-03, hardmode 176pt): Spark vs NVFP4 lab checkpoint

    User-flagged must-have: our lane served the Spark-variant checkpoint with quality losses vs lab references; A/B the checkpoint under matched stacks.

    Cell Stack Checkpoint Hardmode d0 c1 d4096 c1 d8192 c1 notes
    r17ablabimg lab img + lab recipe (eager, fp8_ds_mla, k3, default MoE, day-0 fork 0.1.dev20051+g487ecf187) NVFP4 93 (164/176) ★★★★★ 24.4 26.5 22.8 78 pass / 2 fail; TC-45 PASS; matches lab's own 92 ref
    r17ablabck shipped r17 img + ship flags (graphs, knobs, k3, default MoE — moe-b12x unsupported on this ckpt: layer-45 is MXFP8) NVFP4 88 (154/176) 27.5 29.6 26.6 46.4/46.7 c2/c4 at d0
    r17moeonly (ship) shipped r17 img Spark 88 (154/176) 23.5 22.5 26.6 —

    Verdict:

    1. Checkpoint is not the quality factor on the r17 stack (88 = 88 both checkpoints) — but the NVFP4 lab checkpoint is significantly FASTER (+4 tok/s d0, +7 d4096) and lighter rules (no --moe-backend b12x).
    2. The lab day-0 stack retains +5 quality (93 vs 88) regardless of checkpoint — its eager + fp8_ds_mla + default-everything runtime is what the lab's own 92-hardmode reference was measured on. Enforce-eager costs prefill pp at c2/c4 (0.9-1.1k vs 1.5-1.6k) but TG is equal-or-better.
    3. Recipes: glm53-flash-nvfp4-r17abi (ship flags + NVFP4, moe flag removed) is the fastest quality-parity lane; glm53-flash-nvfp4-lab remains the quality champion (93).

    Night-shift queue (2026-09-05)

    Prepared lanes + builds queued for the night (cluster is serving the 4x aiden lane during the day — stop it first, then all 4 nodes are free). Order = value/duration; each item lists prereqs + expected verdict.

    1. r18 fork build (docker-images/glm53-jovian-r12, ~1-2 h build). Stage on the build node: /tmp/vllm-src.tar.gz (94 MB) + /tmp/b12x-src.tar.gz (29 MB) — pins per versions.env (vllm a8c796f3+pr582+pr646 → 343661fe; b12x b58f34ea+pr284 → 1adb8a38), then TAG=glm53-jovian-r12:local bash build.sh. Test campaign: (a) Re-baseline the quality anchor first — DONE 2026-09-06: ran glm53-flash-nvfp4-r17abi (2x, r17 image) on 2nodes-dac with the current tool: NEW BASELINE = 153/176 (87/100, deployability 69) — report in recipes/glm53-flash-nvfp4-r17abi/reports/2026-09-06_r18-rebaseline-r17abi-2x.md (2 pts below the historical 155/176 anchor: DAC fabric + NAS weights; scoring confirmed current-tool-compatible: 88 scenarios × 2 pts = 176). Gate ALL new lanes against 153/176 (≥151 = parity, <151 = regression); (b) **re-bisect `--linear-backend b12x`** — new b12x A16 precision selection may clear the 44/100 toxin (29.6 c1 unlock if clean), (c) RoCE one-shot on the 4x lane (VLLM_ENABLE_ROCE_ALLREDUCE=1; expect −5–19% decode per-step per their #597 numbers), (d) `--kv-cache-dtype nvfp4_ds_mla` probe (new B12X KV path), (e) #646 geometry check: 2048/256/256 split blocks selected + repeat-request prefix resume at the fine boundary (their regression scenario: retention 0, states 32/16, resume at 32) — if broken, drop the 646 pin from versions.env and re-cut (rest of r18 is independent). 1b. **DCP4 probe** (4x lane, no build dependency — run on the current r17abi image first, re-run on r18 if positive). Add `--decode-context-parallel-size 4` (stock vLLM flag, verified in the r18 tree: vllm/config/parallel.py; DCP>1 requires MLA — GLM53 qualifies). Reference (sparkring MTP3 mesh profile, 4 Sparks, DCP4 + 24 GiB fp8 KV/rank): ~4.32M logical KV tokens (vs our ~1.25M at 12 GiB DCP1), 231.3 aggregate decode tok/s peak, prefill ~2.7k flat 8K→128K. Verdict: KV-token capacity per node at our pin, decode C1/C8/C16 vs baseline, prefill 8K/64K. Their fabric is a direct HW-forwarded mesh (cheaper all-reduce than our switch) — measure, do not extrapolate.
    2. RedHatAI MTP lane (2x, ~30 min). Re-run huggingface-cli download RedHatAI/GLM-5.3-Flash-NVFP4 on both nodes (model_mtp.safetensors 7.6 GB is new since the initial pulls; HF_HUB_OFFLINE=1). Serve glm53-flash-nvfp4-redhat (MTP k=5 per vendor card) → compare vs the 90/100 no-MTP baseline + decode speed; memory watch +7.6 GB/node (KV pin 10→8 GiB if boot dies; k=3 fallback).
    3. eugr b12x image verify + GLM lane (2x, ~1 h). Pull eugr/spark-vllm-b12x:nightly-20260904 (10.9 GB) on the 2 nodes, read /workspace/build-metadata.yaml + /workspace/b12x-source-commit (record exact pin in the recipe header). huggingface-cli download local-inference-lab/GLM-5.3-Flash-NVFP4-Spark on both nodes. Serve glm53-flash-nvfp4-eugr → A/B vs r18: full b12x cluster quality (linear-b12x toxin retest on their stack) + RoCE 2x decode numbers. 4-node cell: glm53-flash-nvfp4-eugr-4x (needs all 4 nodes free) vs the 2x lane — 2x stays the production default unless 4x is dramatic. 3b. DFlash2 spec A/B (same eugr image, after 3 boots). Swap the spec to "method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2", "num_speculative_tokens":7,"draft_tensor_parallel_size":2, "kv_cache_dtype":"auto" (separate drafter repo — huggingface-cli download on both nodes first; exists on HF, single safetensors). VS the recipe's MTP k=5: decode at C1/C4 + quality spot-check (forum numbers are perf-only: ~60 c1 / ~73 c4 tok/s on 4x — their 4x/TP4 config, so 2x numbers will differ).
    4. DSV4F-Vision eugr lane (2x, prefer dgx3/dgx4 — weight reuse). Reuse the per-node ~/dsv4vision-hf copies (verify the HF_HOME mapping vs the recipe's default HF cache) or re-download Vision-Exp (168 GB). Serve dsv4-flash-vision-eugr → compare vs the fork lane (dsv4-flash-vision, DSpark-6): quality (text + vision smoke) + decode
      • TTFT; spec A/B: k=3 adaptive (vendor card) vs k=6 (fork lane's proven value).

    Night-shift results (2026-09-05/06, all seed 42, hardmode, c1,2)

    Re-baseline (queue 1a): r17abi 2x/2nodes-dac = 153/176 (87/100, dep 69) with tool-eval-bench 2.6.1.dev45+gcf54b4bfe (88 scenarios). This is the new canonical anchor — all lanes below gate against it. Report: recipes/glm53-flash-nvfp4-r17abi/reports/.

    r18 build (5 attempts on dgx4, log r18build*.log): three root causes — (1) transient snapshot.ubuntu.com 503s; (2) patch_vllm_spark_kv_cache_cleanup.py anchor gone (upstream replaced the profiling flow with MemorySnapshot/request_memory which already cleans before the snapshot — patch now detects the new flow and exits 0); (3) staged-deps submodule stubs: both flash-attention/csrc/cutlass AND MSA/python/fmha_sm100/cutlass were empty dirs in deps-src.tar.gz (the r18 tree dropped implicit cutlass include wiring / MSA's install needs the dir) — populated with each project's own pin (FA: nvidia/cutlass 62750a2b, MSA: nvidia/cutlass eb61c911) and purged 23k macOS AppleDouble ._* files. Image: tag glm53-jovian-r12:local, tree 67e7e6e (base a8c796f3 + #582 + #646), 16.1 GB, on all 4 nodes.

    Lane results (gate = 153/176 baseline; quality is the gate, dep = deployability):

    Lane /176 /100 dep KV tokens verdict
    r17abi 2x (baseline) 153 87 69 ~1.25M anchor
    r18 2x (ds4) 157 89 71 — PASS (+4)
    RedHat MTP k=5 2x 156 89 70 — PASS (MTP works, +3)
    eugr b12x MTP k=5 2x 143 81 65 — FAIL (−10)
    r18 4x RoCE TP4 156 89 76 1.47M PASS
    r18 4x DCP4 (TP4+--decode-context-parallel-size 4+--cp-kv-cache-interleave-size 4) 161 91 75 4.84M BEST
    r18 4x nvfp4_ds_mla KV 161 91 77 2.35M PASS (best dep)

    Key findings:

    • DCP4 works on our switch fabric (sparkring steal confirmed): 4.84M KV tokens at the SAME 10 GiB/node pin — 3.3× plain 4x, and beats their 4.32M @ 24 GiB/rank (their HW mesh gave them 231 t/s aggregate; our RoCE switch numbers are in the report). Best quality of the night (91). B12X is NOT DCP-compatible (engine auto-falls the DCP cell to FLASH_ATTN attention; that's what benched).
    • nvfp4_ds_mla KV dtype (r18-tree value, vllm/config/cache.py): 1.59× the fp8 capacity at the same pin, tied quality, best deployability. Combine with DCP4 next (projected ~7.7M KV).
    • #646 geometry live (queue 1e): serve logs show "Setting attention block size to 2304" = the 2048+256 split-block geometry; no retention/states anomalies seen, no fallback needed.
    • eugr image lane fails the quality gate (143 vs 153) on 2x — the linear-b12x toxin persists on their full b12x cluster; eugr-4x cell skipped (2x already decides it).

    Blocked (documented, not solvable from recipes):

    • DFlash2 drafter lane (queue 3b): the drafter inherits the engine's FP8 KV dtype; on SM121 B12X is rejected for it (kv_cache_dtype constraint) and FLASH_ATTN too ("FP8 KV cache requires FA3 on SM90 or FA4 on SM100"). With drafter kv_cache_dtype: "auto" it fails the FORK's own assertion: ValueError: GLM MLA parent-page stride must be an exact number of C4 pages. Fork-side fix needed (drafter KV geometry under C4 paging) before the k=7 A/B can run.
    • dsv4-flash-vision-eugr (queue 4): the eugr 0904 image's DeepseekV4ForCausalLM (vllm/models/deepseek_v4/nvidia/model.py) contains NO aligner module — the image predates V4-Vision support; the Vision-Exp checkpoint's vision weights have nowhere to land. Re-check on a newer eugr image; fork lane (dsv4-flash-vision) remains the only V4-Vision path.

    Open decisions: (1) production 4x candidate = r18 + DCP4 (quality 91, 4.84M KV) — needs a production recipe + operator sign-off; (2) r18 2x adoption as the new 2x default (157 vs 153 baseline); (3) DCP4+nvfp4 KV combined cell; (4) eugr lanes stay demoted.