Главная Обзор Помощь
Вход
sm121
/
launcher
2
0
Ответвить 0
Файлы Задачи 0 Запросы на слияние 0 Вики
Дерево: 08a930e477
Ветки Метки
launcher
/
specs
/
tech-architecture
/
glm53-recipe-state.md

glm53-recipe-state.md 28 KB
История Исходник

GLM-5.3-Flash NVFP4 — three quant variants measured on 2× DGX Spark (ds4)

Measured 2026-08-27/28 on cluster ds4 (dgx3+dgx4, switched fabric), all three served by sm121 through the launcher, same benchmark invocation:

tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53 \
  --seed 42 --hardmode --perf --concurrency 1,2

All three share served name glm53 — clients never changed between runs.

Results

recipe quant quality points decode tg t/s (c1) tg c2 TTFT deployability MTP
glm53-flash-nvfp4-lab local-inference-lab mixed NVFP4/MXFP8-MTP 92 (eager) / 89 (graphs) 162/176 28.4 40.5 5.6 s 72 ✅ k=3, accept len ~2.9–3.5, ~63–83 %
glm53-flash-nvfp4-redhat RedHatAI W4A4 compressed-tensors 90 159/176 13.9 13.8 2.2 s 67 ❌ (no MTP weights in checkpoint)
glm53-flash-nvfp4 LibertAIDAI NVFP4-A16 88 154/176 26.2 22.7 6.3 s 69 ✅ k=5
glm53-flash-nvfp4-4p67 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 87 153/176 24.4 39.2 1.4 s 67 ✅ accept ~90%
glm53-flash-sparkrun community sparkrun image (aidendle94), NVFP4, eager, NO MTP (image lacks the fix) 89 154/176 14.3 23.8 1.4 s 66 ❌ (n/a in image)
glm53-flash-exl3 brandonmusic EXL3/TR3 4bpw (Entrpi image + instanttensor), DFlash2-7, graphs 88 155/176 23.1 24.1 2.6 s d0 77 ✅ accept 48–81 %

(lab scored /174 — TC-53 was excluded from its run as an infrastructure failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s total — harness drops timed-out scenarios from the denominator; had it scored like redhat's 2/2, lab would be 157/176 = 89 — no change). Quality % is comparable. LibertAIDAI is the only variant whose MTP regresses under c2 (26.2→22.7) — lab's k=3 draft scales, LibertAIDAI's k=5 does not.)

Prefill (pp2048) 0.3–1.7 K t/s depending on depth/concurrency — noisy, all three in the same band (NAS load-influenced); decode is the stable discriminator.

Read of the numbers

  • lab (eager) wins outright: 92/100 best quality AND 2× decode (28–40 t/s vs ~14 baseline). Acceptance lengths matched upstream's 2.55–3.5 range.
  • redhat (90/100 with graphs) is a baseline-decode dead-end on this box: its checkpoint ships zero MTP tensors (verified index: 146,745 tensors, none under layer 45), so it is stuck at ~14 t/s — no longer the quality pick either.
  • lab --enforce-eager = day-driver; redhat = fallback; LibertAIDAI = proven first but k=5 MTP regresses at c2 (eager might lift it too — untested).

Images & patches

image dir base patches
glm53-flash-nvfp4 vllm/vllm-openai:glm53-flash-arm64-cu130 kingjones30 NoPE-MLA mod (4-part, anchor-asserted)
glm53-flash-nvfp4-lab same + 4 FujitsuPolycom patches (Apache-2.0, see patches/UPSTREAM.md): model.py naming shim, modelopt MTP namespace fix, persistent_topk CC-12.x guards ×2
redhat recipe reuses glm53-flash-nvfp4:local (mod is quant-independent)

Boot markers [quantprobe] layer=RoutedExperts prefix=model.layers.45.mlp.experts algo=MXFP8 confirm the MTP fix is live; algo=None there = broken load.

MTP + CUDA graphs quality bug (forum #381350/#44, CONFIRMED 2026-08-28)

0rand: “mtp or dflash plus cuda graphs seriously hit quality — try disabling cuda graph capture”. A/B with identical seed/flags on lab:

arm quality points tg c1 tg c2 TTFT
graphs (CUDA graphs ON) 89 155/174 28.4 43.4 3.9 s
--enforce-eager 92 162/176 28.4 40.5 5.6 s

+3 quality points for −7 % c2 decode — eager is now the lab default. Accept metrics were healthy in BOTH arms (71.6 %, len 3.15 eager), so this is not an acceptance collapse — graph capture corrupts sampled output silently. Corroborating: randomllama (DFlash2 recipe) already flagged “not claiming bit-identical greedy outputs” with graphs on. The b12x glm52 stack runs graphs + MTP k=4 FULL_AND_PIECEWISE benches clean (100/100) — build-specific bug in the glm53 day-0 dev build, do NOT propagate --enforce-eager to glm52. Untested follow-ups: LibertAIDAI + eager (k=5; expect ~88→9x), redhat is graph-neutral (no MTP, stays 90 with graphs).

Operational notes (measured)

redhat @ 524288 ctx (2026-08-28)

262144 booted on the base image; 512K needed three changes (each verified by a real boot that died without it):

  1. --kernel-config {"enable_flashinfer_autotune":false,"enable_cutedsl_warmup":false}
    • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — autotune scratch at 512K shapes OOMs the driver (NVRM NV_ERR_NO_MEMORY _memdescAllocInternal); the sparse-MLA kernels' non-autotuned fallback is fine (lab proved it).
  2. --max-num-batched-tokens 4096 + --block-size 256 (lab's memory shape).
  3. The lab image — the killer was launch_persistent_topk ... total_ctas=62 > num_sms*occupancy=48 ... fallback requires >=128KB smem (have 101376), i.e. exactly what the FujitsuPolycom sparse_attn_indexer(_kpool) CC-12.0 guards fix. Indexer CTAs scale with ctx: 262K fit 48 SMs, 512K needs 62. The patches are arch-level → safe for the compressed-tensors checkpoint (quantprobe is a no-op without modelopt; output verified correct).

Result: KV pool 1,334,265 tokens / 2.54× @ 524288 (10 GiB pin; lab gets 1,164,369 from the same pin — its MTP speculator reserves memory before KV sizing), ~17 min clean boot, correct greedy output.

Note: an orphaned rank-1 headless worker survives rank-0 death and keeps the node occupied — always docker rm -f label-filtered leftovers before a retry serve (or the launcher refuses with "only 1 nodes").

  • lab weight load: 92.19 GiB / 700 s sequential from the NAS HF cache; --kv-cache-memory-bytes 10737418240 → KV 1,164,369 tokens / 2.22× @ 512K (matches upstream to the digit — do NOT switch back to GMU sizing).
  • First launcher serve of lab died on both ranks at ~54 s (exit 1, logs lost to --rm); re-run was clean twice. Suspected transient (cold imports + simultaneous launch). If it recurs: re-launch without --rm and keep logs.
  • sm121 serve immediately after sm121 stop can fail with "cluster has only 1 nodes" — occupancy race, nodes not yet released; wait ~10 s and retry.
  • LibertAIDAI/redhat recipes still run FlashInfer autotune (60-min possible boot, VLLM_ENGINE_READY_TIMEOUT_S=3600 set); the lab recipe disables it via --kernel-config per upstream's proven boot.
  • Chat-template caveat: reasoning_effort:"low" still emits a reasoning trace (budget ≠ 0); max_tokens:48 can yield content: null, finish: length — use ≥256 for smoke tests.

Follow-ups

  • Vendor instanttensor loader into the lab image (~30 s vs 700 s weight load).
  • Consider adding --kernel-config autotune-off to the LibertAIDAI recipe (boot minutes shorter; redhat now has it, see 512K section — verify before trusting on LibertAIDAI).
  • MTP for redhat: RedHatAI ships MTP as separate variant repos (proven pattern: RedHatAI/GLM-5.2-NVFP4-FP8-mtp-FP8 = full quant + mtp.safetensors alongside main weights). Nothing for 5.3-Flash yet (org search 2026-08-28: only NVFP4 + BF16). If/when GLM-5.3-Flash-NVFP4-mtp-* lands, the recipe change is: swap model + add --speculative-config (k=3). Cross-grafting the lab's MXFP8 MTP as an external speculative-config.model draft is theoretically possible but untested in vLLM — don't chase before the official drop.

4p67: --enforce-eager vs CUDA graphs (2026-08-29, same seed/invocation)

Same recipe both arms (MTP k=3 included), only --enforce-eager removed; graphs arm verified to capture (FULL 5/5 + PIECEWISE 11/11), KV identical (1,164,369 tok pinned).

metric eager graphs
quality 87 (153/176) 88 (154/176)
deployability 67 68
tg t/s c1 (d0/d4096/d8192) 21.1 / 23.2 / 24.4 23.6 / 22.6 / 22.9
tg t/s c2 (d0/d4096) 39.2 / 22.9 33.1 / 24.0
TTFT (d0 / d8192) 1.45 s / 6.17 s 1.41 s / 6.10 s
MTP draft accept 90.0–92.2 % 89.0–97.0 %

Findings:

  • No graphs+MTP quality regression on 4p67 (88 vs 87 = 1 point, run-to-run noise; the failure sets differ only by 5 shuffled partials). The lab 92→89 graphs regression (forum #44) does NOT reproduce on this quant — so the bug is build/quant-specific, not universal.
  • Graphs buy nothing on decode here: every cell within ±5%, and the headline c2 number is worse with graphs (39.2→33.1). MTP already eats the launch-gap that graphs would recover.
  • Prefill rows are prefix-cache-polluted (recipe enables prefix caching and the perf probe repeats prompts); cold prefill ~1.4–1.5 K t/s both arms.
  • Decision: --enforce-eager stays in glm53-flash-nvfp4-4p67 — it is measurably free here and stays immune to the lab graph bug class.

Reports: /tmp/runs/2026/08/2026-08-29T09-09-14…--(eager), …10-27-35.818420Z_181bbd18.md (graphs).

sparkrun (community image) A/B (2026-08-30, seed 42)

aidendle94/sparkrun-vllm-glm53-gb10:production-1.0 (same day-0 vLLM dev base as our lab image, arm64-only) serving the same lab quant: quality 89 (154/176) but decode tg 14.3/23.8 t/s (c1/c2) vs lab-eager 24.4/39.2 and responsiveness 12/100 (median turn 11.7 s) — works, not performant:

  • MTP impossible in that image (no modelopt-MTP namespace fix).
  • --moe-backend marlin must be pinned (their auto = TileLang JIT storm, worker OOM-killed at boot), nvrtc.h bind-mount needed, plain fp8 KV (their mla kernel asserts pe_dim==64 for fp8_ds_mla). Report: /tmp/runs/2026/08/2026-08-30T09-41-57.562243Z_108a798b.md.

EXL3 lane (Entrpi community image) — boot diagnosis (2026-08-30)

ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2 + brandonmusic GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. VERIFIED SERVING 2026-08-30 evening: the loader-side OOM chain (--load-format instanttensor overlay) resolved — stock loader balloons ~80 GiB CPU-anon streaming the checkpoint; instanttensor streams ~175 MB/s with tiny buffers. ~22.8 tok/s with DFlash2 k=7 (accept 48-81% by content); math smoke exact. DFlash2 drafter switched to the upstream incoai/GLM-5.3-Flash-DFlash2 (downloaded to NAS at operator request); MTP3-language-only + v84 image lanes remain SM120-only upstream. Fixed in-recipe: NGC entrypoint (needs explicit vllm serve), drafter swap to the MXFP8 mirror (incoai raw drafter absent from the NAS hub), GMU 0.86. BLOCKED: global OOM kills the worker at 75-95% of shard load on BOTH streamed-from- NAS nodes (repro 2×; dmesg oom-killer 12:22:25 / 13:12:24). Upstream's "validated NFS topology" does not hold on our fabric — the mirror topology (per-node local NVMe) is the next step; dgx4 is 98% full (candidate cleanup targets ~1.3T of re-downloadable zai BF16 repos there) — awaiting operator call.

EXL3 lane bench (2026-08-30 evening, seed 42, same invocation)

glm53-flash-exl3 (brandonmusic GLM-5.3-Flash-tr3-4bpw 4bpw, Entrpi v1-dflash2 image + instanttensor overlay) — quality 88 (155/176, the failure set is the usual 16 shuffled partials/fails; nothing new flags), decode tg 23.1/24.1 t/s (DFlash2 k=7 drafting keeps C1≈C2 — the flattest concurrency profile of any glm53 lane), TTFT 2.6 s @d0, pp ~817-1,004 (noisy, NAS-adjacent). The headline: deployability 77/100 — the best of the four lanes (lab 72, 4p67 67, sparkrun 66), driven by responsiveness 50/100 with a 3.0 s median turn. Quality still trails lab-eager-with-MTP3 (92) by ~4 points — the 4bpw price. Report: /tmp/runs/2026/08/2026-08-30T17-43-04.770487Z_63900bf7.md.

0rand winning-config trial on the lab quant (2026-08-30 evening)

Forum 381350 #117/#119 report 93/100 (164/176) for the lab quant with GMU 0.89, MNBT 2048, KV auto-sized (1.05M pool), MTP-3. Reproduced on ds4 as glm53-flash-nvfp4-lab-0rand (variant in tree): KV auto-size lands at 1,167,586 tokens — same capacity as our pinned 10 GiB, so the pin is unnecessary; smoke exact. Bench (--seed 42 --hardmode --perf --concurrency 1,2,4): quality 90 (158/176), tg 27.1/34.6/37.2 c1/c2/c4, TTFT d0 1.3 s, responsiveness 25 (6.2 s median), deployability 70. vs our lab baseline (92 = 162/176; c1 28.4 / c2 40.5; resp 21, deploy 72): quality −2, c2 decode −15%, c1 ≈, but c4 decode 37.2 in an arm the baseline never ran, and responsiveness +4. On OUR stack the 0rand combo is not a strict win; keep the baseline as the daily driver, keep the variant in tree for concurrency-hungry serving. Possible confounder: their run posts a single 14.8 s median turn at responsiveness 8 — an earlier bench build. Report: /tmp/runs/2026/08/2026-08-30T19-33-22.444571Z_4616f476.md.

MNBT 1024-vs-2048 assertion on the lab quant (2026-08-30/31 night)

Forum #124 ("1024 = quality-first") asserted. Our c1,2,4 run with MNBT 1024 + seqs 16 + GMU 0.89 + auto-KV (pool 1,051,792): quality 88, denominator 146 (harness dropped ~30 pts of scenarios), tg 27.6/27.0/30.4, TTFT d0 8.6 s, resp 24 (6.4 s). vs the same-protocol MNBT-2048 run one hour earlier: quality 90 (158/176, full denominator), tg 27.1/34.6/37.2, resp 25 (6.2 s). Verdict: at our protocol (reasoning_effort-low, temp 0, seed 42) 2048 wins on both quality percentage and decode; the 1024 advantage was measured under their thinking-max + temp-0.1 + trials-2 protocol. 0rand's #123 already noted "2048 batch + MTP4 ... loses quality (89 vs 93)" for THEIR protocol — different regime, same lesson: batch tuning is protocol-dependent. Variant stays in tree; record both. Reports: 2048 = ...19-33-22Z_4616f476.md, 1024 = ...22-04-48Z_4a70e6be.md.

DeepSeek-V4-Flash-Vision-Exp lane — campaign state (2026-08-31 night)

Day-0 port landed (sfxnz plugin 7a958128b8">@7a958128b8 vendored under docker-images/dsv4-flash-vision/, bases digest-pinned; sm121 build works, 35-40 min roundtrip). Lane verified TEXT-serving on ds4 (+NVMe/NAS weights):

  • eugr b12x base: 1024-token gen in 29.4 s = 34.9 tok/s with DSpark-6 (accept 0.714, length 2.55), text math exact, plugin loads ViT+aligner weights (whole checkpoint present, 82.28 GiB/node).
  • fixing their gaps so far: plugin init.py must ship the general- plugins register (raw.githubusercontent flaked on re-fetch — fixed via the contents API); wrapper class needs supports_multimodal = True (OpenAI layer refuses image_url otherwise — interfaces.supports_multimodal is a getattr check).
  • OPEN: OpenAI-image path raises "Model class ... has no registered multimodal processor" — the day-0 plugin never registers a _processor_factory (their shipped tests exercise the loader directly, not the API). Two paths: implement the BaseMultiModalProcessor inside our plugin (grid-tokens-per-image machinery exists in dsv4_vision.image_processor; fork interface = MultimodalRegistry. register_processor factory in multimodal/registry.py), or wait for the upstream repo to ship it. aiden-fork A/B build launched.

dsv4vision MM-into-OpenAI campaign — 2026-09-01 checkpoint

Built the full OpenAI multimodal stack in our plugin (processor trio, grid/sentinel tokens, embed_multimodal + embed_input_ids merge; verified in -image: embedding rows byte-exact vs aligner(ViT(patches))[perm]) and got the engine through profiling+warmup with mm registered. Two hard walls remain (before the first image_url token):

  1. The fork's V2 "always-embed" mm arm passes input_batch.input_ids[:n] — a wrong-rank slice that arrives as [B, T]-shaped 2-D ids (observed (17,17) for a 17-token prompt, (7,7) on DSpark-6 verify steps) — and downstream logits degenerate (all-<｜begin▁of▁sentence｜>).
  2. The AOT/dynamo trace of inputs_embeds freezes the dummy's 2-D shape and then asserts "wrong number of dimensions3 for op: input" on the native 3-D stream (tilelang mhc kernels are rank-strict both ways).

State: the whole mm branch is env-gated (VLLM_DSV4_MM=1), default OFF = the verified text-only lane — restored EXACT on the new per-node NVMe weights (~/dsv4vision-hf on dgx3+dgx4, 167 GiB each, recipe HF_HOME flipped from the NAS hub; shard load 7 min → <1 min, boot ~6 min end-to-end). Bridges done tonight for the next attempt: encoder-runner + speculator buffer shape-follow patches (patch_mm_hc_streams.py), placeholder `<｜deepseek_image｜>= in-vocab id 129264,get_placeholder_str` chat integration, AOT=0 knob for mm experiments. nid: fork A/B (aiden base) still pending its pip-layout patch; dspark_gate patch now auto-finds the fork's venv root.

dsv4vision tool-eval-bench (c1, 2026-09-01)

tool-eval-bench run --base-url http://dgx3.lan:30000/v1 --model deepseek-v4-flash-vision --seed 42 --hardmode --perf --concurrency 1 → report /private/tmp/runs/2026/09/2026-09-01T17-46-28.764372Z_89b18250.md:

recipe quality points decode tg t/s c1 (d0/d4096/d8192) pp2048 t/s TTFT deployability
dsv4-flash-vision (text lane, local-NVMe, DSpark-6) 81 143/176 — — — —

perf block: tg 34.1 / 31.0 / 29.1, pp2048 3,907 / 2,714 / 2,449, TTFT 1.9-4.9 s (spec), scenario-median turn 2.4 s, deployability 74 (best of any ds4 lane) — long-context capability is the raw 1M-context draw. 3 safety-critical failures (TC-42 extra-param, TC-43 empty required param, TC-60 sleeper injection) — the day-0 0731-style tool formatting is the gap vs the GLM lanes (thinking-pair JSON fencing was already a known day-0 weakness). Quality 81 < sparkrun 89 < lab 92.

New 2026-09-01 checkpoints: Intel W4A16-AutoRound + lab NVFP4-Spark

Checkpoints on the NAS hub (hf download'd):

  • Intel/GLM-5.3-Flash-W4A16-AutoRound rev 5eee1846f0 — AutoRound 0.15 W4A16 sym int4 g168, 99.84% of BF16 (GSM8K .9712/MMLU .8620), selfattn/hc/indexer/embed lm_head FP16 by design (their ignore_layers), pure GPTQ packing (qweight/qzeros/scales — qweight count 37152).
  • local-inference-lab/GLM-5.3-Flash-NVFP4-Spark rev 06972cdee7 — NVFP4 W4A4 whole network (in_proj_qkvbfg_a.weight_scale etc.) + NVFP4 MTP expert layer in separate model-mtp-0000x shards (lil.yaml: mtp_tokens 5, dflash2_tokens 7, default_speculator mtp, block 256, disable flashinfer autotune, INSTANTTENSOR env sizes; arch Glm5NextForConditionalGeneration + vision blocks in the index). This is the rtx6kpro r12-stack checkpoint (their doc: MTP:3 254 tok/s CC1 @ TP4, Sieve 333-403).

Recipes added: recipes/glm53-flash-intel-w4a16 (timothystewart6/vllm-gb10 v0.28.0-gb10.2 arm64, gptq_marlin, entrypoint declared — tini AssertionError taught us the image has ENTRYPOINT []+CMD bash), recipes/glm53-flash-nvfp4-spark (lab image). Boot outcomes:

  • Intel lane died in the fork's Transformers-backend weight loader: "no module or parameter named 'layers.0.self_attn.q_conv1d'" — the tjstewart image has NO native glm5_next arch (vllm archs list lacks it; transformers 5.16.1 has Glm5NextForConditionalGeneration) → needs the local-inference-lab fork (native glm5next) or a transformers name-shim mod.
  • NVFP4-Spark on our lab fork: KeyError on in_proj_qkvbfg_a.weight_scale at 80% shard load — whole-network NVFP4 W4A4 is beyond the R8-era lab fork loader (r12 stack territory).

⇒ THE unifying next step (user directive): build our own arm64/sm_121a image of local-inference-lab's fork — source contract from models/glm-5.3-flash.md (local-inference-lab/vllm dev/jovian-judgement + b12x master + 6/5 PRs; kk3 build rules apply: CUDA 13.0.88, sm_121f gencode rule for block-scaled FP4). RTX docs archive: /tmp/lil-docs (ds4-flash-v6, ds4dspark-v20-r33, infernal r21, pcie-autotune) for the DeepSeek quality track.

2026-09-02: r12 fork built + NVFP4-Spark serving on it (the day's win)

Built glm53-jovian-r12:local in-docker on dgx3 (docker-images/glm53-jovian-r12/): vllm 0.1.dev20489+gcc161e220 (local-inference-lab/vllm base 54f6e9826c + PR heads 550/552/553/554/555/557 merged in order — the doc's "composition commit" 7d8a09c is not a public ref) + b12x 1.3.0 (master 139e0405 + PR heads 260-270), on the day-0 vllm/vllm-openai:glm53-flash-arm64-cu130@905c02… base. Build fixes applied along the way: apt git, setuptools-rust, ninja/cmake, cu13 wheel include-header symlinks (cusparse.h), unversioned libnvrtc.so symlink for the deepgemm_C link step.

NVFP4-Spark boot on it (recipe glm53-flash-nvfp4-spark, retargeted):

  • whole-network NVFP4 W4A4 (incl. attention in_proj_qkvbfg scales) now LOADS — the r12 fork's modelopt handles it (our lab fork: KeyError).
  • Attention wall solved: the sm12x auto-priority list only offers [TRITON_MLA (no sparse), FLASHINFER_MLA_SPARSE_SM120 (head 576 only)] — neither fits GLM head-512. Fix = --attention-backend B12X (model hook in glm5next/nvidia/attention.py: _select_sparse_backend) for the main model, and "attention_backend":"B12X" INSIDE --speculative-config for the MTP draft (eagle/utils.py: _make_eagle_draft_vllm_config takes the draft's backend from the speculative config, not the global flag — user's hint).
  • Served: exact "391" verified, MTP-3 Glm5NextMTPModel resolved, KV fp8_ds_mla default, 2-node TP2. c1 bench running.
  • Intel W4A16 recipe retargeted to the same image (gptq_marlin + B12X attn).
  • patch_cuda_sm12x.py (B12X into sm12x priorities) staged in the image dir for the next rebuild — the image-baked version of the same fix.

2026-09-02 (cont): bench + clean rebuild per operator direction

  • NVFP4-Spark (r12 fork image, day-0 base) c1 bench: 88/100 (120/136, default 69-scenario suite; deployability 67, median turn 8.1s — TP2 GB10 is slower per-turn than the eugr vision lane's 2.2s on the same suite, also 88). Same-suite A/B vs vavrusa lane: 88 = 88. The lab's 92 was the 88-scenario/176-point suite — that invocation's re-run is pending.
  • Operator: docker_image_dir in the spark recipe pointed at the lab dir — fixed to glm53-jovian-r12 (intel recipe same).
  • Operator: "start with cuda image, solar-open2 pattern but r12-specific". Dockerfile/versions.env rewritten: raw nvidia/cuda:13.2.0-devel-ubuntu24.04 (a5b6256e…, solar-pinned) → apt stage (solar locks) → torch 2.13.0 cu130 + flashinfer 0.6.17 (whl index) → vllm wheel (fork base 54f6e9826c + PR-head merges 550-557, use_existing_torch, ccache) → b12x wheel (master + PR heads 260-270, pure-python) → runner installs both wheels + deps, bakes patch_cuda_sm12x.py, GB10 gate (vllm/b12x/_flashkda_C imports, B12X enum, patch presence). No day-0 vllm contamination; ENTRYPOINT [] CMD bash. Build running on dgx3 (tmux r12build → /tmp/r12build2.log), same tag glm53-jovian-r12:local (running serve keeps the old image; re-verify lane after build).

2026-09-02 (cont 2): lil — the lab's own launcher, read

local-inference-lab/lil = standalone Go launcher (their sm121 equivalent): reads each HF model repo's lil.yaml at runtime + embedded family policy + discovered topology (local or spark-rdma via SSH), renders/validates/runs a typed vLLM invocation. lil render <model> prints the exact env + command.

GLM family policy (configs/models/_bases.yaml) — the lab's canonical launch for these checkpoints, vs our spark recipe today:

  • --attention-backend B12X, --linear-backend b12x, --moe-backend b12x (we have only the global attention flag; linear/moe backends MISSING)
  • spec config: {"method":"mtp","num_speculative_tokens":5 (lil.yaml mtp_tokens), "moe_backend":, "attention_backend":"B12X"} — we use k=3 and no moe_backend. MTP:3 is the r12 x86-qualified mode; the checkpoint's manifest default is 5.
  • --load-format instanttensor (+INSTANTTENSOR_BUFFER_SIZE/CHUNK_SIZE env)
  • --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096, --max-model-len auto
  • --reasoning-parser glm45, --tool-call-parser glm47
  • DFlash2 (the GLM-side second speculator, method "dflash") needs the separate draft checkpoint local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8.

    • TODO: after the clean image lands, align the spark recipe to the lil policy (moe/linear backend b12x, spec moe_backend, instanttensor, mamba-cache-mode, parsers) and A/B k=3 vs k=5.

    Clean r12 build: fix log (builds 2-9)

    1. flashinfer: whl-index path dead → flashinfer-python 0.6.17 from PyPI (fork's setup.py skips flashinfer-cubin at build; runtime fetches cubins).
    2. git protocol v2 flake in-container → GIT_PROTOCOL=0 + protocol.version 0.
    3. setuptools: fork build needs setuptools-rust + setuptools-scm (uv hint).
    4. DeepGEMM FetchContent: pinned SHA on non-default nv_dev branch → GitHub 404s raw-SHA fetch → DEEPGEMM_SRC_DIR pre-clone stage (cmake escape hatch).
    5. PR 550 closed (unmerged) → refs/pull/550/head deleted; all PR heads now pinned by commit SHA (vllm 550-557, b12x 260-270). Raw-SHA fetches verified working.
    6. Operator rule: no build while a serve container runs (OOM risk on 128 GB unified) — spark lane stopped for the build window.

    2026-09-02 (cont 3): clean r12 image SHIPPED

    Image glm53-jovian-r12:local rebuilt from raw nvidia/cuda:13.2.0-devel (solar pattern, r12-specific content): 16.1 GB (vs 37.6 GB day-0-based), gates green (vllm 0.1.dev20489+g06f922207, b12x, _flashkda_C, B12X enum, sm121 patch), synced dgx3→dgx4. Boot fixes found on first clean boot:

    • tini "[FATAL] exec local-inference-lab/… failed" → clean image is ENTRYPOINT [] (solar contract), so the RECIPE now declares entrypoint = "vllm","serve".
    • FlashInfer JIT OOM: 12 parallel nvcc jobs (mxfp8_gemm_cutlass_sm120 etc.) get OOM-killed (code 137) while 93 GB weights + 10 GB KV are resident on 121 GB unified → MAX_JOBS=2 in recipe [env]; JIT (~15 min) then survives. Boot total ~35 min (NFS shards 16 min + JIT 15 min). c1 bench (default 69-scenario suite, same as the vavrusa run): 90/100 (124/138; quality 90, deployability 69, median 7.9 s) — vs 88 on the day-0-based image, same suite. Clean image is now the reference GLM lane.

    Remaining lab-alignment (next boot, one change-set at a time): --linear-backend b12x + --moe-backend b12x, spec moe_backend:b12x, A/B k=3 vs k=5 (lil default), --load-format instanttensor (+env), --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096, --reasoning-parser glm45, --tool-call-parser glm47. Then: Intel W4A16 boot on the clean image (queued, needs cluster).

    2026-09-02 (cont 4): lab-stack perf push — eugr build pipeline adopted

    Goal (user): 30 tok/s in --perf. Found the missing pieces:

    1. lil render reference (built lil from local-inference-lab/lil @ cb58d54, rendered with testdata manifest GLM-5.3-Flash-NVFP4-Spark + spark.yaml): our recipe flags now 100% match the family policy (_bases.yaml glm: b12x linear/moe, B12X attention, instanttensor, block 256, seqs 8, batched 4096, glm45/glm47, mamba align) + spec {mtp:5, moe_backend b12x, B12X} + env VLLM_USE_AOT_COMPILE=1 / V2_MODEL_RUNNER=1 / FLASHINFER_SAMPLER=1 / BREAKABLE_CUDAGRAPH=0. Boot 35m -> ~7m (instanttensor loader, user: keep). Perf after engine switch: d0 c1 24.2 / c4 31.8 tok/s (was 10.3/16.5). Quality dropped with full lil set (16/22 at TC-22) -> bisection in progress; user: instanttensor is fine, problem elsewhere.
    2. eugr/spark-vllm-docker (the lab's actual build repo, main): 804-line Dockerfile, CUDA 13.0.2 base, 12 build-time vllm patches (docker/patchvllm*.py), b12x from source (lukealonso/b12x; lab: local-inference-lab/b12x master), nightly DockerHub images eugr/spark-vllm[-b12x]:latest (stale 08-23, user: dockerhub is the wrong source — build from source). Ported all 12 patches into docker-images/glm53-jovian-r12/patches-eugr/
      • Dockerfile (VLLM_PRESERVE_SM12X_TARGET=1, VLLM_PATCH_B12X_C128A_ALIGNMENT=1; patches are idempotent, several no-op on our r12 ref). Build blocker: node github.com access flaky ("could not read Username" on FetchContent clones) -> pre-fetched all 9 sub-deps on the Mac into deps-src.tar.gz (triton_kernels@v3.5.1, MSA, FlashMLA, FlashKDA, flash-attention, qutlass@e74319e, tml-fa4@b206834, cutlass@v4.4.2) + the SRC_DIR env escapes each cmake documents. Build 14 compiling.