launcher/history/night-shifts/2026-10-23.md

Night shift 2026-10-22 → 23 — report

Stand dgx5–8 (production dgx1–4 untouched all night). Plan: 2026-10-22-night-plan.md.

1. The headline: the full GLM-5.3 works, and speculation is what makes it usable

glm53-mxfp4-fp8-4x-r39 — RedHatAI/GLM-5.3-MXFP4xFP8 (740B total / ~40B active, 78 layers, DSA, 1M native window) on four Sparks at TP4.

	decode c1	decode c2 (agg)	score (v2.8.0)	median turn
first boot, no speculation	19.0 t/s	—	88/100	8.8 s
MTP k=3	33.4–34.9 t/s	47.6 t/s	—	—
MTP k=5	33.1–37.3 t/s	47.9 t/s	—	—
MTP k=3 @ 131072 ctx	~31 t/s (benchy)	36.5 t/s	90/100 (175/194)	5.7 s
reference: GLM-5.3-Flash 2x	52 t/s	51	95/100	3.6 s
reference: production 1017 4x (SGLang EP2)	48–51 t/s	64–68	86–87/100	1.4 s

So the biggest model we can run now scores 90/100 with 5.7 s turns — better quality than production, on four nodes nobody else uses, at ~60% of prod's decode speed. k=5 buys nothing over k=3: drafts are accepted 87–93% of the time (Mean acceptance length 4.36–4.66 of 5), so verification, not the ladder, is the limit.

Boot chain (each failure cost a 30-minute reload; all recorded in the recipe header):

1. kindling spark-glm53:v10 — multi-node worker init dies (is_in_the_same_node: gloo/tcp), the fork is TP2-shaped.
2. r39 + --attention-backend B12X — TypeError: DeepseekV32B12xAttention got an unexpected keyword 'index_group_builder' (the re-cut indexer API is newer than r39's class).
3. r39 + --linear-backend marlin — prepare_fp8_layer_for_marlin trips on the ReplicatedLinear modules the re-cut puts fp8_block on.
4. r39 default dispatch — mm_fp4 does not support backend 'cute-dsl' with capability 121.
5. VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel — MXFP4 linears fall to Marlin, fp8_block keeps its own backend. Boots.

Karmic A/B (spark-glm53-karmic:local, the local-inference-lab fork): native B12X_MXFP4_MXFP8 MoE experts vs r39's MarlinExperts = 19.2 vs 19.0 t/s. At 99 GiB of weights per rank decode is bandwidth-bound; a faster GEMM is not the lever, speculation is.

2. Two hard limits found by measurement

DCP4 does not exist for this checkpoint. Explicitly selecting the backend answers why:

ValueError: Selected backend AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120
is not valid for this configuration. Reason: ['DCP not supported']

Auto-selection finds no MLA backend at all once DCP is on; same on both forks, block 64 and 256. The DSA/sparse-MLA path has no context-parallel implementation in our images, so the 512K/1M windows of the Flash sibling are out of reach. The window is therefore KV-pin-bound only (52.7 KB/token/rank; 146,944 tokens at a 7.5 GiB pin).

And the KV window is not the usable prompt length. The needle ladder:

prompt	result
40,269 tok	✅ correct, TTFT ~85 s
~120K tok	❌ RPC call to sample_tokens timed out (even with VLLM_RPC_TIMEOUT=3600) → EngineDead, and dgx6 + dgx7 wedged (ping alive, sshd starved)
~150K tok (earlier, 196K window)	❌ same, plus the same wedge

Recovery both times: HA plug cycle of the two nodes (off 35 s → on), back in ~1 min, no data loss (weights live on the NAS). The cause is not the KV pool: after 98.86 GiB of weights a Spark has ~22 GiB, and a long prefill's scratch does not fit — smaller chunks and a bigger RPC budget do not help.

Practical envelope for this lane: ≤64K prompts (the 131072 window buys queueing headroom, not a 128K prompt); 8K→64K ladders only. Real 128K+ needs TP8, i.e. halving the weights per rank.

3. FLARE — works, and is honest about where it stands

NVIDIA published NVIDIA/flare-local-inference on 2026-10-20; the full procedure and the six failure modes are in specs/tech-architecture/flare-bringup.md. The production checkpoint (deepseek-ai/DeepSeek-V4.1-Flash-1017, pinned revision) prepared at TP4 across dgx5–8 and served by the Rust runtime: 1M context advertised, 24 GiB KV per rank (28.9M tokens), two RoCE rails per peer. First boot: prefill ~1,430–1,700 t/s, decode 19.2 t/s c1 / 29.6 c2 — ~2.4× slower than our SGLang EP2 production lane on the same model, with nothing tuned. Their upstream moved 12+ commits during the night, including "Preset DeepSeek V4.1 packed GPU speculation" and adaptive verification scoring; that is the obvious next pass.

The agents were stopped and disabled afterwards (unit files kept), and a check confirmed nothing returns at boot.

4. Dashboard / add-on: five releases, one honest post-mortem

1.3.4 → 1.3.9. Features: the recipes panel's Model column now has the same hover affordances as the recipe name — ⧉ copies the model id, HF opens the checkpoint page on the Hub (only for org/name ids) — plus an inline SVG favicon on both pages, a build-version badge under the dashboard title, and a webui entry so the HA add-on page has an "OPEN WEB UI" button.

The release chain itself is worth remembering:

version	what went wrong
1.3.4	built from a launcher commit that never reached gitea (a chained git push -q failed silently)
1.3.5	rebuild: buttons + icon land, no badge yet
1.3.6	badge present but shows vdev — a Docker ARG never reaches the compiler
1.3.7	ENV in the builder — still dev: the warm cargo cache (target/ on a cache mount) served the old literal
1.3.8	cargo:rerun-if-env-changed — still dev (the cached fingerprint wins)
1.3.9	the version is generated by build.rs into OUT_DIR, and the CI log prints the value it baked

Verified in the published image, not just in source: docker run … sm121 web …; curl /api/state → {"version":"1.3.9", …}.

5. Research (the night's reading list)

• RedHatAI/GLM-5.3-speculator.dspark — 2.5 GB, 3-layer Qwen3 draft fed by hidden states from target layers 2/20/39/58/75, up to 8 drafts, 1M context, 41.16% acceptance / 4.29 accepted length at 524K–1M. Downloaded; on r39 it passes config and draft-backend selection, then dies in qwen3_dflash._build_context_kv_buffers — the speculator targets vLLM 0.32 / speculators 0.7.x. Retry belongs on a vLLM-main image (eugr nightly, already staged on two nodes).
• vLLM 0.32.0 released; RedHatAI/GLM-5.3-MXFP4xFP8 re-cut again on 22.10 14:05 (150 files).
• tunamitom/b12x@76b76347 — MXFP4-A4 prefill for MoE (CuTe mxfp4_mma_m16n8k64_f32_e2m1_ue8m0, dynamic E8M0 activation scales, decode stays exact W4A16, switch at B12X_W4A16_MXFP4_PREFILL_MIN_TOKENS=512), reported ~+10% prefill and "should work for Flash too". Directly relevant to our prefill pain → todo #23.
• madeby561/vllm:mimo-v26-flash-b12x-20260923-rc4 (12.9 GB) and its opus55/kk-winner siblings → todo #22. Caveat: the tag is amd64-only, so a Spark needs an arm64 manifest or a source build.
• FLARE, sparkring (sparkring, sparkcache, net-sircl, glm53-flash-tp4-spark), local-inference-lab blackwell-llm-docker (92★) and the TensorFold family all moved; nothing newer than what we already track.

6. State of the fleet in the morning

dgx5–8: free (four nodes, no containers, all online). dgx1–4: production 1017 untouched. Add-on: 1.3.9 published (buttons, icon, version badge, OPEN WEB UI button). All work committed and pushed: launcher 60ffdda, recipes 2b7b4c7, add-on c6fa22f.

7. What is worth doing next

1. DSpark on a vLLM-main image (eugr nightly staged): the 2.5 GB speculator is the single biggest remaining lever on the full GLM-5.3 (MTP already took 19 → 35 t/s; a real drafter with 4.3 accepted tokens should go further).
2. b12x A4 prefill (todo #23): +10% prefill claimed, and prefill is exactly what limits our usable prompt length.
3. FLARE tuning pass on 1017 (--scheduler-policy throughput, --max-concurrency, adaptive verification) to see whether the Rust runtime can close the 2.4× decode gap on our own production model.
4. Only then the quality question: is 90/100 on four otherwise-idle nodes worth a production slot? The Flash 2x lane still leads at 95/100 on two nodes.