launcher/history/night-shifts/2026-09-05.md

Night shift 2026-09-05/06 — r18 campaign (GLM-5.3-Flash)

Full lane matrix: re-baselined anchor, r18 build (base a8c796f3 + #582 + #646, tree 67e7e6e), 2x and 4x cells, RedHat/eugr model A/B. All benches: tool-eval-bench 2.6.1.dev45+gcf54b4bfe, seed 42, hardmode (88 scenarios), --perf (llama-benchy pp2048/tg128, runs=3, c1+c2). Score = /176 weighted points; /100 = quality score; dep = deployability (α=0.7).

Combined table (score + perf)

Lane	image	score	/100	dep	KV tokens	pp c1	tg c1	TTFT c1	pp c2	tg c2	TTFT c2	tg c2 @8k-ctx	verdict
r17abi 2x (anchor)	r17 (glm53-jovian-r12)	153	87	69	~1.25M	1,577	29.0	1,362ms	1,595	41.7	2,093ms	29.7	re-baseline
r18 2x (ds4)	r18	157	89	71	~1.25M	1,473	30.7	1,481ms	1,374	40.2	2,268ms	30.6	PASS, 2x production candidate
RedHat MTP k=5 2x	lab	156	89	70	~1.25M	1,340	26.8	1,638ms	460	22.4	4,667ms	14.7	quality PASS, perf FAIL at c2
eugr MTP k=5 2x	eugr b12x	143	81	65	~1.25M	1,292	31.1	2,397ms	1,388	41.6	2,751ms	17.7	FAIL (−10 vs anchor)
r18 4x RoCE	r18	156	89	76	1.47M	2,904	47.7	762ms	2,413	70.9	1,399ms	35.9	PASS, throughput king
r18 4x DCP4	r18	161	91	75	4.84M	1,639	43.1	1,521ms	1,756	51.1	1,637ms	30.0	BEST score, prefill tax
r18 4x nvfp4 KV	r18	161	91	77	2.35M	2,614	45.3	873ms	2,250	62.6	1,368ms	35.6	PASS, best deployability
r18 2x nvfp4 KV (10G)	r18	154	88	70	2.14M	(bench: see report)	—	—	—	—	—	—	PASS, no regression vs 157
r18 2x nvfp4 KV + Spark ckpt	r18	158	90	70	2.14M	(bench: see report)	—	—	—	—	—	—	BEST 2x
r18 4x nvfp4 KV + Spark ckpt (16G)	r18	—	—	—	—	—	—	—	—	—	—	—	BLOCKED: mm_mxfp8 not TP4-tileable
r18 4x DFlash2 k=7	r18	—	—	—	—	—	—	—	—	—	—	—	BLOCKED (fork C4-page assert)
dsv4-flash-vision eugr 2x	eugr b12x	—	—	—	—	—	—	—	—	—	—	—	BLOCKED (no V4-Vision in image)

pp/tg = tokens/s (prefill 2048 / generate 128); TTFT = ms; d0 = no extra context, @8k-ctx = 8192 tokens of history.

Reading the table

• 4x RoCE = the all-round production candidate: best prefill (2.9k, 2× 2x), best c2 decode (70.9), half the 2x TTFT, quality 156 = 2x r18.
• 4x DCP4 = capacity + quality play: 4.84M KV (3.3× plain 4x at the same 10GiB pin — the sparkring steal) and the best score (161), but the DCP ring exchange costs prefill (1.6k vs 2.9k).
• 4x nvfp4 KV ≈ RoCE perf, +1.59× KV, best deployability (77). Next cell: DCP4 + nvfp4 KV combined (~7.7M KV projected).
• RedHat MTP k=5: quality fine, but the lab image's MTP path collapses under concurrency (pp c2 460 t/s, TTFT 4.7s) — dropped from the candidate set once perf was on the table.
• eugr: no edge anywhere (quality −10, decode degrades at depth); 4x cell skipped. Lanes demoted.
• nvfp4 KV at 2x: no quality regression (154 vs 157 non-Spark, within noise) + 1.7× KV capacity (2.14M @ 10G) — adopt.
• Spark checkpoint on r18: +1 over non-Spark (158, 90/100) — the best 2x lane of the campaign; 4x blocked by the mm_mxfp8 TP4 constraint.
• 4x cells now pinned at 16 GiB KV (NVFP4 4x headroom; 2x stays 10).
• JIT persistence live: cold serve ~45 min → 9 min warm (per-node caches).
• r18 2x beats the anchor on both axes (157/89 vs 153/87) — 2x default candidate pending operator sign-off.

Blocked lanes (documented in recipe headers)

• DFlash2 k=7 drafter: drafter inherits the engine's FP8 KV dtype; B12X and FLASH_ATTN both reject it on SM121 ("FP8 KV cache requires FA3 on SM90 or FA4 on SM100"); with drafter kv_cache_dtype=auto the fork's own assertion fires ("GLM MLA parent-page stride must be an exact number of C4 pages"). Needs a fork-side fix.
• dsv4-flash-vision eugr: the eugr 0904 image's DeepseekV4ForCausalLM has no aligner module (predates V4-Vision support); the Vision-Exp vision weights fail to load. Fork lane remains the only V4-Vision path.

Ops log (condensed)

• Re-baseline (queue 1a) first, on the current r17 image, before any r18 gate — all comparisons above use the 153/176 anchor.
• r18 build on dgx4: 5 attempts. Root causes: transient apt 503s; kv-cache-cleanup patch anchor replaced upstream (MemorySnapshot flow — patch now skips cleanly); staged-deps submodule stubs — both flash-attention/csrc/cutlass and MSA/python/fmha_sm100/cutlass were empty dirs in deps-src.tar.gz (r18 tree dropped implicit cutlass include wiring; MSA install needs the dir) — populated with each project's own pin (nvidia/cutlass 62750a2b / eb61c911), 23k macOS AppleDouble files purged.
• DCP4 requires --decode-context-parallel-size 4 + --cp-kv-cache-interleave-size 4 (GLM C4 DCP); B12X is not DCP-compatible (engine falls to FLASH_ATTN for that cell).
• #646 geometry verified live: "attention block size 2304" = the 2048+256 split in the serve log.
• Image separation (operator direction): docker-images/glm53-jovian-r18/
  • glm53-jovian-r18:local tag; r12 dir restored to r17 state (git checkout 582343c); r12 tag removed from nodes (points at nothing until an honest r17 rebuild — r17-era deps staging unrecoverable, see state doc).
• 4x recipes need tensor_parallel_size = 4 (world size must cover nnodes); 2x→4x port of the r17abi recipe is the r18-4x family.

Reports

Full per-lane reports: recipes/<lane>/reports/ (r17abi: 2026-09-06_r18-rebaseline-r17abi-2x.md, 2026-09-06_r18-image-ds4.md; redhat/eugr: 2026-09-06_*.md; r18-4x family: 2026-09-06_r18-4x*.md). Analysis + open decisions: specs/tech-architecture/glm53-recipe-state.md → "Night-shift results".

Night-list revisit (2026-09-06, operator request): honest statuses

Missing from the night matrix — aiden community-image lanes (operator is right: no aiden image ran in this campaign; the historical aiden scores came from the pool-176 era and must not be compared with the 88-scenario hardmode anchors above). Tasked for the next window:

• aidendle94/sparkrun-vllm-glm53-gb10:production-1.0/1.1 — GLM-5.3 aiden stack (torch 2.12 / flashinfer 0.6.15 lineage, on all 4 nodes), recipes: glm53-flash-aiden-2x, -4x, -4x-labdrafter, glm53-full-nvfp4-spark-aiden.
• NEW tag to investigate: aidendle94/sparkrun-vllm-ds4-gb10:production-3.73-vision (pushed 2026-09-06): DeepSeek-V4-Flash 2x-Spark serving (entrypoint deepseek-v4-flash-2x-spark-entrypoint, pinned launcher serve-ds4-flash-spark.sh, SPARK_RECIPE_REF 8b00816d), pins: torch 2.12.0 / flashinfer 0.6.15 @ ce40d25a / NCCL @ 5067397c / DeepGEMM a6b593d, model deepseek-ai/DeepSeek-V4-Flash(-Vision) rev-pinned, SPARK_KV_CACHE = 26214034944 B ≈ 24.4 GiB @ GMU 0.82 (independent confirmation of the 24 GiB/rank headroom theory), CUDAGRAPH_MODE switch (FULL_AND_PIECEWISE). Where "vision" fits (operator confirmation): this is the DeepSeek-V4-Flash-Vision** aiden lane** — a direct competitor to our fork lane dsv4-flash-vision (DSpark-6) and the eugr image that FAILED to carry V4-Vision at all. The 'vision' branch effectively replaces the blocked dsv4-flash-vision-eugr row with an actionable one. Existing recipes dsv4-vision-aiden-2x/4x point at this family already.

Status ledger (main matrix above): DONE = r17abi baseline, r18 2x/4x/DCP4/ nvfp4KV, RedHat MTP, eugr 2x(FAIL), spark-2x-nvfp4kv (158, best 2x); BLOCKED = dflash2 drafter (r18 fork C4 assert), dsv4-vision-eugr (no aligner in eugr img), 4x Spark ckpt (mm_mxfp8 TP4); NEW QUEUE = aiden glm53 lanes re-bench (same-basis), prod-3.73-vision investigation + boot on ds4.

2026-09-06 night, part 2 (operator: "switched back to pluto", dgx1-4 free)

Dual-bench window: both lanes benched in parallel, seed 42 hardmode + perf, tool-eval-bench 2.6.1.dev46 (±1 pt version drift vs the dev45 anchors).

lane	nodes	score	dep	pp c1 d0	tg c2 d0	TTFT d0 c1	KV tokens
aiden production-1.2 (DFlash2 k=7, eager, fp8 KV)	2x 2nodes-dac	159 (90)	—	1,271	30.1	1,802ms	461,730
r27 (2x spark, nvfp4 KV, MTP)	2x ds4	156 (88)	—	1,356	31.4	1,535ms	2,206,618

Readings:

• aiden = first-ever canonical measurement of the community GLM-5.3 lane (159/176, new 2x quality record by +1 over r18-spark 158 — within noise, not a clear win; DFlash2 acceptance 4.27–4.94 healthy). The drafter lane is NOT dead: it was the r18 fork's C4-page assert, not the drafter. aiden profile costs: 5x smaller KV, eager decode, higher TTFT.
• r27 engine not yet at parity: same 2x config as r18-spark scores 156 vs 158 (−2) and perf is close but slower in decode (31.4 vs 36.0 t/s c2). r27 is the tree to keep qualifying (DCP4/DCP-16GiB + DFlash2 re-test still the high-value targets); the r18 image stays the 2x reference until r27 closes the gap.
• r27 image: attempt-4 rebuild (explicit TORCH_CUDA_ARCH_LIST=12.1a in the vllm-builder stage — the attempt-3 build lost the stage-inherited env and shipped sm120-only kernels → cudaErrorNoKernelImageForDevice).

Next queue: prod-3.73-vision (DSV4F-Vision aiden lane) on 2nodes-dac.

Night 2 (2026-09-06→07): aiden + r27 dual bench (pluto switch, dgx1-4 free)

Tool: 2.6.1.dev46+g9d412464a (campaign upgrade from dev45 — same version both runs, cross-lane comparison valid).

Lane	Score	dep	KV tokens	pp c1 d0/d8k	tg c1 d0/d8k	tg c2 d0	TTFT c1 d0
r18-2x-nvfp4kv-spark (anchor)	158 (90)	70	2,136,194	1,861/1,718	24.5/22.5	36.0	1,236ms
aiden prod-1.2 2x (DFlash2 k=7, fp8 KV 4GiB, eager)	159 (90)	—	461,730	1,271/1,669	23.5/22.8	30.1	1,802ms
r27 2x (spark ckpt, nvfp4 KV)	156 (88)	—	2,206,618	1,356/1,682	22.7/22.9	31.4	1,535ms

aiden prod-1.2 = first canonical aiden measurement ever: 159/176 (90.4), 74p/11partial/3fail. DFlash2 acceptance 4.27–4.94 (healthy, not the <0.15 broken-aux signature). Fails: TC-43/58/61. Profile: eager + 4 GiB KV pin → KV pool 461K (small), prefill 1.2–1.7K t/s, decode ~23 t/s. The aiden lane ties/beats the r18-spark anchor by a point in quality but at much smaller context headroom — a real community-stack datapoint, not a replacement.

r27 lane (r27 engine, same 2x spark config as the 158 anchor): 156/176 — −2 vs r18 on this lane. r27 prefill is ~27% slower (1,356 vs 1,861 pp c1), decode in line. The r27 tree has NOT beaten the r18 build on quality yet; the remaining r27 value = the qualified DFlash2 drafter + DCP4 + 16 GiB KV cells (next windows). r27 image: attempt-4 rebuild with explicit TORCH_CUDA_ARCH_LIST=12.1a in vllm-builder (attempt-3 lost the stage inherited env → sm120-only kernels → cudaErrorNoKernelImageForDevice).

Next: prod-3.73-vision (DSV4F-Vision aiden lane) investigation.

Vision lane: prod-3.73-vision (2026-09-07, 2x 2nodes-dac)

161/176 (91.5) — 74p/13partial/1fail; new 2x campaign record (ties the best 4x cells: r18 4x DCP4 / nvfp4KV 161). Model DeepSeek-V4-Flash-Vision-Exp (157 GB, cached, rev 6821d6ad), image aidendle94/sparkrun-vllm-ds4-gb10:production-3.73-vision (vLLM 0.1.dev19023+g30038602b — aiden's ds4-vision 0.20 fork: B12X MLA sparse, b12x MoE/linear, instanttensor, FULL_AND_PIECEWISE graphs, async scheduling, deepseek_v4 parsers). KV 1,442,886 tokens (fp8, 24.4 GiB cap), GMU 0.80, no spec (MTP/DSpark off).

Perf (llama-benchy pp2048 tg128): c1 1,501/25.4 (d0) → 1,863/25.3 (d8k); c2 1,766/37.0 → 2,030/21.5. TTFT d0 2.5s c1 / 1.75s c2. Fails: TC-38 only; partials: 13 (TC-14/46/50/51/53/57/58/61/62/63/68/82/88).

Ops notes (new capabilities added to the launcher for entrypoint-driven images): recipe model = "" now skips the positional model (image's own entrypoint selects it via MODEL_PATH env); docker path now injects NODE_RANK + MASTER_ADDR env (their launcher requires them; CLI flags go through "$@" and win for the standard flags). First boot failed with PP=2 (launcher default pp=nnodes) → the ds4-vision fork asserts TP×PP against device count → recipe pins pipeline_parallel_size = 1.

r27 4x DCP4 (2026-09-07, all 4 nodes, 16 GiB KV)

155/176 (88.1) — 72p/11partial/5fail. −6 vs the r18 4x DCP4 161. KV 7,738,490 tokens (16 GiB pin, as projected; vs 4.84M at 10 GiB on r18). Perf: pp c1 d0 1,713 (d8k 2,013), tg c1 44.0 / c2 53.0 @d0, TTFT 1.4s. Decode is fast; quality is not: the r27 engine trails r18 on the DCP4 cell too (2x: −2; 4x DCP4: −6). Fails: TC-38/43/45/51/80. Verdict so far: r27 = the tree for the future (DCP4 headroom 7.7M, DFlash2 qualified upstream), but the r18 image remains the quality reference for GLM-5.3 2x/4x until r27 closes the gap.

r27 DFlash2 (2026-09-07) — BLOCKED (tree-level, root-caused)

First boot attempt: drafter inherited the main model's fp8_ds_mla KV → "No valid attention backend" (FP8 KV needs FA3/SM90 or FA4/SM100; SM121 has neither) → fixed with kv_cache_dtype: "auto" (bf16) in the speculative config. Second boot: ValueError: GLM MLA parent-page stride must be an exact number of C4 pages (glm5next/nvidia/pooled_indexer.py) — the same assert family that blocked r18 dflash2, now confirmed in the r27 tree: with the drafter pooled into the main model's pages the parent stride breaks the C4 index-page alignment. Not an r18 artifact; r27's upstream DFlash2 qualification was sm_120-only (the pooled indexer is an SM12x path). Verdict: DFlash2 on GB10 needs a fork-level patch before it can run; the lane recipe is kept with the blocker documented (draft_model/FLASH_ATTN r18 style and method-dflash/auto both dead-end on the same geometry).

Aiden 4x (2026-09-07, 4nodes-switch, production-1.2+instanttensor)

163/176 (92.6) — 79p/5partial/4fail. NEW CAMPAIGN RECORD (beats r18 4x DCP4 161, vision-373 161, aiden 2x 159). zai-org FP8 (306 G), TP=4, MTP k=5, fp8_e4m3 KV 2,322,623 tokens @ 16 GiB pin, eager. The FP8-MLA+DFlash2 combo stays broken (recipe ships MTP instead, per aiden's guide). Fails: TC-43/51/58/85. Perf run was noisy (one pp cell measured 0; tg 26–42 t/s) — treat the quality as the headline, the perf as indicative only. Ops: the hub production-1.2 image lacks instanttensor → extended via docker-images/glm53-flash-aiden/ (FROM production-1.2 + instanttensor 0.1.9 --no-deps), built on dgx4, on all 4 nodes; the launcher's image sync re-copied it per node (docker load re-stamps the config digest).

Eager A/B (2026-09-07, r18 2x nvfp4kv-spark + --enforce-eager)

159/176 (90.4) vs 158 (90) with graphs — +1, within run-to-run noise. Perf ≈ parity: pp c1 d0 1,856 (anchor 1,861), tg c1 27.1 (24.5), tg c2 36.8 (36.0), TTFT 1,165ms (1,236). Verdict: the "eager improves quality" rumor does not hold on the r18 GB10 stack — eager is a neutral alternative (graphs stay the default; no reason to pay the ~0.6 GiB + capture time savings either way). Fails: TC-43/45/57/76-class set (4 fails, 9 partials).