2026-09-05.md 7.6 KB
Night shift 2026-09-05/06 — r18 campaign (GLM-5.3-Flash)
Full lane matrix: re-baselined anchor, r18 build (base a8c796f3 + #582 + #646,
tree 67e7e6e), 2x and 4x cells, RedHat/eugr model A/B. All benches:
tool-eval-bench 2.6.1.dev45+gcf54b4bfe, seed 42, hardmode (88 scenarios),
--perf (llama-benchy pp2048/tg128, runs=3, c1+c2). Score = /176 weighted
points; /100 = quality score; dep = deployability (α=0.7).
Combined table (score + perf)
| Lane | image | score | /100 | dep | KV tokens | pp c1 | tg c1 | TTFT c1 | pp c2 | tg c2 | TTFT c2 | tg c2 @8k-ctx | verdict |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| r17abi 2x (anchor) | r17 (glm53-jovian-r12) |
153 | 87 | 69 | ~1.25M | 1,577 | 29.0 | 1,362ms | 1,595 | 41.7 | 2,093ms | 29.7 | re-baseline |
| r18 2x (ds4) | r18 | 157 | 89 | 71 | ~1.25M | 1,473 | 30.7 | 1,481ms | 1,374 | 40.2 | 2,268ms | 30.6 | PASS, 2x production candidate |
| RedHat MTP k=5 2x | lab | 156 | 89 | 70 | ~1.25M | 1,340 | 26.8 | 1,638ms | 460 | 22.4 | 4,667ms | 14.7 | quality PASS, perf FAIL at c2 |
| eugr MTP k=5 2x | eugr b12x | 143 | 81 | 65 | ~1.25M | 1,292 | 31.1 | 2,397ms | 1,388 | 41.6 | 2,751ms | 17.7 | FAIL (−10 vs anchor) |
| r18 4x RoCE | r18 | 156 | 89 | 76 | 1.47M | 2,904 | 47.7 | 762ms | 2,413 | 70.9 | 1,399ms | 35.9 | PASS, throughput king |
| r18 4x DCP4 | r18 | 161 | 91 | 75 | 4.84M | 1,639 | 43.1 | 1,521ms | 1,756 | 51.1 | 1,637ms | 30.0 | BEST score, prefill tax |
| r18 4x nvfp4 KV | r18 | 161 | 91 | 77 | 2.35M | 2,614 | 45.3 | 873ms | 2,250 | 62.6 | 1,368ms | 35.6 | PASS, best deployability |
| r18 2x nvfp4 KV (10G) | r18 | 154 | 88 | 70 | 2.14M | (bench: see report) | — | — | — | — | — | — | PASS, no regression vs 157 |
| r18 2x nvfp4 KV + Spark ckpt | r18 | 158 | 90 | 70 | 2.14M | (bench: see report) | — | — | — | — | — | — | BEST 2x |
| r18 4x nvfp4 KV + Spark ckpt (16G) | r18 | — | — | — | — | — | — | — | — | — | — | — | BLOCKED: mm_mxfp8 not TP4-tileable |
| r18 4x DFlash2 k=7 | r18 | — | — | — | — | — | — | — | — | — | — | — | BLOCKED (fork C4-page assert) |
| dsv4-flash-vision eugr 2x | eugr b12x | — | — | — | — | — | — | — | — | — | — | — | BLOCKED (no V4-Vision in image) |
pp/tg = tokens/s (prefill 2048 / generate 128); TTFT = ms; d0 = no extra context, @8k-ctx = 8192 tokens of history.
Reading the table
- 4x RoCE = the all-round production candidate: best prefill (2.9k, 2× 2x), best c2 decode (70.9), half the 2x TTFT, quality 156 = 2x r18.
- 4x DCP4 = capacity + quality play: 4.84M KV (3.3× plain 4x at the same 10GiB pin — the sparkring steal) and the best score (161), but the DCP ring exchange costs prefill (1.6k vs 2.9k).
- 4x nvfp4 KV ≈ RoCE perf, +1.59× KV, best deployability (77). Next cell: DCP4 + nvfp4 KV combined (~7.7M KV projected).
- RedHat MTP k=5: quality fine, but the lab image's MTP path collapses under concurrency (pp c2 460 t/s, TTFT 4.7s) — dropped from the candidate set once perf was on the table.
- eugr: no edge anywhere (quality −10, decode degrades at depth); 4x cell skipped. Lanes demoted.
- nvfp4 KV at 2x: no quality regression (154 vs 157 non-Spark, within noise) + 1.7× KV capacity (2.14M @ 10G) — adopt.
- Spark checkpoint on r18: +1 over non-Spark (158, 90/100) — the best 2x lane of the campaign; 4x blocked by the mm_mxfp8 TP4 constraint.
- 4x cells now pinned at 16 GiB KV (NVFP4 4x headroom; 2x stays 10).
- JIT persistence live: cold serve ~45 min → 9 min warm (per-node caches).
- r18 2x beats the anchor on both axes (157/89 vs 153/87) — 2x default candidate pending operator sign-off.
Blocked lanes (documented in recipe headers)
- DFlash2 k=7 drafter: drafter inherits the engine's FP8 KV dtype; B12X
and FLASH_ATTN both reject it on SM121 ("FP8 KV cache requires FA3 on SM90
or FA4 on SM100"); with drafter
kv_cache_dtype=autothe fork's own assertion fires ("GLM MLA parent-page stride must be an exact number of C4 pages"). Needs a fork-side fix. - dsv4-flash-vision eugr: the eugr 0904 image's
DeepseekV4ForCausalLMhas noalignermodule (predates V4-Vision support); the Vision-Exp vision weights fail to load. Fork lane remains the only V4-Vision path.
Ops log (condensed)
- Re-baseline (queue 1a) first, on the current r17 image, before any r18 gate — all comparisons above use the 153/176 anchor.
- r18 build on dgx4: 5 attempts. Root causes: transient apt 503s;
kv-cache-cleanup patch anchor replaced upstream (MemorySnapshot flow —
patch now skips cleanly); staged-deps submodule stubs — both
flash-attention/csrc/cutlassandMSA/python/fmha_sm100/cutlasswere empty dirs indeps-src.tar.gz(r18 tree dropped implicit cutlass include wiring; MSA install needs the dir) — populated with each project's own pin (nvidia/cutlass62750a2b/eb61c911), 23k macOS AppleDouble files purged. - DCP4 requires
--decode-context-parallel-size 4+--cp-kv-cache-interleave-size 4(GLM C4 DCP); B12X is not DCP-compatible (engine falls to FLASH_ATTN for that cell). - #646 geometry verified live: "attention block size 2304" = the 2048+256 split in the serve log.
- Image separation (operator direction):
docker-images/glm53-jovian-r18/glm53-jovian-r18:localtag; r12 dir restored to r17 state (git checkout 582343c); r12 tag removed from nodes (points at nothing until an honest r17 rebuild — r17-era deps staging unrecoverable, see state doc).
- 4x recipes need
tensor_parallel_size = 4(world size must cover nnodes); 2x→4x port of the r17abi recipe is the r18-4x family.
Reports
Full per-lane reports: recipes/<lane>/reports/ (r17abi: 2026-09-06_r18-rebaseline-r17abi-2x.md,
2026-09-06_r18-image-ds4.md; redhat/eugr: 2026-09-06_*.md; r18-4x family:
2026-09-06_r18-4x*.md). Analysis + open decisions:
specs/tech-architecture/glm53-recipe-state.md → "Night-shift results".
Night-list revisit (2026-09-06, operator request): honest statuses
Missing from the night matrix — aiden community-image lanes (operator is right: no aiden image ran in this campaign; the historical aiden scores came from the pool-176 era and must not be compared with the 88-scenario hardmode anchors above). Tasked for the next window:
aidendle94/sparkrun-vllm-glm53-gb10:production-1.0/1.1— GLM-5.3 aiden stack (torch 2.12 / flashinfer 0.6.15 lineage, on all 4 nodes), recipes:glm53-flash-aiden-2x,-4x,-4x-labdrafter,glm53-full-nvfp4-spark-aiden.- NEW tag to investigate:
aidendle94/sparkrun-vllm-ds4-gb10:production-3.73-vision(pushed 2026-09-06): DeepSeek-V4-Flash 2x-Spark serving (entrypointdeepseek-v4-flash-2x-spark-entrypoint, pinned launcherserve-ds4-flash-spark.sh, SPARK_RECIPE_REF8b00816d), pins: torch 2.12.0 / flashinfer 0.6.15 @ce40d25a/ NCCL @5067397c/ DeepGEMMa6b593d, modeldeepseek-ai/DeepSeek-V4-Flash(-Vision)rev-pinned, SPARK_KV_CACHE = 26214034944 B ≈ 24.4 GiB @ GMU 0.82 (independent confirmation of the 24 GiB/rank headroom theory), CUDAGRAPH_MODE switch (FULL_AND_PIECEWISE). Where "vision" fits: repo tag 3.72-vision (2026-09-03) was already pulled on all nodes; 3.73-vision is today's follow-up.
Status ledger (main matrix above): DONE = r17abi baseline, r18 2x/4x/DCP4/ nvfp4KV, RedHat MTP, eugr 2x(FAIL), spark-2x-nvfp4kv (158, best 2x); BLOCKED = dflash2 drafter (r18 fork C4 assert), dsv4-vision-eugr (no aligner in eugr img), 4x Spark ckpt (mm_mxfp8 TP4); NEW QUEUE = aiden glm53 lanes re-bench (same-basis), prod-3.73-vision investigation + boot on ds4.