# Night shift 2026-09-05/06 — r18 campaign (GLM-5.3-Flash)

Full lane matrix: re-baselined anchor, r18 build (base `a8c796f3` + #582 + #646,
tree `67e7e6e`), 2x and 4x cells, RedHat/eugr model A/B. All benches:
tool-eval-bench `2.6.1.dev45+gcf54b4bfe`, seed 42, hardmode (88 scenarios),
`--perf` (llama-benchy pp2048/tg128, runs=3, c1+c2). Score = /176 weighted
points; /100 = quality score; dep = deployability (α=0.7).

## Combined table (score + perf)

| Lane | image | score | /100 | dep | KV tokens | pp c1 | tg c1 | TTFT c1 | pp c2 | tg c2 | TTFT c2 | tg c2 @8k-ctx | verdict |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| r17abi 2x (anchor) | r17 (`glm53-jovian-r12`) | 153 | 87 | 69 | ~1.25M | 1,577 | 29.0 | 1,362ms | 1,595 | 41.7 | 2,093ms | 29.7 | re-baseline |
| r18 2x (ds4) | r18 | 157 | 89 | 71 | ~1.25M | 1,473 | 30.7 | 1,481ms | 1,374 | 40.2 | 2,268ms | 30.6 | PASS, 2x production candidate |
| RedHat MTP k=5 2x | lab | 156 | 89 | 70 | ~1.25M | 1,340 | 26.8 | 1,638ms | 460 | 22.4 | 4,667ms | 14.7 | quality PASS, perf FAIL at c2 |
| eugr MTP k=5 2x | eugr b12x | 143 | 81 | 65 | ~1.25M | 1,292 | 31.1 | 2,397ms | 1,388 | 41.6 | 2,751ms | 17.7 | FAIL (−10 vs anchor) |
| r18 4x RoCE | r18 | 156 | 89 | 76 | 1.47M | 2,904 | 47.7 | 762ms | 2,413 | 70.9 | 1,399ms | 35.9 | PASS, throughput king |
| r18 4x DCP4 | r18 | 161 | 91 | 75 | 4.84M | 1,639 | 43.1 | 1,521ms | 1,756 | 51.1 | 1,637ms | 30.0 | BEST score, prefill tax |
| r18 4x nvfp4 KV | r18 | 161 | 91 | 77 | 2.35M | 2,614 | 45.3 | 873ms | 2,250 | 62.6 | 1,368ms | 35.6 | PASS, best deployability |
| r18 2x nvfp4 KV (10G) | r18 | 154 | 88 | 70 | 2.14M | (bench: see report) | — | — | — | — | — | — | PASS, no regression vs 157 |
| r18 2x nvfp4 KV + **Spark ckpt** | r18 | **158** | **90** | 70 | 2.14M | (bench: see report) | — | — | — | — | — | — | **BEST 2x** |
| r18 4x nvfp4 KV + Spark ckpt (16G) | r18 | — | — | — | — | — | — | — | — | — | — | — | BLOCKED: mm_mxfp8 not TP4-tileable |
| r18 4x DFlash2 k=7 | r18 | — | — | — | — | — | — | — | — | — | — | — | BLOCKED (fork C4-page assert) |
| dsv4-flash-vision eugr 2x | eugr b12x | — | — | — | — | — | — | — | — | — | — | — | BLOCKED (no V4-Vision in image) |

pp/tg = tokens/s (prefill 2048 / generate 128); TTFT = ms; d0 = no extra
context, @8k-ctx = 8192 tokens of history.

## Reading the table

- **4x RoCE** = the all-round production candidate: best prefill (2.9k, 2×
  2x), best c2 decode (70.9), half the 2x TTFT, quality 156 = 2x r18.
- **4x DCP4** = capacity + quality play: 4.84M KV (3.3× plain 4x at the same
  10GiB pin — the sparkring steal) and the best score (161), but the DCP ring
  exchange costs prefill (1.6k vs 2.9k).
- **4x nvfp4 KV** ≈ RoCE perf, +1.59× KV, best deployability (77).
  Next cell: DCP4 + nvfp4 KV combined (~7.7M KV projected).
- **RedHat MTP k=5**: quality fine, but the lab image's MTP path collapses
  under concurrency (pp c2 460 t/s, TTFT 4.7s) — dropped from the candidate
  set once perf was on the table.
- **eugr**: no edge anywhere (quality −10, decode degrades at depth); 4x cell
  skipped. Lanes demoted.
- **nvfp4 KV at 2x**: no quality regression (154 vs 157 non-Spark, within
  noise) + 1.7× KV capacity (2.14M @ 10G) — adopt.
- **Spark checkpoint on r18**: +1 over non-Spark (158, 90/100) — the best 2x
  lane of the campaign; 4x blocked by the mm_mxfp8 TP4 constraint.
- 4x cells now pinned at 16 GiB KV (NVFP4 4x headroom; 2x stays 10).
- JIT persistence live: cold serve ~45 min → 9 min warm (per-node caches).
- **r18 2x** beats the anchor on both axes (157/89 vs 153/87) — 2x default
  candidate pending operator sign-off.

## Blocked lanes (documented in recipe headers)

- **DFlash2 k=7 drafter**: drafter inherits the engine's FP8 KV dtype; B12X
  and FLASH_ATTN both reject it on SM121 ("FP8 KV cache requires FA3 on SM90
  or FA4 on SM100"); with drafter `kv_cache_dtype=auto` the fork's own
  assertion fires ("GLM MLA parent-page stride must be an exact number of C4
  pages"). Needs a fork-side fix.
- **dsv4-flash-vision eugr**: the eugr 0904 image's
  `DeepseekV4ForCausalLM` has no `aligner` module (predates V4-Vision
  support); the Vision-Exp vision weights fail to load. Fork lane remains
  the only V4-Vision path.

## Ops log (condensed)

- Re-baseline (queue 1a) first, on the current r17 image, before any r18
  gate — all comparisons above use the 153/176 anchor.
- r18 build on dgx4: 5 attempts. Root causes: transient apt 503s;
  kv-cache-cleanup patch anchor replaced upstream (MemorySnapshot flow —
  patch now skips cleanly); **staged-deps submodule stubs** — both
  `flash-attention/csrc/cutlass` and `MSA/python/fmha_sm100/cutlass` were
  empty dirs in `deps-src.tar.gz` (r18 tree dropped implicit cutlass include
  wiring; MSA install needs the dir) — populated with each project's own
  pin (nvidia/cutlass `62750a2b` / `eb61c911`), 23k macOS AppleDouble
  files purged.
- DCP4 requires `--decode-context-parallel-size 4` +
  `--cp-kv-cache-interleave-size 4` (GLM C4 DCP); B12X is not DCP-compatible
  (engine falls to FLASH_ATTN for that cell).
- #646 geometry verified live: "attention block size 2304" = the 2048+256
  split in the serve log.
- Image separation (operator direction): `docker-images/glm53-jovian-r18/`
  + `glm53-jovian-r18:local` tag; r12 dir restored to r17 state
  (`git checkout 582343c`); r12 tag removed from nodes (points at nothing
  until an honest r17 rebuild — r17-era deps staging unrecoverable, see
  state doc).
- 4x recipes need `tensor_parallel_size = 4` (world size must cover nnodes);
  2x→4x port of the r17abi recipe is the r18-4x family.

## Reports

Full per-lane reports: `recipes/<lane>/reports/` (r17abi: `2026-09-06_r18-rebaseline-r17abi-2x.md`,
`2026-09-06_r18-image-ds4.md`; redhat/eugr: `2026-09-06_*.md`; r18-4x family:
`2026-09-06_r18-4x*.md`). Analysis + open decisions:
`specs/tech-architecture/glm53-recipe-state.md` → "Night-shift results".