# Night shift 2026-09-05/06 — r18 campaign (GLM-5.3-Flash)

Full lane matrix: re-baselined anchor, r18 build (base `a8c796f3` + #582 + #646,
tree `67e7e6e`), 2x and 4x cells, RedHat/eugr model A/B. All benches:
tool-eval-bench `2.6.1.dev45+gcf54b4bfe`, seed 42, hardmode (88 scenarios),
`--perf` (llama-benchy pp2048/tg128, runs=3, c1+c2). Score = /176 weighted
points; /100 = quality score; dep = deployability (α=0.7).

## Combined table (score + perf)

| Lane | image | score | /100 | dep | KV tokens | pp c1 | tg c1 | TTFT c1 | pp c2 | tg c2 | TTFT c2 | tg c2 @8k-ctx | verdict |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| r17abi 2x (anchor) | r17 (`glm53-jovian-r12`) | 153 | 87 | 69 | ~1.25M | 1,577 | 29.0 | 1,362ms | 1,595 | 41.7 | 2,093ms | 29.7 | re-baseline |
| r18 2x (ds4) | r18 | 157 | 89 | 71 | ~1.25M | 1,473 | 30.7 | 1,481ms | 1,374 | 40.2 | 2,268ms | 30.6 | PASS, 2x production candidate |
| RedHat MTP k=5 2x | lab | 156 | 89 | 70 | ~1.25M | 1,340 | 26.8 | 1,638ms | 460 | 22.4 | 4,667ms | 14.7 | quality PASS, perf FAIL at c2 |
| eugr MTP k=5 2x | eugr b12x | 143 | 81 | 65 | ~1.25M | 1,292 | 31.1 | 2,397ms | 1,388 | 41.6 | 2,751ms | 17.7 | FAIL (−10 vs anchor) |
| r18 4x RoCE | r18 | 156 | 89 | 76 | 1.47M | 2,904 | 47.7 | 762ms | 2,413 | 70.9 | 1,399ms | 35.9 | PASS, throughput king |
| r18 4x DCP4 | r18 | 161 | 91 | 75 | 4.84M | 1,639 | 43.1 | 1,521ms | 1,756 | 51.1 | 1,637ms | 30.0 | BEST score, prefill tax |
| r18 4x nvfp4 KV | r18 | 161 | 91 | 77 | 2.35M | 2,614 | 45.3 | 873ms | 2,250 | 62.6 | 1,368ms | 35.6 | PASS, best deployability |
| r18 2x nvfp4 KV (10G) | r18 | 154 | 88 | 70 | 2.14M | (bench: see report) | — | — | — | — | — | — | PASS, no regression vs 157 |
| r18 2x nvfp4 KV + **Spark ckpt** | r18 | **158** | **90** | 70 | 2.14M | (bench: see report) | — | — | — | — | — | — | **BEST 2x** |
| r18 4x nvfp4 KV + Spark ckpt (16G) | r18 | — | — | — | — | — | — | — | — | — | — | — | BLOCKED: mm_mxfp8 not TP4-tileable |
| r18 4x DFlash2 k=7 | r18 | — | — | — | — | — | — | — | — | — | — | — | BLOCKED (fork C4-page assert) |
| dsv4-flash-vision eugr 2x | eugr b12x | — | — | — | — | — | — | — | — | — | — | — | BLOCKED (no V4-Vision in image) |

pp/tg = tokens/s (prefill 2048 / generate 128); TTFT = ms; d0 = no extra
context, @8k-ctx = 8192 tokens of history.

## Reading the table

- **4x RoCE** = the all-round production candidate: best prefill (2.9k, 2×
  2x), best c2 decode (70.9), half the 2x TTFT, quality 156 = 2x r18.
- **4x DCP4** = capacity + quality play: 4.84M KV (3.3× plain 4x at the same
  10GiB pin — the sparkring steal) and the best score (161), but the DCP ring
  exchange costs prefill (1.6k vs 2.9k).
- **4x nvfp4 KV** ≈ RoCE perf, +1.59× KV, best deployability (77).
  Next cell: DCP4 + nvfp4 KV combined (~7.7M KV projected).
- **RedHat MTP k=5**: quality fine, but the lab image's MTP path collapses
  under concurrency (pp c2 460 t/s, TTFT 4.7s) — dropped from the candidate
  set once perf was on the table.
- **eugr**: no edge anywhere (quality −10, decode degrades at depth); 4x cell
  skipped. Lanes demoted.
- **nvfp4 KV at 2x**: no quality regression (154 vs 157 non-Spark, within
  noise) + 1.7× KV capacity (2.14M @ 10G) — adopt.
- **Spark checkpoint on r18**: +1 over non-Spark (158, 90/100) — the best 2x
  lane of the campaign; 4x blocked by the mm_mxfp8 TP4 constraint.
- 4x cells now pinned at 16 GiB KV (NVFP4 4x headroom; 2x stays 10).
- JIT persistence live: cold serve ~45 min → 9 min warm (per-node caches).
- **r18 2x** beats the anchor on both axes (157/89 vs 153/87) — 2x default
  candidate pending operator sign-off.

## Blocked lanes (documented in recipe headers)

- **DFlash2 k=7 drafter**: drafter inherits the engine's FP8 KV dtype; B12X
  and FLASH_ATTN both reject it on SM121 ("FP8 KV cache requires FA3 on SM90
  or FA4 on SM100"); with drafter `kv_cache_dtype=auto` the fork's own
  assertion fires ("GLM MLA parent-page stride must be an exact number of C4
  pages"). Needs a fork-side fix.
- **dsv4-flash-vision eugr**: the eugr 0904 image's
  `DeepseekV4ForCausalLM` has no `aligner` module (predates V4-Vision
  support); the Vision-Exp vision weights fail to load. Fork lane remains
  the only V4-Vision path.

## Ops log (condensed)

- Re-baseline (queue 1a) first, on the current r17 image, before any r18
  gate — all comparisons above use the 153/176 anchor.
- r18 build on dgx4: 5 attempts. Root causes: transient apt 503s;
  kv-cache-cleanup patch anchor replaced upstream (MemorySnapshot flow —
  patch now skips cleanly); **staged-deps submodule stubs** — both
  `flash-attention/csrc/cutlass` and `MSA/python/fmha_sm100/cutlass` were
  empty dirs in `deps-src.tar.gz` (r18 tree dropped implicit cutlass include
  wiring; MSA install needs the dir) — populated with each project's own
  pin (nvidia/cutlass `62750a2b` / `eb61c911`), 23k macOS AppleDouble
  files purged.
- DCP4 requires `--decode-context-parallel-size 4` +
  `--cp-kv-cache-interleave-size 4` (GLM C4 DCP); B12X is not DCP-compatible
  (engine falls to FLASH_ATTN for that cell).
- #646 geometry verified live: "attention block size 2304" = the 2048+256
  split in the serve log.
- Image separation (operator direction): `docker-images/glm53-jovian-r18/`
  + `glm53-jovian-r18:local` tag; r12 dir restored to r17 state
  (`git checkout 582343c`); r12 tag removed from nodes (points at nothing
  until an honest r17 rebuild — r17-era deps staging unrecoverable, see
  state doc).
- 4x recipes need `tensor_parallel_size = 4` (world size must cover nnodes);
  2x→4x port of the r17abi recipe is the r18-4x family.

## Reports

Full per-lane reports: `recipes/<lane>/reports/` (r17abi: `2026-09-06_r18-rebaseline-r17abi-2x.md`,
`2026-09-06_r18-image-ds4.md`; redhat/eugr: `2026-09-06_*.md`; r18-4x family:
`2026-09-06_r18-4x*.md`). Analysis + open decisions:
`specs/tech-architecture/glm53-recipe-state.md` → "Night-shift results".

## Night-list revisit (2026-09-06, operator request): honest statuses

Missing from the night matrix — **aiden community-image lanes** (operator is
right: no aiden image ran in this campaign; the historical aiden scores came
from the pool-176 era and must not be compared with the 88-scenario hardmode
anchors above). Tasked for the next window:

- `aidendle94/sparkrun-vllm-glm53-gb10:production-1.0/1.1` — GLM-5.3 aiden
  stack (torch 2.12 / flashinfer 0.6.15 lineage, on all 4 nodes), recipes:
  `glm53-flash-aiden-2x`, `-4x`, `-4x-labdrafter`, `glm53-full-nvfp4-spark-aiden`.
- **NEW tag to investigate: `aidendle94/sparkrun-vllm-ds4-gb10:production-3.73-vision`**
  (pushed 2026-09-06): DeepSeek-V4-Flash 2x-Spark serving (entrypoint
  `deepseek-v4-flash-2x-spark-entrypoint`, pinned launcher
  `serve-ds4-flash-spark.sh`, SPARK_RECIPE_REF 8b00816d), pins: torch **2.12.0** /
  flashinfer **0.6.15** @ ce40d25a / NCCL @ 5067397c / DeepGEMM a6b593d,
  model `deepseek-ai/DeepSeek-V4-Flash(-Vision)` rev-pinned, **SPARK_KV_CACHE
  = 26214034944 B ≈ 24.4 GiB @ GMU 0.82** (independent confirmation of the
  24 GiB/rank headroom theory), CUDAGRAPH_MODE switch (FULL_AND_PIECEWISE).
  Where "vision" fits (operator confirmation): **this is the
  DeepSeek-V4-Flash-**Vision** aiden lane** — a direct competitor to our
  fork lane `dsv4-flash-vision` (DSpark-6) and the eugr image that
  FAILED to carry V4-Vision at all. The 'vision' branch effectively
  replaces the blocked dsv4-flash-vision-eugr row with an actionable one.
  Existing recipes `dsv4-vision-aiden-2x/4x` point at this family already.

Status ledger (main matrix above): DONE = r17abi baseline, r18 2x/4x/DCP4/
nvfp4KV, RedHat MTP, eugr 2x(FAIL), spark-2x-nvfp4kv (158, best 2x); BLOCKED =
dflash2 drafter (r18 fork C4 assert), dsv4-vision-eugr (no aligner in eugr
img), 4x Spark ckpt (mm_mxfp8 TP4); NEW QUEUE = aiden glm53 lanes re-bench
(same-basis), prod-3.73-vision investigation + boot on ds4.

## 2026-09-06 night, part 2 (operator: "switched back to pluto", dgx1-4 free)

Dual-bench window: both lanes benched in parallel, seed 42 hardmode + perf,
tool-eval-bench 2.6.1.dev46 (±1 pt version drift vs the dev45 anchors).

| lane | nodes | score | dep | pp c1 d0 | tg c2 d0 | TTFT d0 c1 | KV tokens |
|---|---|---|---|---|---|---|---|
| **aiden production-1.2** (DFlash2 k=7, eager, fp8 KV) | 2x 2nodes-dac | **159 (90)** | — | 1,271 | 30.1 | 1,802ms | 461,730 |
| **r27 (2x spark, nvfp4 KV, MTP)** | 2x ds4 | 156 (88) | — | 1,356 | 31.4 | 1,535ms | 2,206,618 |

Readings:
- **aiden = first-ever canonical measurement of the community GLM-5.3 lane**
  (159/176, new 2x quality record by +1 over r18-spark 158 — within noise,
  not a clear win; DFlash2 acceptance 4.27–4.94 healthy). The drafter lane
  is NOT dead: it was the r18 fork's C4-page assert, not the drafter.
  aiden profile costs: 5x smaller KV, eager decode, higher TTFT.
- **r27 engine not yet at parity**: same 2x config as r18-spark scores 156
  vs 158 (−2) and perf is close but slower in decode (31.4 vs 36.0 t/s c2).
  r27 is the tree to keep qualifying (DCP4/DCP-16GiB + DFlash2 re-test still
  the high-value targets); the r18 image stays the 2x reference until r27
  closes the gap.
- r27 image: attempt-4 rebuild (explicit TORCH_CUDA_ARCH_LIST=12.1a in the
  vllm-builder stage — the attempt-3 build lost the stage-inherited env and
  shipped sm120-only kernels → cudaErrorNoKernelImageForDevice).

Next queue: prod-3.73-vision (DSV4F-Vision aiden lane) on 2nodes-dac.

## Night 2 (2026-09-06→07): aiden + r27 dual bench (pluto switch, dgx1-4 free)

Tool: 2.6.1.dev46+g9d412464a (campaign upgrade from dev45 — same version both
runs, cross-lane comparison valid).

| Lane | Score | dep | KV tokens | pp c1 d0/d8k | tg c1 d0/d8k | tg c2 d0 | TTFT c1 d0 |
|---|---|---|---|---|---|---|---|
| r18-2x-nvfp4kv-spark (anchor) | 158 (90) | 70 | 2,136,194 | 1,861/1,718 | 24.5/22.5 | 36.0 | 1,236ms |
| **aiden prod-1.2 2x** (DFlash2 k=7, fp8 KV 4GiB, eager) | **159 (90)** | — | 461,730 | 1,271/1,669 | 23.5/22.8 | 30.1 | 1,802ms |
| **r27 2x** (spark ckpt, nvfp4 KV) | 156 (88) | — | 2,206,618 | 1,356/1,682 | 22.7/22.9 | 31.4 | 1,535ms |

**aiden prod-1.2 = first canonical aiden measurement ever**: 159/176 (90.4),
74p/11partial/3fail. DFlash2 acceptance 4.27–4.94 (healthy, not the <0.15
broken-aux signature). Fails: TC-43/58/61. Profile: eager + 4 GiB KV pin →
KV pool 461K (small), prefill 1.2–1.7K t/s, decode ~23 t/s. The aiden lane
ties/beats the r18-spark anchor by a point in quality but at much smaller
context headroom — a real community-stack datapoint, not a replacement.

**r27 lane (r27 engine, same 2x spark config as the 158 anchor)**: 156/176 —
−2 vs r18 on this lane. r27 prefill is ~27% slower (1,356 vs 1,861 pp c1),
decode in line. The r27 tree has NOT beaten the r18 build on quality yet;
the remaining r27 value = the qualified DFlash2 drafter + DCP4 + 16 GiB KV
cells (next windows). r27 image: attempt-4 rebuild with explicit
TORCH_CUDA_ARCH_LIST=12.1a in vllm-builder (attempt-3 lost the stage
inherited env → sm120-only kernels → cudaErrorNoKernelImageForDevice).

Next: prod-3.73-vision (DSV4F-Vision aiden lane) investigation.

## Vision lane: prod-3.73-vision (2026-09-07, 2x 2nodes-dac)

**161/176 (91.5) — 74p/13partial/1fail; new 2x campaign record** (ties the
best 4x cells: r18 4x DCP4 / nvfp4KV 161). Model
DeepSeek-V4-Flash-Vision-Exp (157 GB, cached, rev 6821d6ad), image
`aidendle94/sparkrun-vllm-ds4-gb10:production-3.73-vision`
(vLLM `0.1.dev19023+g30038602b` — aiden's ds4-vision 0.20 fork: B12X MLA
sparse, b12x MoE/linear, instanttensor, FULL_AND_PIECEWISE graphs, async
scheduling, deepseek_v4 parsers). KV 1,442,886 tokens (fp8, 24.4 GiB cap),
GMU 0.80, no spec (MTP/DSpark off).

Perf (llama-benchy pp2048 tg128): c1 1,501/25.4 (d0) → 1,863/25.3 (d8k);
c2 1,766/37.0 → 2,030/21.5. TTFT d0 2.5s c1 / 1.75s c2.
Fails: TC-38 only; partials: 13 (TC-14/46/50/51/53/57/58/61/62/63/68/82/88).

Ops notes (new capabilities added to the launcher for entrypoint-driven
images): recipe `model = ""` now skips the positional model (image's own
entrypoint selects it via MODEL_PATH env); docker path now injects
NODE_RANK + MASTER_ADDR env (their launcher requires them; CLI flags go
through `"$@"` and win for the standard flags). First boot failed with
PP=2 (launcher default pp=nnodes) → the ds4-vision fork asserts
TP×PP against device count → recipe pins `pipeline_parallel_size = 1`.

## r27 4x DCP4 (2026-09-07, all 4 nodes, 16 GiB KV)

**155/176 (88.1) — 72p/11partial/5fail. −6 vs the r18 4x DCP4 161.**
KV **7,738,490 tokens** (16 GiB pin, as projected; vs 4.84M at 10 GiB on r18).
Perf: pp c1 d0 1,713 (d8k 2,013), tg c1 44.0 / c2 53.0 @d0, TTFT 1.4s.
Decode is fast; quality is not: the r27 engine trails r18 on the DCP4
cell too (2x: −2; 4x DCP4: −6). Fails: TC-38/43/45/51/80.
Verdict so far: r27 = the tree for the future (DCP4 headroom 7.7M, DFlash2
qualified upstream), but the r18 image remains the quality reference for
GLM-5.3 2x/4x until r27 closes the gap.

## r27 DFlash2 (2026-09-07) — BLOCKED (tree-level, root-caused)

First boot attempt: drafter inherited the main model's fp8_ds_mla KV →
"No valid attention backend" (FP8 KV needs FA3/SM90 or FA4/SM100; SM121
has neither) → fixed with `kv_cache_dtype: "auto"` (bf16) in the
speculative config. Second boot: **ValueError: GLM MLA parent-page stride
must be an exact number of C4 pages** (glm5next/nvidia/pooled_indexer.py) —
the same assert family that blocked r18 dflash2, now confirmed in the r27
tree: with the drafter pooled into the main model's pages the parent stride
breaks the C4 index-page alignment. Not an r18 artifact; r27's upstream
DFlash2 qualification was sm_120-only (the pooled indexer is an SM12x path).
Verdict: DFlash2 on GB10 needs a fork-level patch before it can run; the
lane recipe is kept with the blocker documented (draft_model/FLASH_ATTN
r18 style and method-dflash/auto both dead-end on the same geometry).

## Aiden 4x (2026-09-07, 4nodes-switch, production-1.2+instanttensor)

**163/176 (92.6) — 79p/5partial/4fail. NEW CAMPAIGN RECORD** (beats r18 4x
DCP4 161, vision-373 161, aiden 2x 159). zai-org FP8 (306 G), TP=4, MTP k=5,
fp8_e4m3 KV 2,322,623 tokens @ 16 GiB pin, eager. The FP8-MLA+DFlash2 combo
stays broken (recipe ships MTP instead, per aiden's guide). Fails:
TC-43/51/58/85. Perf run was noisy (one pp cell measured 0; tg 26–42 t/s) —
treat the quality as the headline, the perf as indicative only.
Ops: the hub production-1.2 image lacks instanttensor → extended via
`docker-images/glm53-flash-aiden/` (FROM production-1.2 + instanttensor
0.1.9 --no-deps), built on dgx4, on all 4 nodes; the launcher's image sync
re-copied it per node (docker load re-stamps the config digest).

## Eager A/B (2026-09-07, r18 2x nvfp4kv-spark + --enforce-eager)

**159/176 (90.4) vs 158 (90) with graphs — +1, within run-to-run noise.**
Perf ≈ parity: pp c1 d0 1,856 (anchor 1,861), tg c1 27.1 (24.5), tg c2
36.8 (36.0), TTFT 1,165ms (1,236). Verdict: the "eager improves quality"
rumor does not hold on the r18 GB10 stack — eager is a neutral alternative
(graphs stay the default; no reason to pay the ~0.6 GiB + capture time
savings either way). Fails: TC-43/45/57/76-class set (4 fails, 9 partials).