dflash2 night A/B: 4x 165/176 (+2, prefill +27%, TTFT -31%) vs MTP3; 2x Spark blocked (C4 page-stride + interleave%4 gates); history + spec

history/night-shifts/2026-09-12.md

@@ -145,3 +145,65 @@ exact capture, SEQS 8, thinking+max, engram-local, deepseek-v4-flash alias).
   generation defect does not exist in r35), LMCache fs-eviction accounting,
   DFlash2 K7 qualified, latest lab line. TC-51 = variance vs regression
   unresolved (needs an r31 dev65 re-bench or a second r35 run).
+
+## Night shift (22:00–00:10 UTC+2 local, unattended ops + DFlash2 A/B)
+
+### DFlash2 4x A/B — WINNER on latency, +2 points
+
+Recipe `glm53-flash-r35-4x-dflash2` (clone of the r35 MTP3 prod, ONLY the
+speculative-config swapped: MTP3 → `{"method":"dflash","model":"...DFlash2-MXFP8",
+"num_speculative_tokens":7,"kv_cache_dtype":"auto"}`). No build, no download:
+the r35 image ships the DFlash2 machinery (DFlash2DraftModel, dedicated
+dflash_attn graph-safe split-KV backend, FlashInferCutlassMxfp8LinearKernel
+for the fused draft K/V projection) and the 1.2 GB MXFP8 draft was already on
+the shared NAS.
+
+Boot gates green: `Resolved architecture: DFlash2DraftModel`, method=dflash
+K7, and — the 08-29 blocker is DEAD — the r35 tree pads the mamba page size
+(+0.70%) so draft/target page geometry unifies instead of the old
+`4096 % 660 != 0` abort. KV: 6,333,399 tokens @512K (12.08x) vs MTP3's
+8,060,928 — the draft's 5 sliding-attention layers + BF16 draft pages eat part
+of the 16 GiB pin.
+
+| | r35 MTP3 (prod, 11:31) | **r35 DFlash2 K7 (22:20)** |
+|---|---|---|
+| score (dev65, seed 42) | 163/176 (93) | **165/176 (94)** |
+| pp / tg / TTFT, d0 c1 | 1,554 / 49.6 / 1,544 ms | **1,977 / 47.3 / 1,071 ms** |
+| pp / tg, d0 c2 | 1,489 / 53.0 | 1,475 / **55.8** |
+| KV | 8.06M tokens | 6.33M tokens |
+
+Same harness both runs (clean A/B). Prefill **+27%**, TTFT **−31%**,
+decode −4.6% @c1 / +5.3% @c2, score +2 (TC-40/50/63 now pass; new partials
+TC-60 + one more; shared fails TC-43/45/51/68). The run package (manifest +
+recipe copy, the 081a0e6 feature) landed on its first live run; found and
+fixed a utc_now_iso bug in the same commit (civil_date got seconds, produced
+year 4900778).
+
+Verdict (operator, 09-12 night): the DFlash2 lane is the better 4x config on
+this bench — but the operator keeps r35 MTP3 as prod for now (the day's
+decision) and logs DFlash2 as the 4x candidate: +2 points, big TTFT/prefill
+gain, −22% KV (6.3M vs 8.06M — matters for the 512K concurrency story).
+Restore path: `sm121 stop + serve` the MTP3 recipe.
+
+### DFlash2 2x Spark — BLOCKED on the engine side (definitive)
+
+`glm53-flash-r35-2x-dflash2` (r31-2x shape, r35 image, NVFP4-Spark target,
+same draft). First attempt (bare TP2, no DCP): both ranks abort 5 min into
+boot with `ValueError: GLM MLA parent-page stride must be an exact number of
+C4 pages` — the r35-tree successor of the 08-29 page-unification blocker.
+Second attempt (+DCP2, interleave 2): `ValueError: GLM C4 DCP requires
+cp_kv_cache_interleave_size to be divisible by 4, got 2`. The interleave must
+be a multiple of 4 → the only working DFlash2 shapes are the lab's certified
+TP4/DCP1 and TP4/DCP4 (4x only). This matches the lab's qualification
+exactly (their 2x Spark profile is MTP3-only). **2x stays MTP3; DFlash2 is
+a 4x-only lane on this tree.** The recipe is kept as a regression canary
+(BLOCKED notes in the file).
+
+Ops note: sm121's `docker run` carries `--rm` by default — a crashed rank-0
+container vanished with its logs. `sm121 serve --keep` suppresses `--rm`
+and was used for the second attempt (logs survived, root cause captured).
+
+### Power-off (operator instruction)
+
+Cluster 4nodes-switch powered off via HA after the A/Bs (prod state: down
+until the next serve; MTP3 r35 = prod of record, DFlash2 = candidate).

recipes/glm53-flash-r35-2x-dflash2/recipe.toml

@@ -1,4 +1,7 @@
 # GLM-5.3-Flash r35 + DFlash2 K7 draft on 2x Spark (DAC) — night A/B part 2.
+# BLOCKED on the engine side (see the args comment: C4 page-stride gates).
+# Kept as a regression canary: if a future lab build fixes 2x DFlash2 page
+# geometry, this recipe is the test. 2x prod stays MTP3 (r31-2x / r35 4x).
 #
 # Clone of recipes/glm53-flash-r31-2x with the r35 image (glm53-jovian-r35:local,
 # checksum 8bb87348, DFlash2 machinery in-tree) and ONLY the speculative-config
@@ -125,6 +128,17 @@ args = [
     # _select_sparse_backend); the sm12x auto-priority list never offers it
     # (FLASHINFER_MLA_SPARSE_SM120 only supports head 576, not GLM's 512).
     "--attention-backend", "B12X",
+    # BLOCKED (2026-09-12 night, r35 img, both nodes): DFlash2 + GLM MLA page
+    # geometry does not work on 2x Spark. Two independent engine-side gates:
+    #   bare TP2: "GLM MLA parent-page stride must be an exact number of
+    #             C4 pages" (5 min into boot, both ranks)
+    #   +DCP2/interleave2: "GLM C4 DCP requires cp_kv_cache_interleave_size
+    #             to be divisible by 4, got 2"
+    # The interleave-multiple-of-4 rule means the working shapes are the lab's
+    # certified TP4/DCP1 and TP4/DCP4 (4x only) — see the 4x A/B, which
+    # passed the same checks (KV 6.33M tokens) and benched 165/176. 2x
+    # DFlash2 stays BLOCKED on the engine side; 2x stays on MTP3.
+
     # pure I/O loader (kept across every config; 171 GB in ~5.5 min vs ~16 NFS)
     "--load-format", "instanttensor",
     # [R17 FULL-CLUSTER TEST] lil manifest backends (quality gate: hardmode

specs/tech-architecture/glm53-recipe-state.md

@@ -1205,3 +1205,39 @@ recompile on first boot):
   stays; r31 = rollback lane, image tagged on all nodes). Full log:
   `history/night-shifts/2026-09-12.md` ("Daytime follow-up: R35 adoption
   + A/B").
+
+### DFlash2 A/B on the r35 lane (2026-09-12 night, 4nodes-switch)
+
+The r35 image ships the DFlash2 machinery natively (DFlash2DraftModel,
+dedicated `dflash_attn` graph-safe split-KV backend, FlashInferCutlassMxfp8
+linear for the fused draft K/V projection) and the 1.2 GB lab MXFP8 draft
+was already on the shared NAS — so the A/B was a recipe-only swap (MTP3 →
+`method:dflash`, K7, kv auto), no build. The 08-29 2x blocker (page
+unification) is gone in the r35 tree: it pads the mamba page size (+0.70%)
+to unify draft/target geometry.
+
+4x DCP4 (same dev65 harness/seed as the prod run — clean A/B):
+
+| | r35 MTP3 (prod) | **r35 DFlash2 K7** |
+|---|---|---|
+| score | 163/176 (93) | **165/176 (94)** |
+| pp / tg / TTFT @d0 c1 | 1,554 / 49.6 / 1,544 ms | **1,977 / 47.3 / 1,071 ms** |
+| tg @d0 c2 | 53.0 | 55.8 |
+| KV @512K | 8.06M tokens (15.4x) | 6.33M tokens (12.1x) |
+
+DFlash2 = prefill +27%, TTFT −31%, decode −4.6% c1 / +5.3% c2, +2 points
+(TC-40/50/63 pass; new partial TC-60 + 1). Operator keeps MTP3 as prod of
+record for now; **DFlash2 K7 is the 4x candidate** (better latency story,
+−22% KV). Context: aiden's 1.1 image (different tree, incoai draft) already
+ran DFlash2 on 2x (157/176) — the 2x block below is specific to the r35
+tree's C4 page model, not to DFlash2 per se.
+
+**2x Spark is BLOCKED on the r35 tree** (both gates hit, both ranks):
+bare TP2 → `GLM MLA parent-page stride must be an exact number of C4
+pages`; +DCP2/interleave-2 → `GLM C4 DCP requires
+cp_kv_cache_interleave_size to be divisible by 4`. The interleave-multiple-
+of-4 rule makes the certified TP4/DCP1 + TP4/DCP4 shapes (4x only) the
+working ones — exactly the lab's qualification set. 2x stays MTP3; recipe
+`glm53-flash-r35-2x-dflash2` kept as a canary. Ops note: sm121 `docker run`
+is `--rm` by default (crashed rank-0 logs vanished); `serve --keep` keeps
+containers + logs for post-mortem.