r27 dflash2 BLOCKED: C4 parent-page stride assert (tree-level, both r18+r27), root cause documented

history/night-shifts/2026-09-05.md

@@ -212,3 +212,18 @@ cell too (2x: −2; 4x DCP4: −6). Fails: TC-38/43/45/51/80.
 Verdict so far: r27 = the tree for the future (DCP4 headroom 7.7M, DFlash2
 qualified upstream), but the r18 image remains the quality reference for
 GLM-5.3 2x/4x until r27 closes the gap.
+
+## r27 DFlash2 (2026-09-07) — BLOCKED (tree-level, root-caused)
+
+First boot attempt: drafter inherited the main model's fp8_ds_mla KV →
+"No valid attention backend" (FP8 KV needs FA3/SM90 or FA4/SM100; SM121
+has neither) → fixed with `kv_cache_dtype: "auto"` (bf16) in the
+speculative config. Second boot: **ValueError: GLM MLA parent-page stride
+must be an exact number of C4 pages** (glm5next/nvidia/pooled_indexer.py) —
+the same assert family that blocked r18 dflash2, now confirmed in the r27
+tree: with the drafter pooled into the main model's pages the parent stride
+breaks the C4 index-page alignment. Not an r18 artifact; r27's upstream
+DFlash2 qualification was sm_120-only (the pooled indexer is an SM12x path).
+Verdict: DFlash2 on GB10 needs a fork-level patch before it can run; the
+lane recipe is kept with the blocker documented (draft_model/FLASH_ATTN
+r18 style and method-dflash/auto both dead-end on the same geometry).

recipes/glm53-flash-r27-dflash2/recipe.toml

@@ -78,8 +78,19 @@
 #
 # PREREQUISITES: weights on the NAS hub (175 GiB): hf download local-inference-lab/GLM-5.3-Flash-NVFP4-Spark
 
+# BLOCKED (2026-09-07, GB10/SM121): boots until the main-model KV layout
+# check in glm5next/nvidia/pooled_indexer.py:
+#   ValueError: GLM MLA parent-page stride must be an exact number of C4 pages
+# The pooled FP8 C4 index tail must fit the parent page stride; with the
+# DFlash2 drafter pooled into the same pages the stride is no longer an
+# exact multiple of the C4 index page. Same class of failure as the r18
+# dflash2 block — it is tree-level (present in r27 @63a82f8d), not an
+# r18-build artifact. r27's upstream DFlash2 qualification ran on
+# x86/sm_120 (rtx6kpro), where the SM12x-only pooled-indexer path is not
+# exercised. A fork-level patch (drafter KV outside the C4-aligned page,
+# or drafter KV in fp8 with an SM121-capable backend) is required first.
 engine = "vllm"
-name = "GLM-5.3-Flash NVFP4 (r27 img, DFlash2 k=7, 2x)"
+name = "GLM-5.3-Flash NVFP4 (r27 img, DFlash2 k=7, 2x) [BLOCKED]
 docker_image = "glm53-jovian-r27:local"  # r27 clone-at-pin build
 docker_image_dir = "glm53-jovian-r27"
 # Required: the clean r12 image follows the solar contract (ENTRYPOINT []
@@ -148,12 +159,15 @@ args = [
     # The draft model's vllm_config takes its attention backend from the
     # SPECULATIVE config, not the global flag (eagle/utils.py:
     # _make_eagle_draft_vllm_config) — B12X here mirrors the main model's.
-    # r18 used draft_model/FLASH_ATTN/kv auto — both rejected on that fork
-# (FP8-KV-drafter backend check + C4-page stride assert). r27 qualifies
-# dflash2 upstream (the #646 2048+256 page split is in this tree); the
-# drafter/depth follow the working aiden GB10 lane (method "dflash").
+    # r18 used draft_model/FLASH_ATTN/kv auto — the FP8-KV-drafter backend check
+# and the C4-page stride assert blocked it on that fork. r27 qualifies
+# dflash2 upstream (the #646 2048+256 page split is in this tree); drafter
+# repo/depth follow the working aiden GB10 lane (method "dflash", k=7).
+# kv_cache_dtype MUST be "auto" (bf16) for the drafter: without it the
+# drafter inherits the main model's fp8_ds_mla KV and every backend on
+# SM121 rejects it (FP8 KV needs FA3/SM90 or FA4/SM100).
 "--speculative-config", """
-'{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7}'""",
+'{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7,"kv_cache_dtype":"auto"}'""",
     "--language-model-only",
     # MTP + CUDA graphs degrades OUTPUT QUALITY on the glm53 day-0 dev build
     # (forum #381350/#44, CONFIRMED by A/B bench 2026-08-28: same seed/flags,