night report 2026-10-22/23: full GLM-5.3 at 90/100 with MTP, DCP impossible, prefill ceiling measured, FLARE brought up, add-on 1.3.4-1.3.9

Consolidates the shift: the boot chain and its four failure modes, the
speculation numbers (19 -> 35 t/s, 88 -> 90/100, 8.8 -> 5.7 s), the two hard
limits (no DCP for the DSA path; long prefill wedges the hosts at ~120K and
needs an HA plug cycle), the FLARE bring-up and its first-boot numbers, the
dashboard releases with the push/cache post-mortem, the research reading list
and the four candidate next steps.

history/night-shifts/2026-10-23.md

@@ -0,0 +1,159 @@
+# Night shift 2026-10-22 → 23 — report
+
+Stand dgx5–8 (production dgx1–4 untouched all night). Plan:
+[`2026-10-22-night-plan.md`](2026-10-22-night-plan.md).
+
+## 1. The headline: the full GLM-5.3 works, and speculation is what makes it usable
+
+`glm53-mxfp4-fp8-4x-r39` — `RedHatAI/GLM-5.3-MXFP4xFP8` (740B total / ~40B
+active, 78 layers, DSA, 1M native window) on four Sparks at TP4.
+
+| | decode c1 | decode c2 (agg) | score (v2.8.0) | median turn |
+|---|---|---|---|---|
+| first boot, no speculation | 19.0 t/s | — | 88/100 | 8.8 s |
+| **MTP k=3** | **33.4–34.9 t/s** | **47.6 t/s** | — | — |
+| MTP k=5 | 33.1–37.3 t/s | 47.9 t/s | — | — |
+| **MTP k=3 @ 131072 ctx** | ~31 t/s (benchy) | 36.5 t/s | **90/100** (175/194) | **5.7 s** |
+| reference: GLM-5.3-Flash 2x | 52 t/s | 51 | 95/100 | 3.6 s |
+| reference: production 1017 4x (SGLang EP2) | 48–51 t/s | 64–68 | 86–87/100 | 1.4 s |
+
+So the biggest model we can run now scores **90/100 with 5.7 s turns** — better
+quality than production, on four nodes nobody else uses, at ~60% of prod's
+decode speed. k=5 buys nothing over k=3: drafts are accepted 87–93% of the time
+(`Mean acceptance length 4.36–4.66 of 5`), so verification, not the ladder,
+is the limit.
+
+**Boot chain** (each failure cost a 30-minute reload; all recorded in the
+recipe header):
+
+1. kindling `spark-glm53:v10` — multi-node worker init dies
+   (`is_in_the_same_node: gloo/tcp`), the fork is TP2-shaped.
+2. r39 + `--attention-backend B12X` — `TypeError: DeepseekV32B12xAttention
+   got an unexpected keyword 'index_group_builder'` (the re-cut indexer API is
+   newer than r39's class).
+3. r39 + `--linear-backend marlin` — `prepare_fp8_layer_for_marlin` trips on the
+   `ReplicatedLinear` modules the re-cut puts fp8_block on.
+4. r39 default dispatch — `mm_fp4 does not support backend 'cute-dsl' with
+   capability 121`.
+5. **`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`** — MXFP4 linears fall
+   to Marlin, fp8_block keeps its own backend. Boots.
+
+**Karmic A/B** (`spark-glm53-karmic:local`, the local-inference-lab fork):
+native `B12X_MXFP4_MXFP8` MoE experts vs r39's `MarlinExperts` = 19.2 vs 19.0
+t/s. At 99 GiB of weights per rank decode is bandwidth-bound; a faster GEMM is
+not the lever, speculation is.
+
+## 2. Two hard limits found by measurement
+
+**DCP4 does not exist for this checkpoint.** Explicitly selecting the backend
+answers why:
+
+```
+ValueError: Selected backend AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120
+is not valid for this configuration. Reason: ['DCP not supported']
+```
+
+Auto-selection finds no MLA backend at all once DCP is on; same on both forks,
+block 64 and 256. The DSA/sparse-MLA path has no context-parallel
+implementation in our images, so the 512K/1M windows of the Flash sibling are
+out of reach. The window is therefore KV-pin-bound only (`52.7 KB/token/rank`;
+146,944 tokens at a 7.5 GiB pin).
+
+**And the KV window is not the usable prompt length.** The needle ladder:
+
+| prompt | result |
+|---|---|
+| 40,269 tok | ✅ correct, TTFT ~85 s |
+| ~120K tok | ❌ `RPC call to sample_tokens timed out` (even with `VLLM_RPC_TIMEOUT=3600`) → EngineDead, and **dgx6 + dgx7 wedged** (ping alive, sshd starved) |
+| ~150K tok (earlier, 196K window) | ❌ same, plus the same wedge |
+
+Recovery both times: HA plug cycle of the two nodes (off 35 s → on), back in
+~1 min, no data loss (weights live on the NAS). The cause is not the KV pool:
+after 98.86 GiB of weights a Spark has ~22 GiB, and a long prefill's scratch
+does not fit — smaller chunks and a bigger RPC budget do not help.
+
+**Practical envelope for this lane:** ≤64K prompts (the 131072 window buys
+queueing headroom, not a 128K prompt); 8K→64K ladders only. Real 128K+ needs
+TP8, i.e. halving the weights per rank.
+
+## 3. FLARE — works, and is honest about where it stands
+
+NVIDIA published `NVIDIA/flare-local-inference` on 2026-10-20; the full
+procedure and the six failure modes are in
+[`specs/tech-architecture/flare-bringup.md`](../../specs/tech-architecture/flare-bringup.md).
+The production checkpoint (`deepseek-ai/DeepSeek-V4.1-Flash-1017`, pinned
+revision) prepared at TP4 across dgx5–8 and served by the Rust runtime:
+1M context advertised, 24 GiB KV per rank (28.9M tokens), two RoCE rails per
+peer. First boot: prefill ~1,430–1,700 t/s, decode 19.2 t/s c1 / 29.6 c2 —
+~2.4× slower than our SGLang EP2 production lane on the same model, with
+nothing tuned. Their upstream moved 12+ commits during the night, including
+"Preset DeepSeek V4.1 packed GPU speculation" and adaptive verification
+scoring; that is the obvious next pass.
+
+The agents were **stopped and disabled** afterwards (unit files kept), and a
+check confirmed nothing returns at boot.
+
+## 4. Dashboard / add-on: five releases, one honest post-mortem
+
+`1.3.4` → `1.3.9`. Features: the recipes panel's Model column now has the same
+hover affordances as the recipe name — `⧉` copies the model id, `HF` opens the
+checkpoint page on the Hub (only for `org/name` ids) — plus an inline SVG
+favicon on both pages, a **build-version badge** under the dashboard title, and
+a `webui` entry so the HA add-on page has an "OPEN WEB UI" button.
+
+The release chain itself is worth remembering:
+
+| version | what went wrong |
+|---|---|
+| 1.3.4 | built from a launcher commit that **never reached gitea** (a chained `git push -q` failed silently) |
+| 1.3.5 | rebuild: buttons + icon land, no badge yet |
+| 1.3.6 | badge present but shows `vdev` — a Docker `ARG` never reaches the compiler |
+| 1.3.7 | `ENV` in the builder — still `dev`: the warm cargo cache (`target/` on a cache mount) served the old literal |
+| 1.3.8 | `cargo:rerun-if-env-changed` — still `dev` (the cached fingerprint wins) |
+| **1.3.9** | the version is generated by `build.rs` into `OUT_DIR`, and the CI log prints the value it baked |
+
+Verified in the published image, not just in source:
+`docker run … sm121 web …; curl /api/state` → `{"version":"1.3.9", …}`.
+
+## 5. Research (the night's reading list)
+
+- **`RedHatAI/GLM-5.3-speculator.dspark`** — 2.5 GB, 3-layer Qwen3 draft fed by
+  hidden states from target layers 2/20/39/58/75, up to 8 drafts, 1M context,
+  41.16% acceptance / 4.29 accepted length at 524K–1M. Downloaded; on r39 it
+  passes config and draft-backend selection, then dies in
+  `qwen3_dflash._build_context_kv_buffers` — the speculator targets **vLLM
+  0.32 / speculators 0.7.x**. Retry belongs on a vLLM-main image (eugr nightly,
+  already staged on two nodes).
+- **vLLM 0.32.0** released; `RedHatAI/GLM-5.3-MXFP4xFP8` re-cut again on 22.10
+  14:05 (150 files).
+- **`tunamitom/b12x@76b76347`** — MXFP4-A4 prefill for MoE (CuTe
+  `mxfp4_mma_m16n8k64_f32_e2m1_ue8m0`, dynamic E8M0 activation scales, decode
+  stays exact W4A16, switch at `B12X_W4A16_MXFP4_PREFILL_MIN_TOKENS=512`),
+  reported ~+10% prefill and "should work for Flash too". Directly relevant to
+  our prefill pain → todo #23.
+- **`madeby561/vllm:mimo-v26-flash-b12x-20260923-rc4`** (12.9 GB) and its
+  `opus55`/`kk-winner` siblings → todo #22. Caveat: the tag is **amd64-only**,
+  so a Spark needs an arm64 manifest or a source build.
+- FLARE, sparkring (`sparkring`, `sparkcache`, `net-sircl`,
+  `glm53-flash-tp4-spark`), local-inference-lab `blackwell-llm-docker` (92★) and
+  the TensorFold family all moved; nothing newer than what we already track.
+
+## 6. State of the fleet in the morning
+
+dgx5–8: **free** (four nodes, no containers, all online). dgx1–4: production
+1017 untouched. Add-on: 1.3.9 published (buttons, icon, version badge, OPEN WEB
+UI button). All work committed and pushed: launcher `60ffdda`, recipes
+`2b7b4c7`, add-on `c6fa22f`.
+
+## 7. What is worth doing next
+
+1. **DSpark on a vLLM-main image** (eugr nightly staged): the 2.5 GB speculator
+   is the single biggest remaining lever on the full GLM-5.3 (MTP already took
+   19 → 35 t/s; a real drafter with 4.3 accepted tokens should go further).
+2. **b12x A4 prefill** (todo #23): +10% prefill claimed, and prefill is exactly
+   what limits our usable prompt length.
+3. **FLARE tuning pass** on 1017 (`--scheduler-policy throughput`,
+   `--max-concurrency`, adaptive verification) to see whether the Rust runtime
+   can close the 2.4× decode gap on our own production model.
+4. Only then the quality question: is 90/100 on four otherwise-idle nodes worth
+   a production slot? The Flash 2x lane still leads at 95/100 on two nodes.