# Night shift 2026-10-22 → 23 — report

Stand dgx5–8 (production dgx1–4 untouched all night). Plan:
[`2026-10-22-night-plan.md`](2026-10-22-night-plan.md).

## 1. The headline: the full GLM-5.3 works, and speculation is what makes it usable

`glm53-mxfp4-fp8-4x-r39` — `RedHatAI/GLM-5.3-MXFP4xFP8` (740B total / ~40B
active, 78 layers, DSA, 1M native window) on four Sparks at TP4.

| | decode c1 | decode c2 (agg) | score (v2.8.0) | median turn |
|---|---|---|---|---|
| first boot, no speculation | 19.0 t/s | — | 88/100 | 8.8 s |
| **MTP k=3** | **33.4–34.9 t/s** | **47.6 t/s** | — | — |
| MTP k=5 | 33.1–37.3 t/s | 47.9 t/s | — | — |
| **MTP k=3 @ 131072 ctx** | ~31 t/s (benchy) | 36.5 t/s | **90/100** (175/194) | **5.7 s** |
| reference: GLM-5.3-Flash 2x | 52 t/s | 51 | 95/100 | 3.6 s |
| reference: production 1017 4x (SGLang EP2) | 48–51 t/s | 64–68 | 86–87/100 | 1.4 s |

So the biggest model we can run now scores **90/100 with 5.7 s turns** — better
quality than production, on four nodes nobody else uses, at ~60% of prod's
decode speed. k=5 buys nothing over k=3: drafts are accepted 87–93% of the time
(`Mean acceptance length 4.36–4.66 of 5`), so verification, not the ladder,
is the limit.

**Boot chain** (each failure cost a 30-minute reload; all recorded in the
recipe header):

1. kindling `spark-glm53:v10` — multi-node worker init dies
   (`is_in_the_same_node: gloo/tcp`), the fork is TP2-shaped.
2. r39 + `--attention-backend B12X` — `TypeError: DeepseekV32B12xAttention
   got an unexpected keyword 'index_group_builder'` (the re-cut indexer API is
   newer than r39's class).
3. r39 + `--linear-backend marlin` — `prepare_fp8_layer_for_marlin` trips on the
   `ReplicatedLinear` modules the re-cut puts fp8_block on.
4. r39 default dispatch — `mm_fp4 does not support backend 'cute-dsl' with
   capability 121`.
5. **`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`** — MXFP4 linears fall
   to Marlin, fp8_block keeps its own backend. Boots.

**Karmic A/B** (`spark-glm53-karmic:local`, the local-inference-lab fork):
native `B12X_MXFP4_MXFP8` MoE experts vs r39's `MarlinExperts` = 19.2 vs 19.0
t/s. At 99 GiB of weights per rank decode is bandwidth-bound; a faster GEMM is
not the lever, speculation is.

## 2. Two hard limits found by measurement

**DCP4 does not exist for this checkpoint.** Explicitly selecting the backend
answers why:

```
ValueError: Selected backend AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120
is not valid for this configuration. Reason: ['DCP not supported']
```

Auto-selection finds no MLA backend at all once DCP is on; same on both forks,
block 64 and 256. The DSA/sparse-MLA path has no context-parallel
implementation in our images, so the 512K/1M windows of the Flash sibling are
out of reach. The window is therefore KV-pin-bound only (`52.7 KB/token/rank`;
146,944 tokens at a 7.5 GiB pin).

**And the KV window is not the usable prompt length.** The needle ladder:

| prompt | result |
|---|---|
| 40,269 tok | ✅ correct, TTFT ~85 s |
| ~120K tok | ❌ `RPC call to sample_tokens timed out` (even with `VLLM_RPC_TIMEOUT=3600`) → EngineDead, and **dgx6 + dgx7 wedged** (ping alive, sshd starved) |
| ~150K tok (earlier, 196K window) | ❌ same, plus the same wedge |

Recovery both times: HA plug cycle of the two nodes (off 35 s → on), back in
~1 min, no data loss (weights live on the NAS). The cause is not the KV pool:
after 98.86 GiB of weights a Spark has ~22 GiB, and a long prefill's scratch
does not fit — smaller chunks and a bigger RPC budget do not help.

**Practical envelope for this lane:** ≤64K prompts (the 131072 window buys
queueing headroom, not a 128K prompt); 8K→64K ladders only. Real 128K+ needs
TP8, i.e. halving the weights per rank.

## 3. FLARE — works, and is honest about where it stands

NVIDIA published `NVIDIA/flare-local-inference` on 2026-10-20; the full
procedure and the six failure modes are in
[`specs/tech-architecture/flare-bringup.md`](../../specs/tech-architecture/flare-bringup.md).
The production checkpoint (`deepseek-ai/DeepSeek-V4.1-Flash-1017`, pinned
revision) prepared at TP4 across dgx5–8 and served by the Rust runtime:
1M context advertised, 24 GiB KV per rank (28.9M tokens), two RoCE rails per
peer. First boot: prefill ~1,430–1,700 t/s, decode 19.2 t/s c1 / 29.6 c2 —
~2.4× slower than our SGLang EP2 production lane on the same model, with
nothing tuned. Their upstream moved 12+ commits during the night, including
"Preset DeepSeek V4.1 packed GPU speculation" and adaptive verification
scoring; that is the obvious next pass.

The agents were **stopped and disabled** afterwards (unit files kept), and a
check confirmed nothing returns at boot.

## 4. Dashboard / add-on: five releases, one honest post-mortem

`1.3.4` → `1.3.9`. Features: the recipes panel's Model column now has the same
hover affordances as the recipe name — `⧉` copies the model id, `HF` opens the
checkpoint page on the Hub (only for `org/name` ids) — plus an inline SVG
favicon on both pages, a **build-version badge** under the dashboard title, and
a `webui` entry so the HA add-on page has an "OPEN WEB UI" button.

The release chain itself is worth remembering:

| version | what went wrong |
|---|---|
| 1.3.4 | built from a launcher commit that **never reached gitea** (a chained `git push -q` failed silently) |
| 1.3.5 | rebuild: buttons + icon land, no badge yet |
| 1.3.6 | badge present but shows `vdev` — a Docker `ARG` never reaches the compiler |
| 1.3.7 | `ENV` in the builder — still `dev`: the warm cargo cache (`target/` on a cache mount) served the old literal |
| 1.3.8 | `cargo:rerun-if-env-changed` — still `dev` (the cached fingerprint wins) |
| **1.3.9** | the version is generated by `build.rs` into `OUT_DIR`, and the CI log prints the value it baked |

Verified in the published image, not just in source:
`docker run … sm121 web …; curl /api/state` → `{"version":"1.3.9", …}`.

## 5. Research (the night's reading list)

- **`RedHatAI/GLM-5.3-speculator.dspark`** — 2.5 GB, 3-layer Qwen3 draft fed by
  hidden states from target layers 2/20/39/58/75, up to 8 drafts, 1M context,
  41.16% acceptance / 4.29 accepted length at 524K–1M. Downloaded; on r39 it
  passes config and draft-backend selection, then dies in
  `qwen3_dflash._build_context_kv_buffers` — the speculator targets **vLLM
  0.32 / speculators 0.7.x**. Retry belongs on a vLLM-main image (eugr nightly,
  already staged on two nodes).
- **vLLM 0.32.0** released; `RedHatAI/GLM-5.3-MXFP4xFP8` re-cut again on 22.10
  14:05 (150 files).
- **`tunamitom/b12x@76b76347`** — MXFP4-A4 prefill for MoE (CuTe
  `mxfp4_mma_m16n8k64_f32_e2m1_ue8m0`, dynamic E8M0 activation scales, decode
  stays exact W4A16, switch at `B12X_W4A16_MXFP4_PREFILL_MIN_TOKENS=512`),
  reported ~+10% prefill and "should work for Flash too". Directly relevant to
  our prefill pain → todo #23.
- **`madeby561/vllm:mimo-v26-flash-b12x-20260923-rc4`** (12.9 GB) and its
  `opus55`/`kk-winner` siblings → todo #22. Caveat: the tag is **amd64-only**,
  so a Spark needs an arm64 manifest or a source build.
- FLARE, sparkring (`sparkring`, `sparkcache`, `net-sircl`,
  `glm53-flash-tp4-spark`), local-inference-lab `blackwell-llm-docker` (92★) and
  the TensorFold family all moved; nothing newer than what we already track.

## 6. State of the fleet in the morning

dgx5–8: **free** (four nodes, no containers, all online). dgx1–4: production
1017 untouched. Add-on: 1.3.9 published (buttons, icon, version badge, OPEN WEB
UI button). All work committed and pushed: launcher `60ffdda`, recipes
`2b7b4c7`, add-on `c6fa22f`.

## 7. What is worth doing next

1. **DSpark on a vLLM-main image** (eugr nightly staged): the 2.5 GB speculator
   is the single biggest remaining lever on the full GLM-5.3 (MTP already took
   19 → 35 t/s; a real drafter with 4.3 accepted tokens should go further).
2. **b12x A4 prefill** (todo #23): +10% prefill claimed, and prefill is exactly
   what limits our usable prompt length.
3. **FLARE tuning pass** on 1017 (`--scheduler-policy throughput`,
   `--max-concurrency`, adaptive verification) to see whether the Rust runtime
   can close the 2.4× decode gap on our own production model.
4. Only then the quality question: is 90/100 on four otherwise-idle nodes worth
   a production slot? The Flash 2x lane still leads at 95/100 on two nodes.