night plan 2026-09-17: full GLM-5.3 + DSpark K8 (speculator.dspark, downloaded), r37 typo verdict (boot D + eugr 0913), 8x trials on the new cables, lab lil-catalog policies

history/night-shifts/2026-09-17.md

@@ -0,0 +1,171 @@
+# Night shift plan — 2026-09-17
+
+## Context since the 09-13 shift
+
+- Prod (DS4.1 4x tonyd2wild) was restored the morning of 09-14, then stopped
+  for the R37 investigation; the R37 typo bisect (A: no linear-b12x, B: prod
+  spec K5, C: full prod contract) all **garbled** → cause is in the r37 tree
+  code, not the flags. Boot D (engram off) never ran: the 4x cluster went
+  dark mid-load (~07:27 UTC 09-14); operator later power-cycled dgx1+2 only
+  and deployed `glm53-flash-eugr-2x` (eugr b12x nightly-20260910) on the
+  2nodes-dac pair. **This lane is running and must NOT be stopped** until
+  the operator says so (explicit instruction 09-17).
+- dev66 bench (09-14): eugr 2x = **161/176 (91)**, d0 c1 1,326 pp / 28.1 tg /
+  1,458 ms — quality lane verdict unchanged (r28 2x = prefill lane).
+- R37 lane status: unqualified for DS4.1 (typos) — pending Boot D + lab
+  issue. The bug report draft exists:
+  `specs/bugs/BUG-2026-09-14-r37-ds41-text-garbling.md`.
+- New tooling shipped 09-14: `sm121 runlock list|close` + report stale
+  cross-check (commits `2f71763`, `706b7a5`).
+
+## What updated upstream (scanned 09-17)
+
+- **RedHatAI/GLM-5.3-speculator.dspark** (09-16, v1.0): external DSpark K8
+  drafter for the FULL GLM-5.3 (zai-org base, GlmMoeDsaForCausalLM). 3-layer
+  Qwen3-style backbone (hidden 6144, sliding window 2048), 2.5 GB bf16,
+  aux hidden states from target layers 2/20/39/58/75, Markov head (vanilla,
+  rank 256) + confidence head, greedy proposal, verifier_accept_k=1, 1M ctx
+  native. Card acceptance: tool_call 4.01, math 6.13, HumanEval 4.76 tokens/
+  step; MRCR 524K–1M 41.16% / 4.293 avg. Deploy contract (from the card):
+  `--speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark",
+  "num_speculative_tokens":8,"method":"dspark"}'`. Trained on vllm
+  0.28.0 / speculators 0.8.0.dev196 — our r37 tree reports 0.26.1rc0 but
+  **has the integration** (grep-verified: DSparkDraftModel method
+  resolution in `config/speculative.py:1333/1404`, speculators config
+  loader at `transformers_utils/configs/speculators`, draft_model field,
+  model registered in `models/registry.py`). Needs `--trust-remote-code`
+  (repo carries config.py custom code) — the full recipe already sets it.
+  Drafter trained on BF16-target hidden states; we serve MXFP4 — acceptance
+  may shift a bit; that's part of what the A/B measures.
+- **lil-catalog updated (snapshot bab43db6, 09-15)** — the lab's qualified
+  launch manifests. Only change: `Qwen3.8-Flash-Next-NVFP4` dropped
+  `limit_per_prompt: image: 1`. Extracts that matter:
+  - `DeepSeek-V4-Flash` (rev 60d8d707): lab policy = **MTP k=2** (moe mxfp4,
+    probabilistic/standard), GMU **0.91**, seqs **64**, batched tokens 8192 —
+    very different from our prod contract (dspark K5, GMU 0.85, seqs 8).
+    NOTE: this is V4-Flash, not our V4.1-Flash; the 0731 variant carries the
+    DSpark head.
+  - `DeepSeek-V4-Flash-Vision-Exp`: dspark K7 + adaptive_verification — the
+    lab qualified the vision lane we have an image for.
+  - `GLM-5.3-NVFP4` (lab's FULL GLM-5.3, 433 GB): NVFP4 routed experts +
+    BF16 MTP expert layer, **MTP k=3 model:target**, B12X attention,
+    long_prefill 2048, async_scheduling, FULL_AND_PIECEWISE + custom_ops
+    all, CUDA_DEVICE_MAX_CONNECTIONS 32, no GMU pin. Their qualified
+    full-model lane — we never ran spec on the full model.
+  - `GLM-5.3-Flash-DFlash2-MXFP8` (1.2 GB, kind: draft): MXFP8 DFlash2
+    drafter variant, compatible with the NVFP4/Spark Flash models.
+- **eugr**: b12x line's latest tag is still `nightly-20260913` (built from
+  the lab's **dev/infernal-invocation** — a NEWER lab branch than our r37
+  tree 3fb88fdf); non-b12x line pushed `nightly-20260917`. Changelog
+  highlights: Qwen3.8-Flash-Next solo recipe with `VLLM_PLE_TABLE_MEMORY=
+  disk` (1M+ KV at GMU 0.8), Vision-Exp B12X + DSpark, GLM Flash b12x
+  loader.
+- **vLLM 0.28.0** (08-26): DeepSeek V4 sparse MLA end-to-end for decode/MTP/
+  DSpark (#51538) — candidate upstream fix for our DCP>1 sparse-decode
+  crash (512K blocker); DFlash2 loading from speculators format (#53797);
+  DSpark confidence-scheduled verification (#47808); tiered KV offload
+  with a disk tier; per-request spec-decode metrics (#48915). Our lab tree
+  (0.26.1rc0) predates all of this — rebasing is still the lab's call.
+- **tonyd2wild** expanded to GLM-5.3-Flash: `ghcr.io/tonyd2wild/
+  vllm-glm53-flash:sm121-v11-dflash2` (+ `sm121-v8` base): 5 SM121 fixes
+  (NoPE-MLA sparse backend, FlashInfer 0.6.17→0.6.18 FA2 NaN, dep re-pins,
+  PDL gating, indexer top-k init) + fp8 KV unlock. Measured TP2 @262K:
+  DFlash2 46.9 tg/s (74.1% acceptance, 2.15×), TP4/1M sibling 68.5 tg/s,
+  pool 2.62M. A third independent stack for the GLM Flash lane.
+- Forum #382120 (EXL3/TR3 kit, 4bpw uncensored): 2.2M-token pool @900K ctx
+  on 2x; their host-sync findings (per-layer `torch.unique` gather →
+  ~700 ms stalls; fixed-shape no-dedup gather + raised fused temp rows)
+  are good context for the lab issues doc.
+
+## Prep done (operator-gated: nothing stopped)
+
+- `RedHatAI/GLM-5.3-speculator.dspark` downloaded to the shared HF cache
+  (2.3 GB, snapshot 198b3ced) ✓
+- r37-image grep: external-dspark support confirmed ✓
+- lil-catalog read + diffed ✓
+
+## Night program
+
+### P0 — Full GLM-5.3 MXFP4 + DSpark K8 on 4x @131K (operator's pick)
+1. Copy `glm53-full-mxfp4-4x` → `glm53-full-mxfp4-4x-dspark8`; keep the
+   qualified boot ladder (linear-backend marlin, interleave 1, 131072,
+   8 GiB KV pin, GMU 0.85) and add the spec config from the card.
+   `draft_tensor_parallel_size` — start with 4 (drafter rides the TP
+   group); if the drafter is small enough consider tp1 later.
+2. Boot probe: expect "Using DSpark…" + drafter load lines; watch the
+   known MXFP4 hazards (MarlinExperts selection, b12x warmup).
+3. Quick quality gate FIRST (typo probe: 300-word RU generation + code
+   identifiers — the lane must produce clean text), then bench
+   `dev67-full-glm53-dspark8` vs anchor: no-spec 158/176 (90),
+   934 pp / 19.0 tg / 2,124 ms. Expect decode ×2–3 (acceptance ~4/step at
+   131K ≥ the MRCR long-context numbers), prefill unchanged.
+4. Teardown: stop, restore whatever the operator wants up.
+
+### P1 — r37 typo verdict (closes the morning's bisect)
+1. **Boot D**: r37 image, full prod contract, engram off (`--engram-config`
+   removed) → typo probe. Clean → the lab's engram disk-read path is the
+   cause; garbled → tree numerics; either way finalize
+   `BUG-2026-09-14-r37-ds41-text-garbling.md` and file it with the lab.
+2. **eugr b12x nightly-20260913 A/B**: pull (11.2 GB), same probe recipe on
+   4x — built from the lab's `dev/infernal-invocation`, one lab-branch
+   newer than our r37. Clean text → the r37/DS4.1-alt line revives on the
+   newer tree.
+3. If both still garble: DS4.1 stays on tonyd2wild prod; R37 = GLM-lane
+   only (where it's proven).
+
+### P2 — 8x trials (all cables are on site)
+Gate first: dgx5–8 must be powered + provisioned (ssh/docker/NFS/HF cache
+mounts). dgx6–8 IPs (88.x ssh + 200.x ib) are unknown to the launcher —
+discover at night start (HA power on → arp/ssh probe → record), then add
+cluster `8nodes` (dgx1–8, ranks 0–7, entity `switch.dgxN`, NCCL block
+copied from `4nodes-switch`). Ask the operator about provisioning state.
+1. **8x full GLM-5.3 MXFP4 @512K DCP1**: TP8 → 47 GiB weights/node → KV
+   budget ≈ 45+ GiB/rank at GMU 0.85 → a 512K request costs 26.3 GiB/rank
+   (52.7 KB/token/rank × 512K) and fits **without DCP** — sidesteps the
+   DCP>1 sparse-decode bug entirely. Pin KV ~28–32 GiB (pool ≈ 550–620K
+   tokens/rank); 768K needs ~39.5 GiB (fits at the top of the budget).
+   + DSpark K8 drafter (2.3 GiB). This is the full-model long-context lane
+   the 4x cluster cannot deliver.
+2. **8x lab GLM-5.3-NVFP4 + MTP k3** (manifest policy): 433 GB → 54.1 GiB/
+   node; the lab's own qualified full-model config; A/B vs (1).
+3. **8x DS4.1-Flash**: aggregate KV ~2× (10.4M → ~20M tokens) — the
+   capacity/1M-context lane for prod traffic.
+Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is
+qualified).
+
+### P3 — optional, time permitting
+1. **Qwen3.8-Flash-Next-NVFP4** quick trial (1–2 nodes): new family,
+   catalog-updated 09-15 (image limit dropped — vision serving);
+   MTP k3, GMU 0.94, gdn_decode b12x, block 16, kv bf16, PLE env vars.
+2. **tonyd2wild GLM-Flash v11-dflash2** on 2x: their 262K DFlash2 stack vs
+   our eugr 2x (1M) — only if the 2x lane becomes free (it must NOT be
+   stopped for this).
+3. **DS4.1-Flash lab-policy A/B** (MTP k2, GMU 0.91, seqs 64): requires
+   downloading `deepseek-ai/DeepSeek-V4-Flash` @60d8d70 (~510 GB) — a
+   multi-hour NAS download; start early if the operator wants it.
+
+### P4 — bookkeeping
+- File the lab issues (garbling bisect + 3 engine bugs from 09-13).
+- Push the ~20 local commits when git.kilork.org returns.
+- End-state: per operator. Default guardrail unchanged: 4nodes-switch
+  powered down at end **unless** the operator leaves an 8x lane serving.
+
+## Guardrails
+
+- **The running 2x glm53 eugr lane (dgx1+2) is not to be stopped or
+  restarted** until the operator says so — all 4x/8x work must wait for or
+  reuse free nodes (dgx3/4 are off; the 4x P0/P1 blocks need them).
+- Typo probe gates any DS4.1/r37 promotion (the 09-14 lesson).
+- `sm121 serve --keep` for first/unknown boots; short-timeout polling.
+- Build heavy images on dgx5 only; `docker save|ssh|docker load` for
+  distribution.
+- Kernel stays 6.17.0-1031 (7.0.0-1019 breaks NCCL/RoCE — never upgrade).
+
+## Open questions for the operator
+
+1. Are dgx6–8 OS-provisioned (ssh kilork, docker, /mnt/nas mount, HF cache
+   path)? Any known IPs? (Needed to define the `8nodes` cluster config.)
+2. Which blocks matter tonight: P0 (DSpark K8 on the full model) is the
+   stated wish; 8x needs your go + node provisioning.
+3. After the night: leave the 2x eugr lane serving, or power everything
+   down?