# Night shift plan — 2026-09-18

## Context since the 09-13 shift

- Prod (DS4.1 4x tonyd2wild) was restored the morning of 09-14, then stopped
  for the R37 investigation; the R37 typo bisect (A: no linear-b12x, B: prod
  spec K5, C: full prod contract) all **garbled** → cause is in the r37 tree
  code, not the flags. Boot D (engram off) never ran: the 4x cluster went
  dark mid-load (~07:27 UTC 09-14); operator later power-cycled dgx1+2 only
  and deployed `glm53-flash-eugr-2x` (eugr b12x nightly-20260910) on the
  2nodes-dac pair. **This lane is running and must NOT be stopped** until
  the operator says so (explicit instruction 09-17).
- dev66 bench (09-14): eugr 2x = **161/176 (91)**, d0 c1 1,326 pp / 28.1 tg /
  1,458 ms — quality lane verdict unchanged (r28 2x = prefill lane).
- 09-18 morning: the operator stopped the 2x eugr lane and powered the whole
  cluster down while he provisioned the four new nodes (dgx5–8). As of this
  plan there is **nothing serving** and **all 8 nodes are powered on** with
  `sw6_mt` on (prepared for the 8x work, see below).
- R37 lane status: unqualified for DS4.1 (typos) — pending Boot D + lab
  issue. The bug report draft exists:
  `specs/bugs/BUG-2026-09-14-r37-ds41-text-garbling.md`.
- New tooling shipped 09-14: `sm121 runlock list|close` + report stale
  cross-check (commits `2f71763`, `706b7a5`).

## What updated upstream (scanned 09-17)

- **RedHatAI/GLM-5.3-speculator.dspark** (09-16, v1.0): external DSpark K8
  drafter for the FULL GLM-5.3 (zai-org base, GlmMoeDsaForCausalLM). 3-layer
  Qwen3-style backbone (hidden 6144, sliding window 2048), 2.5 GB bf16,
  aux hidden states from target layers 2/20/39/58/75, Markov head (vanilla,
  rank 256) + confidence head, greedy proposal, verifier_accept_k=1, 1M ctx
  native. Card acceptance: tool_call 4.01, math 6.13, HumanEval 4.76 tokens/
  step; MRCR 524K–1M 41.16% / 4.293 avg. Deploy contract (from the card):
  `--speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark",
  "num_speculative_tokens":8,"method":"dspark"}'`. Trained on vllm
  0.28.0 / speculators 0.8.0.dev196 — our r37 tree reports 0.26.1rc0 but
  **has the integration** (grep-verified: DSparkDraftModel method
  resolution in `config/speculative.py:1333/1404`, speculators config
  loader at `transformers_utils/configs/speculators`, draft_model field,
  model registered in `models/registry.py`). Needs `--trust-remote-code`
  (repo carries config.py custom code) — the full recipe already sets it.
  Drafter trained on BF16-target hidden states; we serve MXFP4 — acceptance
  may shift a bit; that's part of what the A/B measures.
- **lil-catalog updated (snapshot bab43db6, 09-15)** — the lab's qualified
  launch manifests. Only change: `Qwen3.8-Flash-Next-NVFP4` dropped
  `limit_per_prompt: image: 1`. Extracts that matter:
  - `DeepSeek-V4-Flash` (rev 60d8d707): lab policy = **MTP k=2** (moe mxfp4,
    probabilistic/standard), GMU **0.91**, seqs **64**, batched tokens 8192 —
    very different from our prod contract (dspark K5, GMU 0.85, seqs 8).
    NOTE: this is V4-Flash, not our V4.1-Flash; the 0731 variant carries the
    DSpark head.
  - `DeepSeek-V4-Flash-Vision-Exp`: dspark K7 + adaptive_verification — the
    lab qualified the vision lane we have an image for.
  - `GLM-5.3-NVFP4` (lab's FULL GLM-5.3, 433 GB): NVFP4 routed experts +
    BF16 MTP expert layer, **MTP k=3 model:target**, B12X attention,
    long_prefill 2048, async_scheduling, FULL_AND_PIECEWISE + custom_ops
    all, CUDA_DEVICE_MAX_CONNECTIONS 32, no GMU pin. Their qualified
    full-model lane — we never ran spec on the full model.
  - `GLM-5.3-Flash-DFlash2-MXFP8` (1.2 GB, kind: draft): MXFP8 DFlash2
    drafter variant, compatible with the NVFP4/Spark Flash models.
- **eugr**: b12x line's latest tag is still `nightly-20260913` (built from
  the lab's **dev/infernal-invocation** — a NEWER lab branch than our r37
  tree 3fb88fdf); non-b12x line pushed `nightly-20260917`. Changelog
  highlights: Qwen3.8-Flash-Next solo recipe with `VLLM_PLE_TABLE_MEMORY=
  disk` (1M+ KV at GMU 0.8), Vision-Exp B12X + DSpark, GLM Flash b12x
  loader.
- **vLLM 0.28.0** (08-26): DeepSeek V4 sparse MLA end-to-end for decode/MTP/
  DSpark (#51538) — candidate upstream fix for our DCP>1 sparse-decode
  crash (512K blocker); DFlash2 loading from speculators format (#53797);
  DSpark confidence-scheduled verification (#47808); tiered KV offload
  with a disk tier; per-request spec-decode metrics (#48915). Our lab tree
  (0.26.1rc0) predates all of this — rebasing is still the lab's call.
- **tonyd2wild** expanded to GLM-5.3-Flash: `ghcr.io/tonyd2wild/
  vllm-glm53-flash:sm121-v11-dflash2` (+ `sm121-v8` base): 5 SM121 fixes
  (NoPE-MLA sparse backend, FlashInfer 0.6.17→0.6.18 FA2 NaN, dep re-pins,
  PDL gating, indexer top-k init) + fp8 KV unlock. Measured TP2 @262K:
  DFlash2 46.9 tg/s (74.1% acceptance, 2.15×), TP4/1M sibling 68.5 tg/s,
  pool 2.62M. A third independent stack for the GLM Flash lane.
- Forum #382120 (EXL3/TR3 kit, 4bpw uncensored): 2.2M-token pool @900K ctx
  on 2x; their host-sync findings (per-layer `torch.unique` gather →
  ~700 ms stalls; fixed-shape no-dedup gather + raised fused temp rows)
  are good context for the lab issues doc.

## Prep done (09-18)

- `RedHatAI/GLM-5.3-speculator.dspark` downloaded to the shared HF cache
  (2.3 GB, snapshot 198b3ced) ✓
- r37-image grep: external-dspark support confirmed ✓
- lil-catalog read + diffed ✓
- git.kilork.org recovered — the ~20 queued local commits pushed ✓

### 8x readiness (done 09-18 daytime, verified live)

- All 8 nodes up (dgx5–8 fresh), `sw6_mt` turned on via the HA relay —
  the 200G fabric is invisible without the switch (operator: the switch
  relay is a separate HA entity, not implied by node power).
- **The "two IPs on the IB interface" question is solved**: every Spark
  has TWO dual-port 200G NICs (`enp1s0` + `enP2p1s0`); port0 of each carries
  a 200.x IP and both are on ONE L2 domain (a single 16-port 200G switch,
  cross-ping verified with sudo ping). Dual-rail 2×200G per node is already
  the current `4nodes-switch` convention (`NCCL_IB_HCA=rocep1s0f0,
  roceP2p1s0f0`). Per-node `ib_ip` = NIC A port0: dgx5=200.11, dgx6=200.10,
  dgx7=200.23, dgx8=200.25 (plus the 100.x/101.x DAC pair on dgx1/2 port1,
  untouched). All 16 switch ports link UP.
- dgx5/6: `/mnt/nas` (nfs4 `nas.lan:/sna`, nconnect=8) validated ✓. dgx7/8:
  **fixed** — the fstab line + `/mnt/nas` were missing; added (sudo
  available, NAS exports `/sna *`), mounted, 65 HF repos visible ✓.
- `8nodes-switch` cluster added to `config.toml` (dgx1–8, ranks 0–7,
  entities `switch.dgxN`, `sw6_mt` blocking + `sw7_fan` non-blocking) with
  `[nccl."8nodes-switch"]` = the dual-rail copy of `4nodes-switch`. All 8
  nodes share one ED25519 host key (cloned Ubuntu image) — same pinned
  `SHA256:aUXh…` as dgx1–4; (security footnote: per-node key regen later).
- `sm121 health --cluster 8nodes-switch`: **8/8 online, all free** ✓.
- Dry-run `serve --cluster 8nodes-switch --fit 8` renders correctly:
  TP8/nnodes 8/ranks 0–7, GID auto-detect on all 8, before_serve hooks ×8,
  MASTER_ADDR=200.15 ✓.
- Images: dgx5 is fully pre-seeded (r37/r36/r35 + both dsv41); dgx6–8 have
  no vLLM image yet — **the launcher syncs them natively at serve**
  (build on `--build-node` + checksum-label sync; no manual
  `docker save|ssh|docker load` — an abandoned manual copy attempt left no
  residue, verified).

## Night program

### P0 — Full GLM-5.3 MXFP4 + DSpark K8 on 4x @131K (operator's pick)
STATUS (09-18 ~13:45 UTC): recipe `glm53-full-mxfp4-4x-dspark8` up on
4nodes-switch, quality gate PASSED (RU 300-word text clean, 6/6 identifier
probe), bench `dspark8` running.

Boot log (6 boots, 12:21 → 13:34):
- Boot 1-2: the draft model's non-MLA DFlash attention (head 64) got the
  target's B12X sparse-MLA `fp8_ds_mla` KV dtype — the target's MLA layer
  rewrites the shared `cache_config.cache_dtype` (mla_attention
  `_canonicalize_sparse_mla_kv_cache_dtype`) and the selector has no backend
  for (head 64, bf16, fp8_ds_mla, non-MLA). Explicit `--kv-cache-dtype
  fp8_e4m3` did NOT help (overridden back to fp8_ds_mla).
- Boot 3: `--kv-cache-dtype-skip-layers sliding_window` (the r37 flag for
  exactly this: skipped layers use auto/bf16 KV; the target's DSA layers are
  not sliding-window so they keep fp8_ds_mla) → attention init passed, but
  the draft SlidingWindowSpec inherited a second pollution: the target's
  `cache_config.skip_page_size_padded` (shared-page padding) < the draft's
  bf16 page → `assert page_size_padded >= unpadded_page_size_bytes` in
  kv_cache_interface.py.
- Fix: `patches/patch_dspark_draft_page.py` (anchor patcher, in-container
  mod via the shared HF cache `sm121-patches/`; the draft spec passes
  `page_size_padded=None` — its KV group is independent and must use its
  own page size). Tested throwaway-container first (check→apply→check).
- Boot 5: KV budget — the 8 GiB pin was 80 MB short of the 8.08 GiB the
  engine needs for one 131072 request + the drafter's KV group → pin raised
  to 9 GiB.
- Boot 6 (13:34): up. KV pool 145,664 tokens @131K (concurrency 1.11x —
  the drafter's KV group eats ~9% of the pin vs the no-spec 159,232),
  PIECEWISE+FULL target graphs + separate dspark graph capture (54 s),
  Application startup complete.

1. [done] Recipe + spec config (K8, method dspark, drafter on NAS
   snapshot 198b3ced; the repo is in the shared HF cache, HF_HUB_OFFLINE).
2. [done] Boot: drafter resolved as Qwen3DSparkModel, graphs captured.
3. [done] Quality gate: typo probe clean (the 09-14 r37 DS4.1 garbling did
   NOT appear here), identifiers 6/6. Note: this recipe's glm47 template
   does not honor chat_template_kwargs thinking:false (the model writes its
   plan into content; reasoning_tokens stays 0) — cosmetic, bench handles it.
4. [done] Bench `dev67-full-glm53-dspark8` (13:43 → 14:31 UTC, dev65+g6be685f0e,
   seed 42, hardmode, c1+c2): **153/176 (87/100, ★★★★)** — within run variance
   of the no-spec anchor 158/176 (−5, same as eugr 161 vs 166). 8 fails, all
   wrong_args/missing_step, no garbling: TC-22/TC-68 invalid JSON, TC-43 empty
   query, TC-45 tool_choice, TC-38 crowded namespace 2/4, TC-51 (safety, the
   same send_email-before-confirm quirk eugr 2x had), TC-80 + TC-87 (★★★★★
   preconditioned update / cursor integrity).
   Perf vs anchor (no-spec 158/176, 934 pp / 19.0 tg / 2,124 ms):
   - d0 c1: **892 pp / 33.2 tg / 2,193 ms — decode 1.75×**, prefill −4.5% (spec
     overhead), TTFT +3% (flat).
   - d4096 c1: 847 pp / 28.4 tg / 6,662 ms; d8192 c1: 863 / 27.3 / 10,979 —
     decode holds 27–28 tg at depth.
   Verdict: the K8 drafter is a real +75% decode win on the full model at 131K
   with quality held (card numbers were BF16-target / MRCR; MXFP4 target +
   131K context land below them, as expected). Report:
   `recipes/glm53-full-mxfp4-4x-dspark8/reports/2026-09-18_134322_dspark8.*`.
5. Teardown: stop, then the P1/P2 sequence.

### P1 — r37 typo verdict (closes the morning's bisect)
1. **Boot D**: r37 image, full prod contract, engram off (`--engram-config`
   removed) → typo probe. Clean → the lab's engram disk-read path is the
   cause; garbled → tree numerics; either way finalize
   `BUG-2026-09-14-r37-ds41-text-garbling.md` and file it with the lab.
2. **eugr b12x nightly-20260913 A/B**: pull (11.2 GB), same probe recipe on
   4x — built from the lab's `dev/infernal-invocation`, one lab-branch
   newer than our r37. Clean text → the r37/DS4.1-alt line revives on the
   newer tree.
3. If both still garble: DS4.1 stays on tonyd2wild prod; R37 = GLM-lane
   only (where it's proven).

### P2 — 8x trials (cables on site; nodes provisioned, cluster configured)
Gate: CLEARED — all 8 nodes up + validated, `8nodes-switch` in config,
health green. The launcher syncs missing images at serve time.
1. **8x full GLM-5.3 MXFP4 @512K DCP1**: TP8 → 47 GiB weights/node → KV
   budget ≈ 45+ GiB/rank at GMU 0.85 → a 512K request costs 26.3 GiB/rank
   (52.7 KB/token/rank × 512K) and fits **without DCP** — sidesteps the
   DCP>1 sparse-decode bug entirely. Pin KV ~28–32 GiB (pool ≈ 550–620K
   tokens/rank); 768K needs ~39.5 GiB (fits at the top of the budget).
   + DSpark K8 drafter (2.3 GiB). This is the full-model long-context lane
   the 4x cluster cannot deliver.
2. **8x lab GLM-5.3-NVFP4 + MTP k3** (manifest policy): 433 GB → 54.1 GiB/
   node; the lab's own qualified full-model config; A/B vs (1).
3. **8x DS4.1-Flash — no disk staging** (operator decision, code-verified):
   at TP8 the two 203 GB Engram tables fit in DRAM (1/8 of the table ≈
   51 GiB/rank + weights within 121.6 GiB/node), so the 8x recipe drops
   `DSV41_ENGRAM_DISK`/`DSV41_ENGRAM_LOCAL`/`DSV41_ENGRAM_DIR` and the
   `/var/tmp/engram-local` mount entirely — the baked `engram.py` patch's
   disk mode is env-gated and defaults to off (in-memory path). The engine
   prints the per-rank layout at boot (the recipe header documents it), so
   nothing needs pre-staging. Aggregate KV ~2× (10.4M → ~20M tokens) — the
   capacity/1M-context lane for prod traffic.
Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is
qualified).

### P3 — optional, time permitting
1. **Qwen3.8-Flash-Next-NVFP4** quick trial (1–2 nodes): new family,
   catalog-updated 09-15 (image limit dropped — vision serving);
   MTP k3, GMU 0.94, gdn_decode b12x, block 16, kv bf16, PLE env vars.
2. **tonyd2wild GLM-Flash v11-dflash2** on 2x: their 262K DFlash2 stack vs
   our eugr 2x (1M) — only if the 2x lane becomes free (it must NOT be
   stopped for this).
3. **DS4.1-Flash lab-policy A/B** (MTP k2, GMU 0.91, seqs 64): requires
   downloading `deepseek-ai/DeepSeek-V4-Flash` @60d8d70 (~510 GB) — a
   multi-hour NAS download; start early if the operator wants it.

### P4 — bookkeeping
- File the lab issues (garbling bisect + 3 engine bugs from 09-13).
- End-state: per operator. Default guardrail unchanged: switch powered
  down at end **unless** the operator leaves an 8x lane serving.

## Guardrails

- **Nothing is serving right now** (the 2x eugr lane was stopped 09-18);
  all 8 nodes + `sw6_mt` are ON as prep. First serve of the night is a
  clean slate.
- Typo probe gates any DS4.1/r37 promotion (the 09-14 lesson).
- `sm121 serve --keep` for first/unknown boots; short-timeout polling.
- Image distribution is the launcher's job at serve time (`--build-node`
  + checksum-label sync) — no manual `docker save|ssh|docker load`.
- Kernel stays 6.17.0-1031 (7.0.0-1019 breaks NCCL/RoCE — never upgrade).

## Open questions for the operator

1. Which blocks matter tonight: P0 (DSpark K8 on the full model, 4x) is
   the stated wish; 8x (P2) is fully prepped and ready to go on your word.
2. After the night: power everything down, or leave an 8x lane serving?