# Night shift plan — 2026-09-18

## Context since the 09-13 shift

- Prod (DS4.1 4x tonyd2wild) was restored the morning of 09-14, then stopped
  for the R37 investigation; the R37 typo bisect (A: no linear-b12x, B: prod
  spec K5, C: full prod contract) all **garbled** → cause is in the r37 tree
  code, not the flags. Boot D (engram off) never ran: the 4x cluster went
  dark mid-load (~07:27 UTC 09-14); operator later power-cycled dgx1+2 only
  and deployed `glm53-flash-eugr-2x` (eugr b12x nightly-20260910) on the
  2nodes-dac pair. **This lane is running and must NOT be stopped** until
  the operator says so (explicit instruction 09-17).
- dev66 bench (09-14): eugr 2x = **161/176 (91)**, d0 c1 1,326 pp / 28.1 tg /
  1,458 ms — quality lane verdict unchanged (r28 2x = prefill lane).
- 09-18 morning: the operator stopped the 2x eugr lane and powered the whole
  cluster down while he provisioned the four new nodes (dgx5–8). As of this
  plan there is **nothing serving** and **all 8 nodes are powered on** with
  `sw6_mt` on (prepared for the 8x work, see below).
- R37 lane status: unqualified for DS4.1 (typos) — pending Boot D + lab
  issue. The bug report draft exists:
  `specs/bugs/BUG-2026-09-14-r37-ds41-text-garbling.md`.
- New tooling shipped 09-14: `sm121 runlock list|close` + report stale
  cross-check (commits `2f71763`, `706b7a5`).

## What updated upstream (scanned 09-17)

- **RedHatAI/GLM-5.3-speculator.dspark** (09-16, v1.0): external DSpark K8
  drafter for the FULL GLM-5.3 (zai-org base, GlmMoeDsaForCausalLM). 3-layer
  Qwen3-style backbone (hidden 6144, sliding window 2048), 2.5 GB bf16,
  aux hidden states from target layers 2/20/39/58/75, Markov head (vanilla,
  rank 256) + confidence head, greedy proposal, verifier_accept_k=1, 1M ctx
  native. Card acceptance: tool_call 4.01, math 6.13, HumanEval 4.76 tokens/
  step; MRCR 524K–1M 41.16% / 4.293 avg. Deploy contract (from the card):
  `--speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark",
  "num_speculative_tokens":8,"method":"dspark"}'`. Trained on vllm
  0.28.0 / speculators 0.8.0.dev196 — our r37 tree reports 0.26.1rc0 but
  **has the integration** (grep-verified: DSparkDraftModel method
  resolution in `config/speculative.py:1333/1404`, speculators config
  loader at `transformers_utils/configs/speculators`, draft_model field,
  model registered in `models/registry.py`). Needs `--trust-remote-code`
  (repo carries config.py custom code) — the full recipe already sets it.
  Drafter trained on BF16-target hidden states; we serve MXFP4 — acceptance
  may shift a bit; that's part of what the A/B measures.
- **lil-catalog updated (snapshot bab43db6, 09-15)** — the lab's qualified
  launch manifests. Only change: `Qwen3.8-Flash-Next-NVFP4` dropped
  `limit_per_prompt: image: 1`. Extracts that matter:
  - `DeepSeek-V4-Flash` (rev 60d8d707): lab policy = **MTP k=2** (moe mxfp4,
    probabilistic/standard), GMU **0.91**, seqs **64**, batched tokens 8192 —
    very different from our prod contract (dspark K5, GMU 0.85, seqs 8).
    NOTE: this is V4-Flash, not our V4.1-Flash; the 0731 variant carries the
    DSpark head.
  - `DeepSeek-V4-Flash-Vision-Exp`: dspark K7 + adaptive_verification — the
    lab qualified the vision lane we have an image for.
  - `GLM-5.3-NVFP4` (lab's FULL GLM-5.3, 433 GB): NVFP4 routed experts +
    BF16 MTP expert layer, **MTP k=3 model:target**, B12X attention,
    long_prefill 2048, async_scheduling, FULL_AND_PIECEWISE + custom_ops
    all, CUDA_DEVICE_MAX_CONNECTIONS 32, no GMU pin. Their qualified
    full-model lane — we never ran spec on the full model.
  - `GLM-5.3-Flash-DFlash2-MXFP8` (1.2 GB, kind: draft): MXFP8 DFlash2
    drafter variant, compatible with the NVFP4/Spark Flash models.
- **eugr**: b12x line's latest tag is still `nightly-20260913` (built from
  the lab's **dev/infernal-invocation** — a NEWER lab branch than our r37
  tree 3fb88fdf); non-b12x line pushed `nightly-20260917`. Changelog
  highlights: Qwen3.8-Flash-Next solo recipe with `VLLM_PLE_TABLE_MEMORY=
  disk` (1M+ KV at GMU 0.8), Vision-Exp B12X + DSpark, GLM Flash b12x
  loader.
- **vLLM 0.28.0** (08-26): DeepSeek V4 sparse MLA end-to-end for decode/MTP/
  DSpark (#51538) — candidate upstream fix for our DCP>1 sparse-decode
  crash (512K blocker); DFlash2 loading from speculators format (#53797);
  DSpark confidence-scheduled verification (#47808); tiered KV offload
  with a disk tier; per-request spec-decode metrics (#48915). Our lab tree
  (0.26.1rc0) predates all of this — rebasing is still the lab's call.
- **tonyd2wild** expanded to GLM-5.3-Flash: `ghcr.io/tonyd2wild/
  vllm-glm53-flash:sm121-v11-dflash2` (+ `sm121-v8` base): 5 SM121 fixes
  (NoPE-MLA sparse backend, FlashInfer 0.6.17→0.6.18 FA2 NaN, dep re-pins,
  PDL gating, indexer top-k init) + fp8 KV unlock. Measured TP2 @262K:
  DFlash2 46.9 tg/s (74.1% acceptance, 2.15×), TP4/1M sibling 68.5 tg/s,
  pool 2.62M. A third independent stack for the GLM Flash lane.
- Forum #382120 (EXL3/TR3 kit, 4bpw uncensored): 2.2M-token pool @900K ctx
  on 2x; their host-sync findings (per-layer `torch.unique` gather →
  ~700 ms stalls; fixed-shape no-dedup gather + raised fused temp rows)
  are good context for the lab issues doc.

## Prep done (09-18)

- `RedHatAI/GLM-5.3-speculator.dspark` downloaded to the shared HF cache
  (2.3 GB, snapshot 198b3ced) ✓
- r37-image grep: external-dspark support confirmed ✓
- lil-catalog read + diffed ✓
- git.kilork.org recovered — the ~20 queued local commits pushed ✓

### 8x readiness (done 09-18 daytime, verified live)

- All 8 nodes up (dgx5–8 fresh), `sw6_mt` turned on via the HA relay —
  the 200G fabric is invisible without the switch (operator: the switch
  relay is a separate HA entity, not implied by node power).
- **The "two IPs on the IB interface" question is solved**: every Spark
  has TWO dual-port 200G NICs (`enp1s0` + `enP2p1s0`); port0 of each carries
  a 200.x IP and both are on ONE L2 domain (a single 16-port 200G switch,
  cross-ping verified with sudo ping). Dual-rail 2×200G per node is already
  the current `4nodes-switch` convention (`NCCL_IB_HCA=rocep1s0f0,
  roceP2p1s0f0`). Per-node `ib_ip` = NIC A port0: dgx5=200.11, dgx6=200.10,
  dgx7=200.23, dgx8=200.25 (plus the 100.x/101.x DAC pair on dgx1/2 port1,
  untouched). All 16 switch ports link UP.
- dgx5/6: `/mnt/nas` (nfs4 `nas.lan:/sna`, nconnect=8) validated ✓. dgx7/8:
  **fixed** — the fstab line + `/mnt/nas` were missing; added (sudo
  available, NAS exports `/sna *`), mounted, 65 HF repos visible ✓.
- `8nodes-switch` cluster added to `config.toml` (dgx1–8, ranks 0–7,
  entities `switch.dgxN`, `sw6_mt` blocking + `sw7_fan` non-blocking) with
  `[nccl."8nodes-switch"]` = the dual-rail copy of `4nodes-switch`. All 8
  nodes share one ED25519 host key (cloned Ubuntu image) — same pinned
  `SHA256:aUXh…` as dgx1–4; (security footnote: per-node key regen later).
- `sm121 health --cluster 8nodes-switch`: **8/8 online, all free** ✓.
- Dry-run `serve --cluster 8nodes-switch --fit 8` renders correctly:
  TP8/nnodes 8/ranks 0–7, GID auto-detect on all 8, before_serve hooks ×8,
  MASTER_ADDR=200.15 ✓.
- Images: dgx5 is fully pre-seeded (r37/r36/r35 + both dsv41); dgx6–8 have
  no vLLM image yet — **the launcher syncs them natively at serve**
  (build on `--build-node` + checksum-label sync; no manual
  `docker save|ssh|docker load` — an abandoned manual copy attempt left no
  residue, verified).

## Night program

### P0 — Full GLM-5.3 MXFP4 + DSpark K8 on 4x @131K (operator's pick)
1. Copy `glm53-full-mxfp4-4x` → `glm53-full-mxfp4-4x-dspark8`; keep the
   qualified boot ladder (linear-backend marlin, interleave 1, 131072,
   8 GiB KV pin, GMU 0.85) and add the spec config from the card.
   `draft_tensor_parallel_size` — start with 4 (drafter rides the TP
   group); if the drafter is small enough consider tp1 later.
2. Boot probe: expect "Using DSpark…" + drafter load lines; watch the
   known MXFP4 hazards (MarlinExperts selection, b12x warmup).
3. Quick quality gate FIRST (typo probe: 300-word RU generation + code
   identifiers — the lane must produce clean text), then bench
   `dev67-full-glm53-dspark8` vs anchor: no-spec 158/176 (90),
   934 pp / 19.0 tg / 2,124 ms. Expect decode ×2–3 (acceptance ~4/step at
   131K ≥ the MRCR long-context numbers), prefill unchanged.
4. Teardown: stop, restore whatever the operator wants up.

### P1 — r37 typo verdict (closes the morning's bisect)
1. **Boot D**: r37 image, full prod contract, engram off (`--engram-config`
   removed) → typo probe. Clean → the lab's engram disk-read path is the
   cause; garbled → tree numerics; either way finalize
   `BUG-2026-09-14-r37-ds41-text-garbling.md` and file it with the lab.
2. **eugr b12x nightly-20260913 A/B**: pull (11.2 GB), same probe recipe on
   4x — built from the lab's `dev/infernal-invocation`, one lab-branch
   newer than our r37. Clean text → the r37/DS4.1-alt line revives on the
   newer tree.
3. If both still garble: DS4.1 stays on tonyd2wild prod; R37 = GLM-lane
   only (where it's proven).

### P2 — 8x trials (cables on site; nodes provisioned, cluster configured)
Gate: CLEARED — all 8 nodes up + validated, `8nodes-switch` in config,
health green. The launcher syncs missing images at serve time.
1. **8x full GLM-5.3 MXFP4 @512K DCP1**: TP8 → 47 GiB weights/node → KV
   budget ≈ 45+ GiB/rank at GMU 0.85 → a 512K request costs 26.3 GiB/rank
   (52.7 KB/token/rank × 512K) and fits **without DCP** — sidesteps the
   DCP>1 sparse-decode bug entirely. Pin KV ~28–32 GiB (pool ≈ 550–620K
   tokens/rank); 768K needs ~39.5 GiB (fits at the top of the budget).
   + DSpark K8 drafter (2.3 GiB). This is the full-model long-context lane
   the 4x cluster cannot deliver.
2. **8x lab GLM-5.3-NVFP4 + MTP k3** (manifest policy): 433 GB → 54.1 GiB/
   node; the lab's own qualified full-model config; A/B vs (1).
3. **8x DS4.1-Flash — no disk staging** (operator decision, code-verified):
   at TP8 the two 203 GB Engram tables fit in DRAM (1/8 of the table ≈
   51 GiB/rank + weights within 121.6 GiB/node), so the 8x recipe drops
   `DSV41_ENGRAM_DISK`/`DSV41_ENGRAM_LOCAL`/`DSV41_ENGRAM_DIR` and the
   `/var/tmp/engram-local` mount entirely — the baked `engram.py` patch's
   disk mode is env-gated and defaults to off (in-memory path). The engine
   prints the per-rank layout at boot (the recipe header documents it), so
   nothing needs pre-staging. Aggregate KV ~2× (10.4M → ~20M tokens) — the
   capacity/1M-context lane for prod traffic.
Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is
qualified).

### P3 — optional, time permitting
1. **Qwen3.8-Flash-Next-NVFP4** quick trial (1–2 nodes): new family,
   catalog-updated 09-15 (image limit dropped — vision serving);
   MTP k3, GMU 0.94, gdn_decode b12x, block 16, kv bf16, PLE env vars.
2. **tonyd2wild GLM-Flash v11-dflash2** on 2x: their 262K DFlash2 stack vs
   our eugr 2x (1M) — only if the 2x lane becomes free (it must NOT be
   stopped for this).
3. **DS4.1-Flash lab-policy A/B** (MTP k2, GMU 0.91, seqs 64): requires
   downloading `deepseek-ai/DeepSeek-V4-Flash` @60d8d70 (~510 GB) — a
   multi-hour NAS download; start early if the operator wants it.

### P4 — bookkeeping
- File the lab issues (garbling bisect + 3 engine bugs from 09-13).
- End-state: per operator. Default guardrail unchanged: switch powered
  down at end **unless** the operator leaves an 8x lane serving.

## Guardrails

- **Nothing is serving right now** (the 2x eugr lane was stopped 09-18);
  all 8 nodes + `sw6_mt` are ON as prep. First serve of the night is a
  clean slate.
- Typo probe gates any DS4.1/r37 promotion (the 09-14 lesson).
- `sm121 serve --keep` for first/unknown boots; short-timeout polling.
- Image distribution is the launcher's job at serve time (`--build-node`
  + checksum-label sync) — no manual `docker save|ssh|docker load`.
- Kernel stays 6.17.0-1031 (7.0.0-1019 breaks NCCL/RoCE — never upgrade).

## Open questions for the operator

1. Which blocks matter tonight: P0 (DSpark K8 on the full model, 4x) is
   the stated wish; 8x (P2) is fully prepped and ready to go on your word.
2. After the night: power everything down, or leave an 8x lane serving?