launcher/history/night-shifts/2026-09-17.md

Night shift plan — 2026-09-17

Context since the 09-13 shift

• Prod (DS4.1 4x tonyd2wild) was restored the morning of 09-14, then stopped for the R37 investigation; the R37 typo bisect (A: no linear-b12x, B: prod spec K5, C: full prod contract) all garbled → cause is in the r37 tree code, not the flags. Boot D (engram off) never ran: the 4x cluster went dark mid-load (~07:27 UTC 09-14); operator later power-cycled dgx1+2 only and deployed glm53-flash-eugr-2x (eugr b12x nightly-20260910) on the 2nodes-dac pair. This lane is running and must NOT be stopped until the operator says so (explicit instruction 09-17).
• dev66 bench (09-14): eugr 2x = 161/176 (91), d0 c1 1,326 pp / 28.1 tg / 1,458 ms — quality lane verdict unchanged (r28 2x = prefill lane).
• R37 lane status: unqualified for DS4.1 (typos) — pending Boot D + lab issue. The bug report draft exists: specs/bugs/BUG-2026-09-14-r37-ds41-text-garbling.md.
• New tooling shipped 09-14: sm121 runlock list|close + report stale cross-check (commits 2f71763, 706b7a5).

What updated upstream (scanned 09-17)

• RedHatAI/GLM-5.3-speculator.dspark (09-16, v1.0): external DSpark K8 drafter for the FULL GLM-5.3 (zai-org base, GlmMoeDsaForCausalLM). 3-layer Qwen3-style backbone (hidden 6144, sliding window 2048), 2.5 GB bf16, aux hidden states from target layers 2/20/39/58/75, Markov head (vanilla, rank 256) + confidence head, greedy proposal, verifier_accept_k=1, 1M ctx native. Card acceptance: tool_call 4.01, math 6.13, HumanEval 4.76 tokens/ step; MRCR 524K–1M 41.16% / 4.293 avg. Deploy contract (from the card): --speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark", "num_speculative_tokens":8,"method":"dspark"}'. Trained on vllm 0.28.0 / speculators 0.8.0.dev196 — our r37 tree reports 0.26.1rc0 but has the integration (grep-verified: DSparkDraftModel method resolution in config/speculative.py:1333/1404, speculators config loader at transformers_utils/configs/speculators, draft_model field, model registered in models/registry.py). Needs --trust-remote-code (repo carries config.py custom code) — the full recipe already sets it. Drafter trained on BF16-target hidden states; we serve MXFP4 — acceptance may shift a bit; that's part of what the A/B measures.
• lil-catalog updated (snapshot bab43db6, 09-15) — the lab's qualified launch manifests. Only change: Qwen3.8-Flash-Next-NVFP4 dropped limit_per_prompt: image: 1. Extracts that matter:
  • DeepSeek-V4-Flash (rev 60d8d707): lab policy = MTP k=2 (moe mxfp4, probabilistic/standard), GMU 0.91, seqs 64, batched tokens 8192 — very different from our prod contract (dspark K5, GMU 0.85, seqs 8). NOTE: this is V4-Flash, not our V4.1-Flash; the 0731 variant carries the DSpark head.
  • DeepSeek-V4-Flash-Vision-Exp: dspark K7 + adaptive_verification — the lab qualified the vision lane we have an image for.
  • GLM-5.3-NVFP4 (lab's FULL GLM-5.3, 433 GB): NVFP4 routed experts + BF16 MTP expert layer, MTP k=3 model:target, B12X attention, long_prefill 2048, async_scheduling, FULL_AND_PIECEWISE + custom_ops all, CUDA_DEVICE_MAX_CONNECTIONS 32, no GMU pin. Their qualified full-model lane — we never ran spec on the full model.
  • GLM-5.3-Flash-DFlash2-MXFP8 (1.2 GB, kind: draft): MXFP8 DFlash2 drafter variant, compatible with the NVFP4/Spark Flash models.
• eugr: b12x line's latest tag is still nightly-20260913 (built from the lab's dev/infernal-invocation — a NEWER lab branch than our r37 tree 3fb88fdf); non-b12x line pushed nightly-20260917. Changelog highlights: Qwen3.8-Flash-Next solo recipe with VLLM_PLE_TABLE_MEMORY= disk (1M+ KV at GMU 0.8), Vision-Exp B12X + DSpark, GLM Flash b12x loader.
• vLLM 0.28.0 (08-26): DeepSeek V4 sparse MLA end-to-end for decode/MTP/ DSpark (#51538) — candidate upstream fix for our DCP>1 sparse-decode crash (512K blocker); DFlash2 loading from speculators format (#53797); DSpark confidence-scheduled verification (#47808); tiered KV offload with a disk tier; per-request spec-decode metrics (#48915). Our lab tree (0.26.1rc0) predates all of this — rebasing is still the lab's call.
• tonyd2wild expanded to GLM-5.3-Flash: ghcr.io/tonyd2wild/ vllm-glm53-flash:sm121-v11-dflash2 (+ sm121-v8 base): 5 SM121 fixes (NoPE-MLA sparse backend, FlashInfer 0.6.17→0.6.18 FA2 NaN, dep re-pins, PDL gating, indexer top-k init) + fp8 KV unlock. Measured TP2 @262K: DFlash2 46.9 tg/s (74.1% acceptance, 2.15×), TP4/1M sibling 68.5 tg/s, pool 2.62M. A third independent stack for the GLM Flash lane.
• Forum #382120 (EXL3/TR3 kit, 4bpw uncensored): 2.2M-token pool @900K ctx on 2x; their host-sync findings (per-layer torch.unique gather → ~700 ms stalls; fixed-shape no-dedup gather + raised fused temp rows) are good context for the lab issues doc.

Prep done (operator-gated: nothing stopped)

• RedHatAI/GLM-5.3-speculator.dspark downloaded to the shared HF cache (2.3 GB, snapshot 198b3ced) ✓
• r37-image grep: external-dspark support confirmed ✓
• lil-catalog read + diffed ✓

Night program

P0 — Full GLM-5.3 MXFP4 + DSpark K8 on 4x @131K (operator's pick)

1. Copy glm53-full-mxfp4-4x → glm53-full-mxfp4-4x-dspark8; keep the qualified boot ladder (linear-backend marlin, interleave 1, 131072, 8 GiB KV pin, GMU 0.85) and add the spec config from the card. draft_tensor_parallel_size — start with 4 (drafter rides the TP group); if the drafter is small enough consider tp1 later.
2. Boot probe: expect "Using DSpark…" + drafter load lines; watch the known MXFP4 hazards (MarlinExperts selection, b12x warmup).
3. Quick quality gate FIRST (typo probe: 300-word RU generation + code identifiers — the lane must produce clean text), then bench dev67-full-glm53-dspark8 vs anchor: no-spec 158/176 (90), 934 pp / 19.0 tg / 2,124 ms. Expect decode ×2–3 (acceptance ~4/step at 131K ≥ the MRCR long-context numbers), prefill unchanged.
4. Teardown: stop, restore whatever the operator wants up.

P1 — r37 typo verdict (closes the morning's bisect)

1. Boot D: r37 image, full prod contract, engram off (--engram-config removed) → typo probe. Clean → the lab's engram disk-read path is the cause; garbled → tree numerics; either way finalize BUG-2026-09-14-r37-ds41-text-garbling.md and file it with the lab.
2. eugr b12x nightly-20260913 A/B: pull (11.2 GB), same probe recipe on 4x — built from the lab's dev/infernal-invocation, one lab-branch newer than our r37. Clean text → the r37/DS4.1-alt line revives on the newer tree.
3. If both still garble: DS4.1 stays on tonyd2wild prod; R37 = GLM-lane only (where it's proven).

P2 — 8x trials (all cables are on site)

Gate first: dgx5–8 must be powered + provisioned (ssh/docker/NFS/HF cache mounts). dgx6–8 IPs (88.x ssh + 200.x ib) are unknown to the launcher — discover at night start (HA power on → arp/ssh probe → record), then add cluster 8nodes (dgx1–8, ranks 0–7, entity switch.dgxN, NCCL block copied from 4nodes-switch). Ask the operator about provisioning state.

1. 8x full GLM-5.3 MXFP4 @512K DCP1: TP8 → 47 GiB weights/node → KV budget ≈ 45+ GiB/rank at GMU 0.85 → a 512K request costs 26.3 GiB/rank (52.7 KB/token/rank × 512K) and fits without DCP — sidesteps the DCP>1 sparse-decode bug entirely. Pin KV ~28–32 GiB (pool ≈ 550–620K tokens/rank); 768K needs ~39.5 GiB (fits at the top of the budget).
   • DSpark K8 drafter (2.3 GiB). This is the full-model long-context lane the 4x cluster cannot deliver.
2. 8x lab GLM-5.3-NVFP4 + MTP k3 (manifest policy): 433 GB → 54.1 GiB/ node; the lab's own qualified full-model config; A/B vs (1).
3. 8x DS4.1-Flash: aggregate KV ~2× (10.4M → ~20M tokens) — the capacity/1M-context lane for prod traffic. Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is qualified).

P3 — optional, time permitting

1. Qwen3.8-Flash-Next-NVFP4 quick trial (1–2 nodes): new family, catalog-updated 09-15 (image limit dropped — vision serving); MTP k3, GMU 0.94, gdn_decode b12x, block 16, kv bf16, PLE env vars.
2. tonyd2wild GLM-Flash v11-dflash2 on 2x: their 262K DFlash2 stack vs our eugr 2x (1M) — only if the 2x lane becomes free (it must NOT be stopped for this).
3. DS4.1-Flash lab-policy A/B (MTP k2, GMU 0.91, seqs 64): requires downloading deepseek-ai/DeepSeek-V4-Flash 60d8d70">@60d8d70 (~510 GB) — a multi-hour NAS download; start early if the operator wants it.

P4 — bookkeeping

• File the lab issues (garbling bisect + 3 engine bugs from 09-13).
• Push the ~20 local commits when git.kilork.org returns.
• End-state: per operator. Default guardrail unchanged: 4nodes-switch powered down at end unless the operator leaves an 8x lane serving.

Guardrails

• The running 2x glm53 eugr lane (dgx1+2) is not to be stopped or restarted until the operator says so — all 4x/8x work must wait for or reuse free nodes (dgx3/4 are off; the 4x P0/P1 blocks need them).
• Typo probe gates any DS4.1/r37 promotion (the 09-14 lesson).
• sm121 serve --keep for first/unknown boots; short-timeout polling.
• Build heavy images on dgx5 only; docker save|ssh|docker load for distribution.
• Kernel stays 6.17.0-1031 (7.0.0-1019 breaks NCCL/RoCE — never upgrade).

Open questions for the operator

1. Are dgx6–8 OS-provisioned (ssh kilork, docker, /mnt/nas mount, HF cache path)? Any known IPs? (Needed to define the 8nodes cluster config.)
2. Which blocks matter tonight: P0 (DSpark K8 on the full model) is the stated wish; 8x needs your go + node provisioning.
3. After the night: leave the 2x eugr lane serving, or power everything down?