صفحهٔ اصلی گشت‌و‌گذار راهنما
ورود
sm121
/
launcher
2
0
انشعاب 0
پرونده‌ها مشکلات 0 درخواست واکشی 0 ویکی
درخت: bc5aa791bf
شاخه‌ها تگ‌ها
launcher
/
history
/
night-shifts
/
2026-09-18.md

2026-09-18.md 12 KB
تاريخچه خام

Night shift plan — 2026-09-18

Context since the 09-13 shift

  • Prod (DS4.1 4x tonyd2wild) was restored the morning of 09-14, then stopped for the R37 investigation; the R37 typo bisect (A: no linear-b12x, B: prod spec K5, C: full prod contract) all garbled → cause is in the r37 tree code, not the flags. Boot D (engram off) never ran: the 4x cluster went dark mid-load (~07:27 UTC 09-14); operator later power-cycled dgx1+2 only and deployed glm53-flash-eugr-2x (eugr b12x nightly-20260910) on the 2nodes-dac pair. This lane is running and must NOT be stopped until the operator says so (explicit instruction 09-17).
  • dev66 bench (09-14): eugr 2x = 161/176 (91), d0 c1 1,326 pp / 28.1 tg / 1,458 ms — quality lane verdict unchanged (r28 2x = prefill lane).
  • 09-18 morning: the operator stopped the 2x eugr lane and powered the whole cluster down while he provisioned the four new nodes (dgx5–8). As of this plan there is nothing serving and all 8 nodes are powered on with sw6_mt on (prepared for the 8x work, see below).
  • R37 lane status: unqualified for DS4.1 (typos) — pending Boot D + lab issue. The bug report draft exists: specs/bugs/BUG-2026-09-14-r37-ds41-text-garbling.md.
  • New tooling shipped 09-14: sm121 runlock list|close + report stale cross-check (commits 2f71763, 706b7a5).

What updated upstream (scanned 09-17)

  • RedHatAI/GLM-5.3-speculator.dspark (09-16, v1.0): external DSpark K8 drafter for the FULL GLM-5.3 (zai-org base, GlmMoeDsaForCausalLM). 3-layer Qwen3-style backbone (hidden 6144, sliding window 2048), 2.5 GB bf16, aux hidden states from target layers 2/20/39/58/75, Markov head (vanilla, rank 256) + confidence head, greedy proposal, verifier_accept_k=1, 1M ctx native. Card acceptance: tool_call 4.01, math 6.13, HumanEval 4.76 tokens/ step; MRCR 524K–1M 41.16% / 4.293 avg. Deploy contract (from the card): --speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark", "num_speculative_tokens":8,"method":"dspark"}'. Trained on vllm 0.28.0 / speculators 0.8.0.dev196 — our r37 tree reports 0.26.1rc0 but has the integration (grep-verified: DSparkDraftModel method resolution in config/speculative.py:1333/1404, speculators config loader at transformers_utils/configs/speculators, draft_model field, model registered in models/registry.py). Needs --trust-remote-code (repo carries config.py custom code) — the full recipe already sets it. Drafter trained on BF16-target hidden states; we serve MXFP4 — acceptance may shift a bit; that's part of what the A/B measures.
  • lil-catalog updated (snapshot bab43db6, 09-15) — the lab's qualified launch manifests. Only change: Qwen3.8-Flash-Next-NVFP4 dropped limit_per_prompt: image: 1. Extracts that matter:
    • DeepSeek-V4-Flash (rev 60d8d707): lab policy = MTP k=2 (moe mxfp4, probabilistic/standard), GMU 0.91, seqs 64, batched tokens 8192 — very different from our prod contract (dspark K5, GMU 0.85, seqs 8). NOTE: this is V4-Flash, not our V4.1-Flash; the 0731 variant carries the DSpark head.
    • DeepSeek-V4-Flash-Vision-Exp: dspark K7 + adaptive_verification — the lab qualified the vision lane we have an image for.
    • GLM-5.3-NVFP4 (lab's FULL GLM-5.3, 433 GB): NVFP4 routed experts + BF16 MTP expert layer, MTP k=3 model:target, B12X attention, long_prefill 2048, async_scheduling, FULL_AND_PIECEWISE + custom_ops all, CUDA_DEVICE_MAX_CONNECTIONS 32, no GMU pin. Their qualified full-model lane — we never ran spec on the full model.
    • GLM-5.3-Flash-DFlash2-MXFP8 (1.2 GB, kind: draft): MXFP8 DFlash2 drafter variant, compatible with the NVFP4/Spark Flash models.
  • eugr: b12x line's latest tag is still nightly-20260913 (built from the lab's dev/infernal-invocation — a NEWER lab branch than our r37 tree 3fb88fdf); non-b12x line pushed nightly-20260917. Changelog highlights: Qwen3.8-Flash-Next solo recipe with VLLM_PLE_TABLE_MEMORY= disk (1M+ KV at GMU 0.8), Vision-Exp B12X + DSpark, GLM Flash b12x loader.
  • vLLM 0.28.0 (08-26): DeepSeek V4 sparse MLA end-to-end for decode/MTP/ DSpark (#51538) — candidate upstream fix for our DCP>1 sparse-decode crash (512K blocker); DFlash2 loading from speculators format (#53797); DSpark confidence-scheduled verification (#47808); tiered KV offload with a disk tier; per-request spec-decode metrics (#48915). Our lab tree (0.26.1rc0) predates all of this — rebasing is still the lab's call.
  • tonyd2wild expanded to GLM-5.3-Flash: ghcr.io/tonyd2wild/ vllm-glm53-flash:sm121-v11-dflash2 (+ sm121-v8 base): 5 SM121 fixes (NoPE-MLA sparse backend, FlashInfer 0.6.17→0.6.18 FA2 NaN, dep re-pins, PDL gating, indexer top-k init) + fp8 KV unlock. Measured TP2 @262K: DFlash2 46.9 tg/s (74.1% acceptance, 2.15×), TP4/1M sibling 68.5 tg/s, pool 2.62M. A third independent stack for the GLM Flash lane.
  • Forum #382120 (EXL3/TR3 kit, 4bpw uncensored): 2.2M-token pool @900K ctx on 2x; their host-sync findings (per-layer torch.unique gather → ~700 ms stalls; fixed-shape no-dedup gather + raised fused temp rows) are good context for the lab issues doc.

Prep done (09-18)

  • RedHatAI/GLM-5.3-speculator.dspark downloaded to the shared HF cache (2.3 GB, snapshot 198b3ced) ✓
  • r37-image grep: external-dspark support confirmed ✓
  • lil-catalog read + diffed ✓
  • git.kilork.org recovered — the ~20 queued local commits pushed ✓

8x readiness (done 09-18 daytime, verified live)

  • All 8 nodes up (dgx5–8 fresh), sw6_mt turned on via the HA relay — the 200G fabric is invisible without the switch (operator: the switch relay is a separate HA entity, not implied by node power).
  • The "two IPs on the IB interface" question is solved: every Spark has TWO dual-port 200G NICs (enp1s0 + enP2p1s0); port0 of each carries a 200.x IP and both are on ONE L2 domain (a single 16-port 200G switch, cross-ping verified with sudo ping). Dual-rail 2×200G per node is already the current 4nodes-switch convention (NCCL_IB_HCA=rocep1s0f0, roceP2p1s0f0). Per-node ib_ip = NIC A port0: dgx5=200.11, dgx6=200.10, dgx7=200.23, dgx8=200.25 (plus the 100.x/101.x DAC pair on dgx1/2 port1, untouched). All 16 switch ports link UP.
  • dgx5/6: /mnt/nas (nfs4 nas.lan:/sna, nconnect=8) validated ✓. dgx7/8: fixed — the fstab line + /mnt/nas were missing; added (sudo available, NAS exports /sna *), mounted, 65 HF repos visible ✓.
  • 8nodes-switch cluster added to config.toml (dgx1–8, ranks 0–7, entities switch.dgxN, sw6_mt blocking + sw7_fan non-blocking) with [nccl."8nodes-switch"] = the dual-rail copy of 4nodes-switch. All 8 nodes share one ED25519 host key (cloned Ubuntu image) — same pinned SHA256:aUXh… as dgx1–4; (security footnote: per-node key regen later).
  • sm121 health --cluster 8nodes-switch: 8/8 online, all free ✓.
  • Dry-run serve --cluster 8nodes-switch --fit 8 renders correctly: TP8/nnodes 8/ranks 0–7, GID auto-detect on all 8, before_serve hooks ×8, MASTER_ADDR=200.15 ✓.
  • Images: dgx5 is fully pre-seeded (r37/r36/r35 + both dsv41); dgx6–8 have no vLLM image yet — the launcher syncs them natively at serve (build on --build-node + checksum-label sync; no manual docker save|ssh|docker load — an abandoned manual copy attempt left no residue, verified).

Night program

P0 — Full GLM-5.3 MXFP4 + DSpark K8 on 4x @131K (operator's pick)

  1. Copy glm53-full-mxfp4-4x → glm53-full-mxfp4-4x-dspark8; keep the qualified boot ladder (linear-backend marlin, interleave 1, 131072, 8 GiB KV pin, GMU 0.85) and add the spec config from the card. draft_tensor_parallel_size — start with 4 (drafter rides the TP group); if the drafter is small enough consider tp1 later.
  2. Boot probe: expect "Using DSpark…" + drafter load lines; watch the known MXFP4 hazards (MarlinExperts selection, b12x warmup).
  3. Quick quality gate FIRST (typo probe: 300-word RU generation + code identifiers — the lane must produce clean text), then bench dev67-full-glm53-dspark8 vs anchor: no-spec 158/176 (90), 934 pp / 19.0 tg / 2,124 ms. Expect decode ×2–3 (acceptance ~4/step at 131K ≥ the MRCR long-context numbers), prefill unchanged.
  4. Teardown: stop, restore whatever the operator wants up.

P1 — r37 typo verdict (closes the morning's bisect)

  1. Boot D: r37 image, full prod contract, engram off (--engram-config removed) → typo probe. Clean → the lab's engram disk-read path is the cause; garbled → tree numerics; either way finalize BUG-2026-09-14-r37-ds41-text-garbling.md and file it with the lab.
  2. eugr b12x nightly-20260913 A/B: pull (11.2 GB), same probe recipe on 4x — built from the lab's dev/infernal-invocation, one lab-branch newer than our r37. Clean text → the r37/DS4.1-alt line revives on the newer tree.
  3. If both still garble: DS4.1 stays on tonyd2wild prod; R37 = GLM-lane only (where it's proven).

P2 — 8x trials (cables on site; nodes provisioned, cluster configured)

Gate: CLEARED — all 8 nodes up + validated, 8nodes-switch in config, health green. The launcher syncs missing images at serve time.

  1. 8x full GLM-5.3 MXFP4 @512K DCP1: TP8 → 47 GiB weights/node → KV budget ≈ 45+ GiB/rank at GMU 0.85 → a 512K request costs 26.3 GiB/rank (52.7 KB/token/rank × 512K) and fits without DCP — sidesteps the DCP>1 sparse-decode bug entirely. Pin KV ~28–32 GiB (pool ≈ 550–620K tokens/rank); 768K needs ~39.5 GiB (fits at the top of the budget).
    • DSpark K8 drafter (2.3 GiB). This is the full-model long-context lane the 4x cluster cannot deliver.
  2. 8x lab GLM-5.3-NVFP4 + MTP k3 (manifest policy): 433 GB → 54.1 GiB/ node; the lab's own qualified full-model config; A/B vs (1).
  3. 8x DS4.1-Flash — no disk staging (operator decision, code-verified): at TP8 the two 203 GB Engram tables fit in DRAM (1/8 of the table ≈ 51 GiB/rank + weights within 121.6 GiB/node), so the 8x recipe drops DSV41_ENGRAM_DISK/DSV41_ENGRAM_LOCAL/DSV41_ENGRAM_DIR and the /var/tmp/engram-local mount entirely — the baked engram.py patch's disk mode is env-gated and defaults to off (in-memory path). The engine prints the per-rank layout at boot (the recipe header documents it), so nothing needs pre-staging. Aggregate KV ~2× (10.4M → ~20M tokens) — the capacity/1M-context lane for prod traffic. Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is qualified).

P3 — optional, time permitting

  1. Qwen3.8-Flash-Next-NVFP4 quick trial (1–2 nodes): new family, catalog-updated 09-15 (image limit dropped — vision serving); MTP k3, GMU 0.94, gdn_decode b12x, block 16, kv bf16, PLE env vars.
  2. tonyd2wild GLM-Flash v11-dflash2 on 2x: their 262K DFlash2 stack vs our eugr 2x (1M) — only if the 2x lane becomes free (it must NOT be stopped for this).
  3. DS4.1-Flash lab-policy A/B (MTP k2, GMU 0.91, seqs 64): requires downloading deepseek-ai/DeepSeek-V4-Flash 60d8d70">@60d8d70 (~510 GB) — a multi-hour NAS download; start early if the operator wants it.

P4 — bookkeeping

  • File the lab issues (garbling bisect + 3 engine bugs from 09-13).
  • End-state: per operator. Default guardrail unchanged: switch powered down at end unless the operator leaves an 8x lane serving.

Guardrails

  • Nothing is serving right now (the 2x eugr lane was stopped 09-18); all 8 nodes + sw6_mt are ON as prep. First serve of the night is a clean slate.
  • Typo probe gates any DS4.1/r37 promotion (the 09-14 lesson).
  • sm121 serve --keep for first/unknown boots; short-timeout polling.
  • Image distribution is the launcher's job at serve time (--build-node
    • checksum-label sync) — no manual docker save|ssh|docker load.
  • Kernel stays 6.17.0-1031 (7.0.0-1019 breaks NCCL/RoCE — never upgrade).

Open questions for the operator

  1. Which blocks matter tonight: P0 (DSpark K8 on the full model, 4x) is the stated wish; 8x (P2) is fully prepped and ready to go on your word.
  2. After the night: power everything down, or leave an 8x lane serving?