Эхлэл Бүгдийг харах Тусламж
Нэвтрэх
sm121
/
launcher
2
0
Салаа 0
Файлууд Асуудлууд 0 Хуулах хүсэлтүүд 0 Мэдлэгийн сан
Мод: 4b8e9815bc
Салаанууд Тагууд
launcher
/
history
/
night-shifts
/
2026-09-18.md

2026-09-18.md 15 KB
Түүх Анхны өгөгдөл

Night shift plan — 2026-09-18

Context since the 09-13 shift

  • Prod (DS4.1 4x tonyd2wild) was restored the morning of 09-14, then stopped for the R37 investigation; the R37 typo bisect (A: no linear-b12x, B: prod spec K5, C: full prod contract) all garbled → cause is in the r37 tree code, not the flags. Boot D (engram off) never ran: the 4x cluster went dark mid-load (~07:27 UTC 09-14); operator later power-cycled dgx1+2 only and deployed glm53-flash-eugr-2x (eugr b12x nightly-20260910) on the 2nodes-dac pair. This lane is running and must NOT be stopped until the operator says so (explicit instruction 09-17).
  • dev66 bench (09-14): eugr 2x = 161/176 (91), d0 c1 1,326 pp / 28.1 tg / 1,458 ms — quality lane verdict unchanged (r28 2x = prefill lane).
  • 09-18 morning: the operator stopped the 2x eugr lane and powered the whole cluster down while he provisioned the four new nodes (dgx5–8). As of this plan there is nothing serving and all 8 nodes are powered on with sw6_mt on (prepared for the 8x work, see below).
  • R37 lane status: unqualified for DS4.1 (typos) — pending Boot D + lab issue. The bug report draft exists: specs/bugs/BUG-2026-09-14-r37-ds41-text-garbling.md.
  • New tooling shipped 09-14: sm121 runlock list|close + report stale cross-check (commits 2f71763, 706b7a5).

What updated upstream (scanned 09-17)

  • RedHatAI/GLM-5.3-speculator.dspark (09-16, v1.0): external DSpark K8 drafter for the FULL GLM-5.3 (zai-org base, GlmMoeDsaForCausalLM). 3-layer Qwen3-style backbone (hidden 6144, sliding window 2048), 2.5 GB bf16, aux hidden states from target layers 2/20/39/58/75, Markov head (vanilla, rank 256) + confidence head, greedy proposal, verifier_accept_k=1, 1M ctx native. Card acceptance: tool_call 4.01, math 6.13, HumanEval 4.76 tokens/ step; MRCR 524K–1M 41.16% / 4.293 avg. Deploy contract (from the card): --speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark", "num_speculative_tokens":8,"method":"dspark"}'. Trained on vllm 0.28.0 / speculators 0.8.0.dev196 — our r37 tree reports 0.26.1rc0 but has the integration (grep-verified: DSparkDraftModel method resolution in config/speculative.py:1333/1404, speculators config loader at transformers_utils/configs/speculators, draft_model field, model registered in models/registry.py). Needs --trust-remote-code (repo carries config.py custom code) — the full recipe already sets it. Drafter trained on BF16-target hidden states; we serve MXFP4 — acceptance may shift a bit; that's part of what the A/B measures.
  • lil-catalog updated (snapshot bab43db6, 09-15) — the lab's qualified launch manifests. Only change: Qwen3.8-Flash-Next-NVFP4 dropped limit_per_prompt: image: 1. Extracts that matter:
    • DeepSeek-V4-Flash (rev 60d8d707): lab policy = MTP k=2 (moe mxfp4, probabilistic/standard), GMU 0.91, seqs 64, batched tokens 8192 — very different from our prod contract (dspark K5, GMU 0.85, seqs 8). NOTE: this is V4-Flash, not our V4.1-Flash; the 0731 variant carries the DSpark head.
    • DeepSeek-V4-Flash-Vision-Exp: dspark K7 + adaptive_verification — the lab qualified the vision lane we have an image for.
    • GLM-5.3-NVFP4 (lab's FULL GLM-5.3, 433 GB): NVFP4 routed experts + BF16 MTP expert layer, MTP k=3 model:target, B12X attention, long_prefill 2048, async_scheduling, FULL_AND_PIECEWISE + custom_ops all, CUDA_DEVICE_MAX_CONNECTIONS 32, no GMU pin. Their qualified full-model lane — we never ran spec on the full model.
    • GLM-5.3-Flash-DFlash2-MXFP8 (1.2 GB, kind: draft): MXFP8 DFlash2 drafter variant, compatible with the NVFP4/Spark Flash models.
  • eugr: b12x line's latest tag is still nightly-20260913 (built from the lab's dev/infernal-invocation — a NEWER lab branch than our r37 tree 3fb88fdf); non-b12x line pushed nightly-20260917. Changelog highlights: Qwen3.8-Flash-Next solo recipe with VLLM_PLE_TABLE_MEMORY= disk (1M+ KV at GMU 0.8), Vision-Exp B12X + DSpark, GLM Flash b12x loader.
  • vLLM 0.28.0 (08-26): DeepSeek V4 sparse MLA end-to-end for decode/MTP/ DSpark (#51538) — candidate upstream fix for our DCP>1 sparse-decode crash (512K blocker); DFlash2 loading from speculators format (#53797); DSpark confidence-scheduled verification (#47808); tiered KV offload with a disk tier; per-request spec-decode metrics (#48915). Our lab tree (0.26.1rc0) predates all of this — rebasing is still the lab's call.
  • tonyd2wild expanded to GLM-5.3-Flash: ghcr.io/tonyd2wild/ vllm-glm53-flash:sm121-v11-dflash2 (+ sm121-v8 base): 5 SM121 fixes (NoPE-MLA sparse backend, FlashInfer 0.6.17→0.6.18 FA2 NaN, dep re-pins, PDL gating, indexer top-k init) + fp8 KV unlock. Measured TP2 @262K: DFlash2 46.9 tg/s (74.1% acceptance, 2.15×), TP4/1M sibling 68.5 tg/s, pool 2.62M. A third independent stack for the GLM Flash lane.
  • Forum #382120 (EXL3/TR3 kit, 4bpw uncensored): 2.2M-token pool @900K ctx on 2x; their host-sync findings (per-layer torch.unique gather → ~700 ms stalls; fixed-shape no-dedup gather + raised fused temp rows) are good context for the lab issues doc.

Prep done (09-18)

  • RedHatAI/GLM-5.3-speculator.dspark downloaded to the shared HF cache (2.3 GB, snapshot 198b3ced) ✓
  • r37-image grep: external-dspark support confirmed ✓
  • lil-catalog read + diffed ✓
  • git.kilork.org recovered — the ~20 queued local commits pushed ✓

8x readiness (done 09-18 daytime, verified live)

  • All 8 nodes up (dgx5–8 fresh), sw6_mt turned on via the HA relay — the 200G fabric is invisible without the switch (operator: the switch relay is a separate HA entity, not implied by node power).
  • The "two IPs on the IB interface" question is solved: every Spark has TWO dual-port 200G NICs (enp1s0 + enP2p1s0); port0 of each carries a 200.x IP and both are on ONE L2 domain (a single 16-port 200G switch, cross-ping verified with sudo ping). Dual-rail 2×200G per node is already the current 4nodes-switch convention (NCCL_IB_HCA=rocep1s0f0, roceP2p1s0f0). Per-node ib_ip = NIC A port0: dgx5=200.11, dgx6=200.10, dgx7=200.23, dgx8=200.25 (plus the 100.x/101.x DAC pair on dgx1/2 port1, untouched). All 16 switch ports link UP.
  • dgx5/6: /mnt/nas (nfs4 nas.lan:/sna, nconnect=8) validated ✓. dgx7/8: fixed — the fstab line + /mnt/nas were missing; added (sudo available, NAS exports /sna *), mounted, 65 HF repos visible ✓.
  • 8nodes-switch cluster added to config.toml (dgx1–8, ranks 0–7, entities switch.dgxN, sw6_mt blocking + sw7_fan non-blocking) with [nccl."8nodes-switch"] = the dual-rail copy of 4nodes-switch. All 8 nodes share one ED25519 host key (cloned Ubuntu image) — same pinned SHA256:aUXh… as dgx1–4; (security footnote: per-node key regen later).
  • sm121 health --cluster 8nodes-switch: 8/8 online, all free ✓.
  • Dry-run serve --cluster 8nodes-switch --fit 8 renders correctly: TP8/nnodes 8/ranks 0–7, GID auto-detect on all 8, before_serve hooks ×8, MASTER_ADDR=200.15 ✓.
  • Images: dgx5 is fully pre-seeded (r37/r36/r35 + both dsv41); dgx6–8 have no vLLM image yet — the launcher syncs them natively at serve (build on --build-node + checksum-label sync; no manual docker save|ssh|docker load — an abandoned manual copy attempt left no residue, verified).

Night program

P0 — Full GLM-5.3 MXFP4 + DSpark K8 on 4x @131K (operator's pick)

STATUS (09-18 ~13:45 UTC): recipe glm53-full-mxfp4-4x-dspark8 up on 4nodes-switch, quality gate PASSED (RU 300-word text clean, 6/6 identifier probe), bench dspark8 running.

Boot log (6 boots, 12:21 → 13:34):

  • Boot 1-2: the draft model's non-MLA DFlash attention (head 64) got the target's B12X sparse-MLA fp8_ds_mla KV dtype — the target's MLA layer rewrites the shared cache_config.cache_dtype (mla_attention _canonicalize_sparse_mla_kv_cache_dtype) and the selector has no backend for (head 64, bf16, fp8_ds_mla, non-MLA). Explicit --kv-cache-dtype fp8_e4m3 did NOT help (overridden back to fp8_ds_mla).
  • Boot 3: --kv-cache-dtype-skip-layers sliding_window (the r37 flag for exactly this: skipped layers use auto/bf16 KV; the target's DSA layers are not sliding-window so they keep fp8_ds_mla) → attention init passed, but the draft SlidingWindowSpec inherited a second pollution: the target's cache_config.skip_page_size_padded (shared-page padding) < the draft's bf16 page → assert page_size_padded >= unpadded_page_size_bytes in kv_cache_interface.py.
  • Fix: patches/patch_dspark_draft_page.py (anchor patcher, in-container mod via the shared HF cache sm121-patches/; the draft spec passes page_size_padded=None — its KV group is independent and must use its own page size). Tested throwaway-container first (check→apply→check).
  • Boot 5: KV budget — the 8 GiB pin was 80 MB short of the 8.08 GiB the engine needs for one 131072 request + the drafter's KV group → pin raised to 9 GiB.
  • Boot 6 (13:34): up. KV pool 145,664 tokens @131K (concurrency 1.11x — the drafter's KV group eats ~9% of the pin vs the no-spec 159,232), PIECEWISE+FULL target graphs + separate dspark graph capture (54 s), Application startup complete.
  1. [done] Recipe + spec config (K8, method dspark, drafter on NAS snapshot 198b3ced; the repo is in the shared HF cache, HF_HUB_OFFLINE).
  2. [done] Boot: drafter resolved as Qwen3DSparkModel, graphs captured.
  3. [done] Quality gate: typo probe clean (the 09-14 r37 DS4.1 garbling did NOT appear here), identifiers 6/6. Note: this recipe's glm47 template does not honor chat_template_kwargs thinking:false (the model writes its plan into content; reasoning_tokens stays 0) — cosmetic, bench handles it.
  4. [done] Bench dev67-full-glm53-dspark8 (13:43 → 14:31 UTC, dev65+g6be685f0e, seed 42, hardmode, c1+c2): 153/176 (87/100, ★★★★) — within run variance of the no-spec anchor 158/176 (−5, same as eugr 161 vs 166). 8 fails, all wrong_args/missing_step, no garbling: TC-22/TC-68 invalid JSON, TC-43 empty query, TC-45 tool_choice, TC-38 crowded namespace 2/4, TC-51 (safety, the same send_email-before-confirm quirk eugr 2x had), TC-80 + TC-87 (★★★★★ preconditioned update / cursor integrity). Perf vs anchor (no-spec 158/176, 934 pp / 19.0 tg / 2,124 ms):
    • d0 c1: 892 pp / 33.2 tg / 2,193 ms — decode 1.75×, prefill −4.5% (spec overhead), TTFT +3% (flat).
    • d4096 c1: 847 pp / 28.4 tg / 6,662 ms; d8192 c1: 863 / 27.3 / 10,979 — decode holds 27–28 tg at depth. Verdict: the K8 drafter is a real +75% decode win on the full model at 131K with quality held (card numbers were BF16-target / MRCR; MXFP4 target + 131K context land below them, as expected). Report: recipes/glm53-full-mxfp4-4x-dspark8/reports/2026-09-18_134322_dspark8.*.
  5. Teardown: stop, then the P1/P2 sequence.

P1 — r37 typo verdict (closes the morning's bisect)

  1. Boot D: r37 image, full prod contract, engram off (--engram-config removed) → typo probe. Clean → the lab's engram disk-read path is the cause; garbled → tree numerics; either way finalize BUG-2026-09-14-r37-ds41-text-garbling.md and file it with the lab.
  2. eugr b12x nightly-20260913 A/B: pull (11.2 GB), same probe recipe on 4x — built from the lab's dev/infernal-invocation, one lab-branch newer than our r37. Clean text → the r37/DS4.1-alt line revives on the newer tree.
  3. If both still garble: DS4.1 stays on tonyd2wild prod; R37 = GLM-lane only (where it's proven).

P2 — 8x trials (cables on site; nodes provisioned, cluster configured)

Gate: CLEARED — all 8 nodes up + validated, 8nodes-switch in config, health green. The launcher syncs missing images at serve time.

  1. 8x full GLM-5.3 MXFP4 @512K DCP1: TP8 → 47 GiB weights/node → KV budget ≈ 45+ GiB/rank at GMU 0.85 → a 512K request costs 26.3 GiB/rank (52.7 KB/token/rank × 512K) and fits without DCP — sidesteps the DCP>1 sparse-decode bug entirely. Pin KV ~28–32 GiB (pool ≈ 550–620K tokens/rank); 768K needs ~39.5 GiB (fits at the top of the budget).
    • DSpark K8 drafter (2.3 GiB). This is the full-model long-context lane the 4x cluster cannot deliver.
  2. 8x lab GLM-5.3-NVFP4 + MTP k3 (manifest policy): 433 GB → 54.1 GiB/ node; the lab's own qualified full-model config; A/B vs (1).
  3. 8x DS4.1-Flash — no disk staging (operator decision, code-verified): at TP8 the two 203 GB Engram tables fit in DRAM (1/8 of the table ≈ 51 GiB/rank + weights within 121.6 GiB/node), so the 8x recipe drops DSV41_ENGRAM_DISK/DSV41_ENGRAM_LOCAL/DSV41_ENGRAM_DIR and the /var/tmp/engram-local mount entirely — the baked engram.py patch's disk mode is env-gated and defaults to off (in-memory path). The engine prints the per-rank layout at boot (the recipe header documents it), so nothing needs pre-staging. Aggregate KV ~2× (10.4M → ~20M tokens) — the capacity/1M-context lane for prod traffic. Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is qualified).

P3 — optional, time permitting

  1. Qwen3.8-Flash-Next-NVFP4 quick trial (1–2 nodes): new family, catalog-updated 09-15 (image limit dropped — vision serving); MTP k3, GMU 0.94, gdn_decode b12x, block 16, kv bf16, PLE env vars.
  2. tonyd2wild GLM-Flash v11-dflash2 on 2x: their 262K DFlash2 stack vs our eugr 2x (1M) — only if the 2x lane becomes free (it must NOT be stopped for this).
  3. DS4.1-Flash lab-policy A/B (MTP k2, GMU 0.91, seqs 64): requires downloading deepseek-ai/DeepSeek-V4-Flash 60d8d70">@60d8d70 (~510 GB) — a multi-hour NAS download; start early if the operator wants it.

P4 — bookkeeping

  • File the lab issues (garbling bisect + 3 engine bugs from 09-13).
  • End-state: per operator. Default guardrail unchanged: switch powered down at end unless the operator leaves an 8x lane serving.

Guardrails

  • Nothing is serving right now (the 2x eugr lane was stopped 09-18); all 8 nodes + sw6_mt are ON as prep. First serve of the night is a clean slate.
  • Typo probe gates any DS4.1/r37 promotion (the 09-14 lesson).
  • sm121 serve --keep for first/unknown boots; short-timeout polling.
  • Image distribution is the launcher's job at serve time (--build-node
    • checksum-label sync) — no manual docker save|ssh|docker load.
  • Kernel stays 6.17.0-1031 (7.0.0-1019 breaks NCCL/RoCE — never upgrade).

Open questions for the operator

  1. Which blocks matter tonight: P0 (DSpark K8 on the full model, 4x) is the stated wish; 8x (P2) is fully prepped and ready to go on your word.
  2. After the night: power everything down, or leave an 8x lane serving?