launcher/history/night-shifts/2026-10-22-night-plan.md

Night shift 2026-10-22 → 23 — plan

Stand: dgx5–8 (dgx1–4 = production 1017, untouched). Every experiment on the free four.

Where the day left us

• GLM-5.3 (full) MXFP4×FP8 boots on 4 Sparks (glm53-mxfp4-fp8-4x-r39, jovian r39): 98.86 GiB weights/rank, MTP k=3 → 33–35 t/s c1 (was 19.0 without speculation), 47.9 t/s at c2, quality 88/100 (v2.8.0, un-speculated run).
• DCP4 is not available for this checkpoint on any of our images (DCP not supported from the sparse-MLA backend), so the window is KV-pin-bound only.
• FLARE works (DeepSeek-V4.1-Flash-1017 TP4, 1M ctx advertised, 28.9M KV tokens, Rust runtime) but is ~2.4× slower on decode than our SGLang lane at defaults; agents were stopped and disabled afterwards.
• Karmic A/B: native b12x MoE experts buy +1% (19.2 vs 19.0) — decode is bandwidth-bound, not kernel-bound.

Incident at 22:45 (must not repeat)

A ~150K-token prompt against the 196608-window lane ended in TimeoutError: RPC call to sample_tokens timed out → EngineDead, and then dgx6 + dgx7 hung (ping alive, sshd starved) — the 2026-09-08 OOM-thrash signature. Recovery: HA plug cycle of both nodes (off 35 s → on), back in ~1 min, no data loss (all model data is on the NAS).

Lesson: on GB10 the long-prefill scratch, not the KV pool, is the dangerous budget. After 98.86 GiB of weights only ~22 GiB is left, and an 11 GiB KV pin leaves too little for a 150K prefill's activations. The lane now runs 131072 context, 6 GiB KV pin, 2048 batched tokens (smaller chunks = smaller scratch), VLLM_RPC_TIMEOUT=3600.

Shifts of work

P0 — research (no GPU needed, do first).

1. Upstream/release sweep: jovian-judgement lab locks (new rev past r39?), local-inference-lab GitHub (blackwell-llm-docker, flashinfer fork, llm-inference-bench, rtx6kpro), FujitsuPolycom sparkring + sparkcache + net-sircl + glm53-flash-tp4-spark, NVIDIA flare-local-inference commits since 3b9e86a, eugr nightlies, aidendle94/sparkrun-*, TensorFold 1.0.12 family (is MissingEntry fixed?), deepseek-ai / zai-org / RedHatAI new checkpoints.
2. Forum sweep (Discourse JSON API, nvidia-forum-reading skill): the GLM-5.3, 1017 and "long context on Spark" threads — any new KV/gmu/prefill-scratch numbers.
3. Record every finding with a link + one-line "what it changes for us".

P1 — the formulas (the actual goal of the night). Derive, from measurements rather than folklore, the envelopes that decide a recipe on this hardware:

• KV-per-token formula. For the full GLM-5.3 checkpoint we have 52.7 KB/token/rank (79,616 tokens per 4 GiB). Confirm it is linear and independent of the window, then publish window_max = KV_pin / 52.7 KB.
• Prefill-scratch formula. Measure the host memory a prefill of N tokens needs (N = 8K, 32K, 64K, 128K) at --max-num-batched-tokens 2048 and 4096, on the full model and on the Flash sibling. Goal: a rule like scratch ≈ a + b·N, and a hard ceiling — "do not send N > X tokens on a 99 GiB-weights lane, it hangs the host".
• Speculation formula. decode_tps(k) for k = 1, 3, 5 with acceptance measured — we already know k=3 ≈ k=5; extend to the Flash lane and check whether the acceptance-vs-depth curve from vLLM's metrics predicts the t/s plateau.
• Weights-per-rank → decode curve across our lanes (Flash 47 GiB → 52 t/s, full 99 GiB → 19 t/s un-speculated, 35 t/s with MTP) to get the decode bandwidth coefficient for GB10.

P2 — verify the safe 128K lane. Needle at 128K (deep depth), decode at depth, then tool-eval-bench v2.8.0 (now that context is real) for the quality number at the final configuration; land it in the recipe NOTE + report.

P3 — optional, nodes free. FLARE tuning pass (--scheduler-policy throughput, --max-concurrency, adaptive verification) to see whether the Rust runtime can close the 2.4× gap; or the sparkring glm53-flash-tp4 build if a new digest appeared.

Guardrails for the night

• No stop/power against dgx1–4. No shared-switch changes (sw6_mt is used by both test and prod lanes).
• Any node that stops answering SSH: 3 short retries (30 s), then HA plug cycle of that node only, then re-check before continuing.
• Long prompts are tested in increasing order (8K → 32K → 64K → 128K) and never against a lane whose scratch envelope is unknown.
• Polling: short cycles with an early exit on the first error line; no multi-minute sleeps.