night plan 2026-10-22/23: reproduction of the long-prefill host hang, the three envelopes to measure (KV-per-token, prefill scratch, spec depth), and the guardrails

Also records the FLARE/karmic/GLM state the shift starts from.

history/night-shifts/2026-10-22-night-plan.md

@@ -0,0 +1,88 @@
+# Night shift 2026-10-22 → 23 — plan
+
+Stand: dgx5–8 (dgx1–4 = production 1017, untouched). Every experiment on the
+free four.
+
+## Where the day left us
+
+- **GLM-5.3 (full) MXFP4×FP8 boots on 4 Sparks** (`glm53-mxfp4-fp8-4x-r39`,
+  jovian r39): 98.86 GiB weights/rank, MTP k=3 → **33–35 t/s c1** (was 19.0
+  without speculation), 47.9 t/s at c2, quality 88/100 (v2.8.0, un-speculated
+  run).
+- **DCP4 is not available** for this checkpoint on any of our images
+  (`DCP not supported` from the sparse-MLA backend), so the window is
+  KV-pin-bound only.
+- **FLARE works** (DeepSeek-V4.1-Flash-1017 TP4, 1M ctx advertised, 28.9M KV
+  tokens, Rust runtime) but is ~2.4× slower on decode than our SGLang lane at
+  defaults; agents were stopped and disabled afterwards.
+- Karmic A/B: native b12x MoE experts buy +1% (19.2 vs 19.0) — decode is
+  bandwidth-bound, not kernel-bound.
+
+## Incident at 22:45 (must not repeat)
+
+A ~150K-token prompt against the 196608-window lane ended in
+`TimeoutError: RPC call to sample_tokens timed out` → EngineDead, and then
+**dgx6 + dgx7 hung** (ping alive, sshd starved) — the 2026-09-08 OOM-thrash
+signature. Recovery: HA plug cycle of both nodes (off 35 s → on), back in ~1 min,
+no data loss (all model data is on the NAS).
+
+Lesson: on GB10 the long-prefill scratch, not the KV pool, is the dangerous
+budget. After 98.86 GiB of weights only ~22 GiB is left, and an 11 GiB KV pin
+leaves too little for a 150K prefill's activations. The lane now runs 131072
+context, 6 GiB KV pin, 2048 batched tokens (smaller chunks = smaller scratch),
+`VLLM_RPC_TIMEOUT=3600`.
+
+## Shifts of work
+
+**P0 — research (no GPU needed, do first).**
+1. Upstream/release sweep: `jovian-judgement` lab locks (new rev past r39?),
+   local-inference-lab GitHub (`blackwell-llm-docker`, flashinfer fork,
+   `llm-inference-bench`, `rtx6kpro`), FujitsuPolycom `sparkring` +
+   `sparkcache` + `net-sircl` + `glm53-flash-tp4-spark`, NVIDIA
+   `flare-local-inference` commits since `3b9e86a`, eugr nightlies,
+   `aidendle94/sparkrun-*`, TensorFold 1.0.12 family (is `MissingEntry` fixed?),
+   `deepseek-ai` / `zai-org` / `RedHatAI` new checkpoints.
+2. Forum sweep (Discourse JSON API, `nvidia-forum-reading` skill): the GLM-5.3,
+   1017 and "long context on Spark" threads — any new KV/gmu/prefill-scratch
+   numbers.
+3. Record every finding with a link + one-line "what it changes for us".
+
+**P1 — the formulas (the actual goal of the night).**
+Derive, from measurements rather than folklore, the envelopes that decide a
+recipe on this hardware:
+
+- **KV-per-token formula.** For the full GLM-5.3 checkpoint we have
+  52.7 KB/token/rank (79,616 tokens per 4 GiB). Confirm it is linear and
+  independent of the window, then publish `window_max = KV_pin / 52.7 KB`.
+- **Prefill-scratch formula.** Measure the host memory a prefill of N tokens
+  needs (N = 8K, 32K, 64K, 128K) at `--max-num-batched-tokens 2048` and 4096,
+  on the full model and on the Flash sibling. Goal: a rule like
+  `scratch ≈ a + b·N`, and a hard ceiling — "do not send N > X tokens on a
+  99 GiB-weights lane, it hangs the host".
+- **Speculation formula.** decode_tps(k) for k = 1, 3, 5 with acceptance
+  measured — we already know k=3 ≈ k=5; extend to the Flash lane and check
+  whether the acceptance-vs-depth curve from vLLM's metrics predicts the t/s
+  plateau.
+- **Weights-per-rank → decode** curve across our lanes (Flash 47 GiB → 52 t/s,
+  full 99 GiB → 19 t/s un-speculated, 35 t/s with MTP) to get the decode
+  bandwidth coefficient for GB10.
+
+**P2 — verify the safe 128K lane.** Needle at 128K (deep depth), decode at
+depth, then tool-eval-bench v2.8.0 (now that context is real) for the quality
+number at the final configuration; land it in the recipe NOTE + report.
+
+**P3 — optional, nodes free.** FLARE tuning pass (`--scheduler-policy
+throughput`, `--max-concurrency`, adaptive verification) to see whether the
+Rust runtime can close the 2.4× gap; or the sparkring `glm53-flash-tp4` build
+if a new digest appeared.
+
+## Guardrails for the night
+
+- No `stop`/`power` against dgx1–4. No shared-switch changes (`sw6_mt` is used
+  by both test and prod lanes).
+- Any node that stops answering SSH: 3 short retries (30 s), then HA plug cycle
+  of that node only, then re-check before continuing.
+- Long prompts are tested in increasing order (8K → 32K → 64K → 128K) and never
+  against a lane whose scratch envelope is unknown.
+- Polling: short cycles with an early exit on the first error line; no
+  multi-minute sleeps.