|
|
@@ -0,0 +1,88 @@
|
|
|
+# Night shift 2026-10-22 → 23 — plan
|
|
|
+
|
|
|
+Stand: dgx5–8 (dgx1–4 = production 1017, untouched). Every experiment on the
|
|
|
+free four.
|
|
|
+
|
|
|
+## Where the day left us
|
|
|
+
|
|
|
+- **GLM-5.3 (full) MXFP4×FP8 boots on 4 Sparks** (`glm53-mxfp4-fp8-4x-r39`,
|
|
|
+ jovian r39): 98.86 GiB weights/rank, MTP k=3 → **33–35 t/s c1** (was 19.0
|
|
|
+ without speculation), 47.9 t/s at c2, quality 88/100 (v2.8.0, un-speculated
|
|
|
+ run).
|
|
|
+- **DCP4 is not available** for this checkpoint on any of our images
|
|
|
+ (`DCP not supported` from the sparse-MLA backend), so the window is
|
|
|
+ KV-pin-bound only.
|
|
|
+- **FLARE works** (DeepSeek-V4.1-Flash-1017 TP4, 1M ctx advertised, 28.9M KV
|
|
|
+ tokens, Rust runtime) but is ~2.4× slower on decode than our SGLang lane at
|
|
|
+ defaults; agents were stopped and disabled afterwards.
|
|
|
+- Karmic A/B: native b12x MoE experts buy +1% (19.2 vs 19.0) — decode is
|
|
|
+ bandwidth-bound, not kernel-bound.
|
|
|
+
|
|
|
+## Incident at 22:45 (must not repeat)
|
|
|
+
|
|
|
+A ~150K-token prompt against the 196608-window lane ended in
|
|
|
+`TimeoutError: RPC call to sample_tokens timed out` → EngineDead, and then
|
|
|
+**dgx6 + dgx7 hung** (ping alive, sshd starved) — the 2026-09-08 OOM-thrash
|
|
|
+signature. Recovery: HA plug cycle of both nodes (off 35 s → on), back in ~1 min,
|
|
|
+no data loss (all model data is on the NAS).
|
|
|
+
|
|
|
+Lesson: on GB10 the long-prefill scratch, not the KV pool, is the dangerous
|
|
|
+budget. After 98.86 GiB of weights only ~22 GiB is left, and an 11 GiB KV pin
|
|
|
+leaves too little for a 150K prefill's activations. The lane now runs 131072
|
|
|
+context, 6 GiB KV pin, 2048 batched tokens (smaller chunks = smaller scratch),
|
|
|
+`VLLM_RPC_TIMEOUT=3600`.
|
|
|
+
|
|
|
+## Shifts of work
|
|
|
+
|
|
|
+**P0 — research (no GPU needed, do first).**
|
|
|
+1. Upstream/release sweep: `jovian-judgement` lab locks (new rev past r39?),
|
|
|
+ local-inference-lab GitHub (`blackwell-llm-docker`, flashinfer fork,
|
|
|
+ `llm-inference-bench`, `rtx6kpro`), FujitsuPolycom `sparkring` +
|
|
|
+ `sparkcache` + `net-sircl` + `glm53-flash-tp4-spark`, NVIDIA
|
|
|
+ `flare-local-inference` commits since `3b9e86a`, eugr nightlies,
|
|
|
+ `aidendle94/sparkrun-*`, TensorFold 1.0.12 family (is `MissingEntry` fixed?),
|
|
|
+ `deepseek-ai` / `zai-org` / `RedHatAI` new checkpoints.
|
|
|
+2. Forum sweep (Discourse JSON API, `nvidia-forum-reading` skill): the GLM-5.3,
|
|
|
+ 1017 and "long context on Spark" threads — any new KV/gmu/prefill-scratch
|
|
|
+ numbers.
|
|
|
+3. Record every finding with a link + one-line "what it changes for us".
|
|
|
+
|
|
|
+**P1 — the formulas (the actual goal of the night).**
|
|
|
+Derive, from measurements rather than folklore, the envelopes that decide a
|
|
|
+recipe on this hardware:
|
|
|
+
|
|
|
+- **KV-per-token formula.** For the full GLM-5.3 checkpoint we have
|
|
|
+ 52.7 KB/token/rank (79,616 tokens per 4 GiB). Confirm it is linear and
|
|
|
+ independent of the window, then publish `window_max = KV_pin / 52.7 KB`.
|
|
|
+- **Prefill-scratch formula.** Measure the host memory a prefill of N tokens
|
|
|
+ needs (N = 8K, 32K, 64K, 128K) at `--max-num-batched-tokens 2048` and 4096,
|
|
|
+ on the full model and on the Flash sibling. Goal: a rule like
|
|
|
+ `scratch ≈ a + b·N`, and a hard ceiling — "do not send N > X tokens on a
|
|
|
+ 99 GiB-weights lane, it hangs the host".
|
|
|
+- **Speculation formula.** decode_tps(k) for k = 1, 3, 5 with acceptance
|
|
|
+ measured — we already know k=3 ≈ k=5; extend to the Flash lane and check
|
|
|
+ whether the acceptance-vs-depth curve from vLLM's metrics predicts the t/s
|
|
|
+ plateau.
|
|
|
+- **Weights-per-rank → decode** curve across our lanes (Flash 47 GiB → 52 t/s,
|
|
|
+ full 99 GiB → 19 t/s un-speculated, 35 t/s with MTP) to get the decode
|
|
|
+ bandwidth coefficient for GB10.
|
|
|
+
|
|
|
+**P2 — verify the safe 128K lane.** Needle at 128K (deep depth), decode at
|
|
|
+depth, then tool-eval-bench v2.8.0 (now that context is real) for the quality
|
|
|
+number at the final configuration; land it in the recipe NOTE + report.
|
|
|
+
|
|
|
+**P3 — optional, nodes free.** FLARE tuning pass (`--scheduler-policy
|
|
|
+throughput`, `--max-concurrency`, adaptive verification) to see whether the
|
|
|
+Rust runtime can close the 2.4× gap; or the sparkring `glm53-flash-tp4` build
|
|
|
+if a new digest appeared.
|
|
|
+
|
|
|
+## Guardrails for the night
|
|
|
+
|
|
|
+- No `stop`/`power` against dgx1–4. No shared-switch changes (`sw6_mt` is used
|
|
|
+ by both test and prod lanes).
|
|
|
+- Any node that stops answering SSH: 3 short retries (30 s), then HA plug cycle
|
|
|
+ of that node only, then re-check before continuing.
|
|
|
+- Long prompts are tested in increasing order (8K → 32K → 64K → 128K) and never
|
|
|
+ against a lane whose scratch envelope is unknown.
|
|
|
+- Polling: short cycles with an early exit on the first error line; no
|
|
|
+ multi-minute sleeps.
|
