night-shift 2026-09-13 plan + R37 (jovian) build profile

- profiles/jovian-r37-ds41.env: vLLM c687594b8a8 (tree 3fb88fdf) + b12x
  f1c4e9dd5b1 (tree 480ea232), LMCache/FlashKDA/DeepGEMM unchanged, torch
  2.13.0 / flashinfer 0.6.17 (dep gates re-checked against the r37 tree, no
  resync); r37 = DS4.1 lock with -43% kernel-dispatch overhead (+43.65%
  uncached 32K prefill in their qualification); GLM stays r35
- history/night-shifts/2026-09-13.md: the plan (P0 full GLM-5.3 MXFP4 boot
  ladder + bench, P1 r37 build on dgx5, P2 r37 A/B, P3 eugr nightly-20260913),
  state at handoff, and the updates found: kernel regression 7.0.0-1019
  breaking NCCL/RoCE (KHO CMA scratch, NV-Kernels#590) - do not upgrade;
  #382897 77 tok/s is warmed counting-peak (prose 23/reasoning 39 verified
  consistent with our numbers) + the SM-clock latch check

history/night-shifts/2026-09-13.md

@@ -0,0 +1,204 @@
+# Night shift 2026-09-13 → 2026-09-14
+
+Prepared 2026-09-13 ~21:30 UTC by the assistant. Operator brief: "посмотри
+обновления, поищи по форумам, готовь план для ночной смены".
+
+## State at handoff
+
+| | |
+|---|---|
+| `4nodes-switch` | ON, **serving the eugr DS4.1 prod** (`dsv41-flash-4x-b12x`, started 13:31 UTC, 512K, DSpark k=5, Engram-on-disk + per-rank local sparse). This is the agent's own brain — **confirm before stopping**. |
+| `1node-5` (dgx5) | ON, idle — the build node (no RoCE / 10 GbE only, so it cannot join the 4x fabric: 5-node lanes are out). |
+| Checkpoint | `RedHatAI/GLM-5.3-MXFP4` downloading to the NAS: ~110 GB / 403.4 GB at **114 MB/s → ETA ~40 min**. Verify before P0. |
+| Kernel | all nodes on **6.17.0-1031-nvidia** — do NOT upgrade (see Updates #3). |
+| git | `git.kilork.org` still unreachable → commit locally; ~10 unpushed commits. |
+
+## Updates found tonight (external)
+
+1. **R37 lock published 2026-09-13** (lab, DS4.1): image
+   `localinferencelab/vllm:jovian-judgement-community-20260913-r37`. The
+   headline change is a **kernel-dispatch overhead removal** (repeated CPU
+   arg-list expansion, mutable-schema inspection, scalar dtype allocations,
+   tensor-equality calls) → **uncached 32K prefill +43.65%** (14,896 → 21,399
+   tok/s), C1 output +17.7%, verifier +11.9%, Sieve +8.2% (their RTX quartet;
+   integrated, not per-PR). Also corrects the named `high` reasoning budget
+   50 → 75. Pins: vLLM `c687594b8a8` (tree `3fb88fdf…`), b12x `f1c4e9dd5b1`
+   (tree `480ea232…`), LMCache `29bc5a2e` unchanged, DeepGEMM `8b1392b9`,
+   torch 2.13.0 / flashinfer 0.6.17 — **no dependency resync needed**.
+   → *GLM* stays at **R35** (still the latest qualified GLM lock); R37 is the
+   DS4.1 lock. **Action: build profile written** →
+   `docker-images/glm53-jovian-r27/profiles/jovian-r37-ds41.env` (P1).
+2. **eugr `nightly-20260913`** image built (13:09 UTC, supersedes
+   `nightly-20260910` = our DS4.1 prod image). Changelog not at the usual raw
+   path (404) — read the repo/thread during the shift. Cheap A/B (P3).
+3. **⚠ Kernel regression (forum #383023):** `7.0.0-1019-nvidia` breaks
+   NCCL/RoCE — `ibv_reg_mr_iova2 failed with error Cannot allocate memory`
+   during TP communication. Reproducible kernel-only A/B; root cause traced
+   (2026-09-13) to **KHO's default scratch reservation marking ~9.3 GiB as
+   unaccounted `MIGRATE_CMA`**; fix PR NVIDIA/NV-Kernels#590. Working:
+   `6.17.0-1032` (and our `6.17.0-1031`). **Action: do not upgrade the
+   kernel on any node; if a node ever shows RDMA ENOMEM, check `uname -r`
+   first.**
+4. **Forum #382897** ("DS4.1-Flash 552B on 4× Spark, 77.2 tok/s c1"): our
+   upstream author (tonyd2wild — the repo our `dsv41-flash` image is built
+   from). The 77 tok/s is a **warmed, streaming, counting-prompt peak** with
+   up-to-5.95/6 draft acceptance; the same post lists **prose 23, reasoning
+   39, math 47, code 52** tok/s — i.e. our measured 47–49 tg on agentic
+   traffic is consistent; peak-on-repetition ≠ agent workload (clickbait, as
+   the operator suspected). Two useful nuggets: 203 GB of Engram stays on
+   disk so 552B fits 4x, and **"2 GPUs were latched below 1 GHz — unplug the
+   Spark to clear it, a reboot won't"** → add a clock check before benches.
+5. New lab GLM docs: `models/glm-5.3-flash-spark-tp2.md` (09-12, Spark **TP2
+   + vision** serving & measured capacity — GLM-5.3-Flash is natively
+   multimodal and all our GLM lanes are text-only `--language-model-only`; a
+   possible capability lane later). Also `models/glm-5.1-mxfp4.md` and the
+   `kld/glm-5.3-flash-*` quant-fidelity reports.
+6. Other recent threads worth a skim only: #382939 (optimizing NVIDIA's
+   official `nvidia/GLM-5.3-Flash-NVFP4` on 2x), #382697 / #382927 (2x vs 4x
+   comparisons), #382584 / #381911 (DS4-Flash-Vision-Exp on 1x).
+
+## P0 — GLM-5.3 (full) MXFP4 on 4× — the headline
+
+Recipe `recipes/glm53-full-mxfp4-4x` (r35 stack, DCP4, 512K, KV 8 GiB pin,
+GMU 0.85, no spec on the first boot). Read its header for the full memory
+plan; the arithmetic in one line: **93.9 GiB weights/node + ~12 GiB engine +
+8 GiB KV + ~4 GiB activations ≈ 118 of 121.63 GiB**.
+
+### P0.0 prerequisites (~10 min)
+1. **Free the cluster** — only after confirming with the operator (it may be
+   serving the agent):
+   `sm121 stop --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch`, wait
+   ~12 s before any serve.
+2. **Download complete**: `du -sh $NAS/models--RedHatAI--GLM-5.3-MXFP4` ≈
+   403 GB and **141 shards** present (an `.incomplete` file = resume with
+   `huggingface-cli download RedHatAI/GLM-5.3-MXFP4`). HF_HUB_OFFLINE=1 makes
+   a missing shard a hard load failure.
+3. **Clock check** (nugget #4): per node
+   `nvidia-smi --query-gpu=clocks.sm,clocks_event_reasons.active --format=csv,noheader`
+   — a SM clock latched below ~1 GHz invalidates every bench number.
+4. `sm121 health --cluster 4nodes-switch`.
+
+### P0.1 boot probe — conservative (~60–90 min)
+Edit the recipe args: **remove** `--decode-context-parallel-size 4` and
+`--cp-kv-cache-interleave-size 4`; set `--kv-cache-memory-bytes 4294967296`
+(4 GiB) and `--max-model-len 131072`. Then:
+
+```
+sm121 serve --keep --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch
+```
+
+Poll with `docker logs --tail 2 sm121_glm53-full-mxfp4-4x_0` every 240–280 s.
+
+- Expect ~94 GiB/rank of weights (~10–20 min via instanttensor).
+- **ABORT SIGNAL — a second "Loading safetensors" pass**: that is the
+  2026-09-08 double-load pattern that OOM-thrashed all four nodes (sshd down,
+  HA power-cycle). `sm121 stop` immediately; do not wait to see if it settles.
+- Success markers to record verbatim: `Model loading took X GiB`, the
+  `GPU KV cache size: …` line, `Application startup complete`.
+- **The KV line is the most valuable number of the night** — it gives the
+  real bytes/token for the full model (the parked estimate was ~54 KB/token;
+  everything about 512K depends on it).
+- If the boot dies on memory: record the numbers and go to P0.4.
+
+### P0.2 smoke + capability (~20 min)
+- Chat: `chat_template_kwargs: {"reasoning_effort":"low"}` (glm45 parser,
+  glm47 tool parser). Check coherent content + a tool-call round-trip.
+- **Token-loop check** (the W4A4-on-new-arch risk): generate ~300 tokens and
+  confirm no repeated-token degeneration.
+- Long-context sanity if the pool allows: filler + needle at the largest
+  context that fits, temperature 0 (the operator's floor is 512K — this is
+  the capability claim).
+
+### P0.3 the 512K attempt (DCP4) — (~90 min + bench)
+Restore the DCP4 lines, `--max-model-len 524288`, KV pin 8 GiB. Boot again.
+- If it rejects DCP (the `GLM C4 DCP requires cp_kv_cache_interleave_size to
+  be divisible by 4` / page-stride class that blocks DFlash2 on 2x) → the
+  full model is DCP-incompatible; decision point (P0.4).
+- Verify the KV line reports concurrency at 524288.
+- Bench: `sm121 bench --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch
+  --label dev65-full-mxfp4-dcp4` (~2.5 h, under pty automatically).
+- Compare against the **GLM-5.3-Flash r35 prod anchor**: 163/176 (93),
+  1,554 pp / 49.6 tg / 1,544 ms TTFT @d0 c1, KV 8.06M @512K. A ~740B model
+  beating that on quality would be the story of the week; matching prefill
+  while holding 512K would already make it a capability lane.
+
+### P0.4 fallbacks if it does not fit / degrades
+- Bench the P0.1 shape (no DCP, 131072, 4 GiB KV) as the honest capability
+  datapoint.
+- 5–6 nodes are **not** available (dgx5 is 10 GbE-only, off the RoCE fabric),
+  so 4x is the ceiling for this model tonight.
+- If it thrashes: `sm121 stop`; if the nodes do not recover, HA power-cycle
+  (`sm121 power off/on --cluster 4nodes-switch`, ~3–5 min boot).
+- Record "full GLM-5.3 on 4x = does not fit / fits with N ctx" — a negative
+  is a valuable result here (it retires the question).
+
+### P0.5 teardown (10 min)
+`sm121 stop`; restore prod if wanted
+(`sm121 serve --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch`);
+write the report + history entry + commit locally.
+
+## P1 — build the R37 image (background, dgx5, no GPU contention)
+
+The profile is written and dep-checked: `profiles/jovian-r37-ds41.env`
+(vLLM `c687594b8a8`, b12x `f1c4e9dd5b1`, LMCache/FlashKDA/DeepGEMM unchanged,
+torch 2.13.0 / FI 0.6.17 same as our GB10 pins).
+
+1. Create the recipe pair (copy, then swap 3 lines each):
+   - `recipes/dsv41-flash-jovian-r36-4x` → `recipes/dsv41-flash-jovian-r37-4x`
+     (`docker_image = "jovian-r37-ds41:local"`,
+     `build_profile = "jovian-r37-ds41"`; keeps the composed_model
+     local-Engram hardlinks + 512K + DSpark K7 args).
+   - optionally `recipes/glm53-flash-r35-4x-dcp4` →
+     `recipes/glm53-flash-r37-4x-dcp4` (same image, GLM args) — the image is
+     two-layer, so one build serves both lanes.
+2. `sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan`
+   (~14 min warm buildx cache) → transfer to dgx1–4 (~4 min, 16 GB, parallel
+   `docker save | ssh | docker load`).
+3. Sanity checks in the image (read-only, cheap): `GlmMoeDsaForCausalLM` in
+   `registry.py`, `mxfp4-pack-quantized` scheme, `liburing.so.2`, the b12x
+   native loader present.
+4. Note: the r35/r36 images now read as STALE against the build dir (it
+   gained profiles) — warn-only, but a rebuild makes labels match.
+
+## P2 — R37 A/B (needs the 4x cluster; only after P0)
+
+- `dsv41-flash-jovian-r37-4x`: boot 512K + DSpark K7 + local Engram, bench
+  dev65 → compare with the **r36 datapoint** (154/176, 3,774 pp / 822 ms
+  TTFT) and the **eugr prod** (167/176, 1,642 pp / 1,738 ms).
+  Purpose: does R37's +43.65% prefill claim hold on GB10, and does it close
+  the quality gap that kept the lab stack out of prod? This is the only
+  route we have to a *faster* DS4.1 prod.
+- Optional GLM arm: same image with the r35 GLM args (MTP3/DCP4/512K) →
+  does the dispatch fix lift GLM prefill too? Baseline 163/176, 1,554 pp.
+- Delete the r36 recipes once R37 is measured (keep both recipes until then).
+
+## P3 — eugr nightly-20260913 (cheap, only if ≥3 h remain)
+
+Swap `docker_image` in `dsv41-flash-4x-b12x` (or a copy) to
+`eugr/spark-vllm-b12x:nightly-20260913`, boot, bench dev65 → A/B against the
+nightly-20260910 prod numbers (167/176, 1,642 pp). ~2.5–3 h total.
+
+## Guardrails
+
+- **Never** `stop` / `power off` a serving cluster without the operator's OK.
+- `serve --keep` for every first/unknown boot (default `--rm` wipes crash
+  logs); the 09-08/09-13 posts-mortems depended on it.
+- Poll in short loops (240–280 s); macOS has no `timeout` binary.
+- Bench convention: dev65 = 88 scenarios, seed 42, `--hardmode --perf
+  --concurrency 1,2`; reports auto-land in `recipes/<lane>/reports/`
+  (manifest + recipe copy included).
+- OOM-thrash drill: container stays "Up", workers respawn forever, sshd can
+  die on 2–3 nodes → `sm121 stop`, then HA power-cycle if needed.
+- One serve at a time on the 4x cluster (TP4 uses all four nodes; a stray
+  second serve is an instant OOM).
+- Commit locally after every milestone (git server down).
+
+## Open questions for the operator
+
+1. OK to take the 4× cluster off prod tonight (agent downtime), and for how
+   long? P0 needs the 4 nodes exclusively.
+2. Priority if time is short: (a) the full-model capability datapoint,
+   (b) R37 (build + A/B), (c) both?
+3. If the full model cannot hold 512K on 4×, is a smaller context
+   (131K–262K) acceptable for this lane, or is 512K a hard requirement
+   (= the lane is dead on this hardware)?