# Night shift 2026-09-13 → 2026-09-14

Prepared 2026-09-13 ~21:30 UTC by the assistant. Operator brief: "посмотри
обновления, поищи по форумам, готовь план для ночной смены".

## State at handoff

| | |
|---|---|
| `4nodes-switch` | ON, **serving the eugr DS4.1 prod** (`dsv41-flash-4x-b12x`, started 13:31 UTC, 512K, DSpark k=5, Engram-on-disk + per-rank local sparse). This is the agent's own brain — **confirm before stopping**. |
| `1node-5` (dgx5) | ON, idle — the build node (no RoCE / 10 GbE only, so it cannot join the 4x fabric: 5-node lanes are out). |
| Checkpoint | `RedHatAI/GLM-5.3-MXFP4` downloading to the NAS: ~110 GB / 403.4 GB at **114 MB/s → ETA ~40 min**. Verify before P0. |
| Kernel | all nodes on **6.17.0-1031-nvidia** — do NOT upgrade (see Updates #3). |
| git | `git.kilork.org` still unreachable → commit locally; ~10 unpushed commits. |

## Updates found tonight (external)

1. **R37 lock published 2026-09-13** (lab, DS4.1): image
   `localinferencelab/vllm:jovian-judgement-community-20260913-r37`. The
   headline change is a **kernel-dispatch overhead removal** (repeated CPU
   arg-list expansion, mutable-schema inspection, scalar dtype allocations,
   tensor-equality calls) → **uncached 32K prefill +43.65%** (14,896 → 21,399
   tok/s), C1 output +17.7%, verifier +11.9%, Sieve +8.2% (their RTX quartet;
   integrated, not per-PR). Also corrects the named `high` reasoning budget
   50 → 75. Pins: vLLM `c687594b8a8` (tree `3fb88fdf…`), b12x `f1c4e9dd5b1`
   (tree `480ea232…`), LMCache `29bc5a2e` unchanged, DeepGEMM `8b1392b9`,
   torch 2.13.0 / flashinfer 0.6.17 — **no dependency resync needed**.
   → *GLM* stays at **R35** (still the latest qualified GLM lock); R37 is the
   DS4.1 lock. **Action: build profile written** →
   `docker-images/glm53-jovian-r27/profiles/jovian-r37-ds41.env` (P1).
2. **eugr `nightly-20260913`** image built (13:09 UTC, supersedes
   `nightly-20260910` = our DS4.1 prod image). Changelog not at the usual raw
   path (404) — read the repo/thread during the shift. Cheap A/B (P3).
3. **⚠ Kernel regression (forum #383023):** `7.0.0-1019-nvidia` breaks
   NCCL/RoCE — `ibv_reg_mr_iova2 failed with error Cannot allocate memory`
   during TP communication. Reproducible kernel-only A/B; root cause traced
   (2026-09-13) to **KHO's default scratch reservation marking ~9.3 GiB as
   unaccounted `MIGRATE_CMA`**; fix PR NVIDIA/NV-Kernels#590. Working:
   `6.17.0-1032` (and our `6.17.0-1031`). **Action: do not upgrade the
   kernel on any node; if a node ever shows RDMA ENOMEM, check `uname -r`
   first.**
4. **Forum #382897** ("DS4.1-Flash 552B on 4× Spark, 77.2 tok/s c1"): our
   upstream author (tonyd2wild — the repo our `dsv41-flash` image is built
   from). The 77 tok/s is a **warmed, streaming, counting-prompt peak** with
   up-to-5.95/6 draft acceptance; the same post lists **prose 23, reasoning
   39, math 47, code 52** tok/s — i.e. our measured 47–49 tg on agentic
   traffic is consistent; peak-on-repetition ≠ agent workload (clickbait, as
   the operator suspected). Two useful nuggets: 203 GB of Engram stays on
   disk so 552B fits 4x, and **"2 GPUs were latched below 1 GHz — unplug the
   Spark to clear it, a reboot won't"** → add a clock check before benches.
5. New lab GLM docs: `models/glm-5.3-flash-spark-tp2.md` (09-12, Spark **TP2
   + vision** serving & measured capacity — GLM-5.3-Flash is natively
   multimodal and all our GLM lanes are text-only `--language-model-only`; a
   possible capability lane later). Also `models/glm-5.1-mxfp4.md` and the
   `kld/glm-5.3-flash-*` quant-fidelity reports.
6. Other recent threads worth a skim only: #382939 (optimizing NVIDIA's
   official `nvidia/GLM-5.3-Flash-NVFP4` on 2x), #382697 / #382927 (2x vs 4x
   comparisons), #382584 / #381911 (DS4-Flash-Vision-Exp on 1x).

## P0 — GLM-5.3 (full) MXFP4 on 4× — the headline

Recipe `recipes/glm53-full-mxfp4-4x` (r35 stack, DCP4, 512K, KV 8 GiB pin,
GMU 0.85, no spec on the first boot). Read its header for the full memory
plan; the arithmetic in one line: **93.9 GiB weights/node + ~12 GiB engine +
8 GiB KV + ~4 GiB activations ≈ 118 of 121.63 GiB**.

### P0.0 prerequisites (~10 min)
1. **Free the cluster** — only after confirming with the operator (it may be
   serving the agent):
   `sm121 stop --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch`, wait
   ~12 s before any serve.
2. **Download complete**: `du -sh $NAS/models--RedHatAI--GLM-5.3-MXFP4` ≈
   403 GB and **141 shards** present (an `.incomplete` file = resume with
   `huggingface-cli download RedHatAI/GLM-5.3-MXFP4`). HF_HUB_OFFLINE=1 makes
   a missing shard a hard load failure.
3. **Clock check** (nugget #4): per node
   `nvidia-smi --query-gpu=clocks.sm,clocks_event_reasons.active --format=csv,noheader`
   — a SM clock latched below ~1 GHz invalidates every bench number.
4. `sm121 health --cluster 4nodes-switch`.

### P0.1 boot probe — conservative (~60–90 min)
Edit the recipe args: **remove** `--decode-context-parallel-size 4` and
`--cp-kv-cache-interleave-size 4`; set `--kv-cache-memory-bytes 4294967296`
(4 GiB) and `--max-model-len 131072`. Then:

```
sm121 serve --keep --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch
```

Poll with `docker logs --tail 2 sm121_glm53-full-mxfp4-4x_0` every 240–280 s.

- Expect ~94 GiB/rank of weights (~10–20 min via instanttensor).
- **ABORT SIGNAL — a second "Loading safetensors" pass**: that is the
  2026-09-08 double-load pattern that OOM-thrashed all four nodes (sshd down,
  HA power-cycle). `sm121 stop` immediately; do not wait to see if it settles.
- Success markers to record verbatim: `Model loading took X GiB`, the
  `GPU KV cache size: …` line, `Application startup complete`.
- **The KV line is the most valuable number of the night** — it gives the
  real bytes/token for the full model (the parked estimate was ~54 KB/token;
  everything about 512K depends on it).
- If the boot dies on memory: record the numbers and go to P0.4.

### P0.2 smoke + capability (~20 min)
- Chat: `chat_template_kwargs: {"reasoning_effort":"low"}` (glm45 parser,
  glm47 tool parser). Check coherent content + a tool-call round-trip.
- **Token-loop check** (the W4A4-on-new-arch risk): generate ~300 tokens and
  confirm no repeated-token degeneration.
- Long-context sanity if the pool allows: filler + needle at the largest
  context that fits, temperature 0 (the operator's floor is 512K — this is
  the capability claim).

### P0.3 the 512K attempt (DCP4) — (~90 min + bench)
Restore the DCP4 lines, `--max-model-len 524288`, KV pin 8 GiB. Boot again.
- If it rejects DCP (the `GLM C4 DCP requires cp_kv_cache_interleave_size to
  be divisible by 4` / page-stride class that blocks DFlash2 on 2x) → the
  full model is DCP-incompatible; decision point (P0.4).
- Verify the KV line reports concurrency at 524288.
- Bench: `sm121 bench --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch
  --label dev65-full-mxfp4-dcp4` (~2.5 h, under pty automatically).
- Compare against the **GLM-5.3-Flash r35 prod anchor**: 163/176 (93),
  1,554 pp / 49.6 tg / 1,544 ms TTFT @d0 c1, KV 8.06M @512K. A ~740B model
  beating that on quality would be the story of the week; matching prefill
  while holding 512K would already make it a capability lane.

### P0.4 fallbacks if it does not fit / degrades
- Bench the P0.1 shape (no DCP, 131072, 4 GiB KV) as the honest capability
  datapoint.
- 5–6 nodes are **not** available (dgx5 is 10 GbE-only, off the RoCE fabric),
  so 4x is the ceiling for this model tonight.
- If it thrashes: `sm121 stop`; if the nodes do not recover, HA power-cycle
  (`sm121 power off/on --cluster 4nodes-switch`, ~3–5 min boot).
- Record "full GLM-5.3 on 4x = does not fit / fits with N ctx" — a negative
  is a valuable result here (it retires the question).

### P0.5 teardown (10 min)
`sm121 stop`; restore prod if wanted
(`sm121 serve --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch`);
write the report + history entry + commit locally.

## P1 — build the R37 image (background, dgx5, no GPU contention)

The profile is written and dep-checked: `profiles/jovian-r37-ds41.env`
(vLLM `c687594b8a8`, b12x `f1c4e9dd5b1`, LMCache/FlashKDA/DeepGEMM unchanged,
torch 2.13.0 / FI 0.6.17 same as our GB10 pins).

1. Create the recipe pair (copy, then swap 3 lines each):
   - `recipes/dsv41-flash-jovian-r36-4x` → `recipes/dsv41-flash-jovian-r37-4x`
     (`docker_image = "jovian-r37-ds41:local"`,
     `build_profile = "jovian-r37-ds41"`; keeps the composed_model
     local-Engram hardlinks + 512K + DSpark K7 args).
   - optionally `recipes/glm53-flash-r35-4x-dcp4` →
     `recipes/glm53-flash-r37-4x-dcp4` (same image, GLM args) — the image is
     two-layer, so one build serves both lanes.
2. `sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan`
   (~14 min warm buildx cache) → transfer to dgx1–4 (~4 min, 16 GB, parallel
   `docker save | ssh | docker load`).
3. Sanity checks in the image (read-only, cheap): `GlmMoeDsaForCausalLM` in
   `registry.py`, `mxfp4-pack-quantized` scheme, `liburing.so.2`, the b12x
   native loader present.
4. Note: the r35/r36 images now read as STALE against the build dir (it
   gained profiles) — warn-only, but a rebuild makes labels match.

## P2 — R37 A/B (needs the 4x cluster; only after P0)

- `dsv41-flash-jovian-r37-4x`: boot 512K + DSpark K7 + local Engram, bench
  dev65 → compare with the **r36 datapoint** (154/176, 3,774 pp / 822 ms
  TTFT) and the **eugr prod** (167/176, 1,642 pp / 1,738 ms).
  Purpose: does R37's +43.65% prefill claim hold on GB10, and does it close
  the quality gap that kept the lab stack out of prod? This is the only
  route we have to a *faster* DS4.1 prod.
- Optional GLM arm: same image with the r35 GLM args (MTP3/DCP4/512K) →
  does the dispatch fix lift GLM prefill too? Baseline 163/176, 1,554 pp.
- Delete the r36 recipes once R37 is measured (keep both recipes until then).

## P3 — eugr nightly-20260913 (cheap, only if ≥3 h remain)

Swap `docker_image` in `dsv41-flash-4x-b12x` (or a copy) to
`eugr/spark-vllm-b12x:nightly-20260913`, boot, bench dev65 → A/B against the
nightly-20260910 prod numbers (167/176, 1,642 pp). ~2.5–3 h total.

## Guardrails

- **Never** `stop` / `power off` a serving cluster without the operator's OK.
- `serve --keep` for every first/unknown boot (default `--rm` wipes crash
  logs); the 09-08/09-13 posts-mortems depended on it.
- Poll in short loops (240–280 s); macOS has no `timeout` binary.
- Bench convention: dev65 = 88 scenarios, seed 42, `--hardmode --perf
  --concurrency 1,2`; reports auto-land in `recipes/<lane>/reports/`
  (manifest + recipe copy included).
- OOM-thrash drill: container stays "Up", workers respawn forever, sshd can
  die on 2–3 nodes → `sm121 stop`, then HA power-cycle if needed.
- One serve at a time on the 4x cluster (TP4 uses all four nodes; a stray
  second serve is an instant OOM).
- Commit locally after every milestone (git server down).

## Open questions for the operator

1. OK to take the 4× cluster off prod tonight (agent downtime), and for how
   long? P0 needs the 4 nodes exclusively.
2. Priority if time is short: (a) the full-model capability datapoint,
   (b) R37 (build + A/B), (c) both?
3. If the full model cannot hold 512K on 4×, is a smaller context
   (131K–262K) acceptable for this lane, or is 512K a hard requirement
   (= the lane is dead on this hardware)?