launcher/history/night-shifts/2026-09-13.md

Night shift 2026-09-13 → 2026-09-14

Prepared 2026-09-13 ~21:30 UTC by the assistant. Operator brief: "посмотри обновления, поищи по форумам, готовь план для ночной смены".

State at handoff

4nodes-switch	ON, serving the eugr DS4.1 prod (dsv41-flash-4x-b12x, started 13:31 UTC, 512K, DSpark k=5, Engram-on-disk + per-rank local sparse). This is the agent's own brain — confirm before stopping.
1node-5 (dgx5)	ON, idle — the build node (no RoCE / 10 GbE only, so it cannot join the 4x fabric: 5-node lanes are out).
Checkpoint	RedHatAI/GLM-5.3-MXFP4 downloading to the NAS: ~110 GB / 403.4 GB at 114 MB/s → ETA ~40 min. Verify before P0.
Kernel	all nodes on 6.17.0-1031-nvidia — do NOT upgrade (see Updates #3).
git	git.kilork.org still unreachable → commit locally; ~10 unpushed commits.

Updates found tonight (external)

1. R37 lock published 2026-09-13 (lab, DS4.1): image localinferencelab/vllm:jovian-judgement-community-20260913-r37. The headline change is a kernel-dispatch overhead removal (repeated CPU arg-list expansion, mutable-schema inspection, scalar dtype allocations, tensor-equality calls) → uncached 32K prefill +43.65% (14,896 → 21,399 tok/s), C1 output +17.7%, verifier +11.9%, Sieve +8.2% (their RTX quartet; integrated, not per-PR). Also corrects the named high reasoning budget 50 → 75. Pins: vLLM c687594b8a8 (tree 3fb88fdf…), b12x f1c4e9dd5b1 (tree 480ea232…), LMCache 29bc5a2e unchanged, DeepGEMM 8b1392b9, torch 2.13.0 / flashinfer 0.6.17 — no dependency resync needed. → GLM stays at R35 (still the latest qualified GLM lock); R37 is the DS4.1 lock. Action: build profile written → docker-images/glm53-jovian-r27/profiles/jovian-r37-ds41.env (P1).
2. eugr nightly-20260913 image built (13:09 UTC, supersedes nightly-20260910 = our DS4.1 prod image). Changelog not at the usual raw path (404) — read the repo/thread during the shift. Cheap A/B (P3).
3. ⚠ Kernel regression (forum #383023): 7.0.0-1019-nvidia breaks NCCL/RoCE — ibv_reg_mr_iova2 failed with error Cannot allocate memory during TP communication. Reproducible kernel-only A/B; root cause traced (2026-09-13) to KHO's default scratch reservation marking ~9.3 GiB as unaccounted MIGRATE_CMA; fix PR NVIDIA/NV-Kernels#590. Working: 6.17.0-1032 (and our 6.17.0-1031). Action: do not upgrade the kernel on any node; if a node ever shows RDMA ENOMEM, check uname -r first.
4. Forum #382897 ("DS4.1-Flash 552B on 4× Spark, 77.2 tok/s c1"): our upstream author (tonyd2wild — the repo our dsv41-flash image is built from). The 77 tok/s is a warmed, streaming, counting-prompt peak with up-to-5.95/6 draft acceptance; the same post lists prose 23, reasoning 39, math 47, code 52 tok/s — i.e. our measured 47–49 tg on agentic traffic is consistent; peak-on-repetition ≠ agent workload (clickbait, as the operator suspected). Two useful nuggets: 203 GB of Engram stays on disk so 552B fits 4x, and "2 GPUs were latched below 1 GHz — unplug the Spark to clear it, a reboot won't" → add a clock check before benches.
5. New lab GLM docs: models/glm-5.3-flash-spark-tp2.md (09-12, Spark **TP2
   • vision** serving & measured capacity — GLM-5.3-Flash is natively multimodal and all our GLM lanes are text-only --language-model-only; a possible capability lane later). Also models/glm-5.1-mxfp4.md and the kld/glm-5.3-flash-* quant-fidelity reports.
6. Other recent threads worth a skim only: #382939 (optimizing NVIDIA's official nvidia/GLM-5.3-Flash-NVFP4 on 2x), #382697 / #382927 (2x vs 4x comparisons), #382584 / #381911 (DS4-Flash-Vision-Exp on 1x).

P0 — GLM-5.3 (full) MXFP4 on 4× — the headline

Recipe recipes/glm53-full-mxfp4-4x (r35 stack, DCP4, 512K, KV 8 GiB pin, GMU 0.85, no spec on the first boot). Read its header for the full memory plan; the arithmetic in one line: 93.9 GiB weights/node + ~12 GiB engine + 8 GiB KV + ~4 GiB activations ≈ 118 of 121.63 GiB.

P0.0 prerequisites (~10 min)

1. Free the cluster — only after confirming with the operator (it may be serving the agent): sm121 stop --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch, wait ~12 s before any serve.
2. Download complete: du -sh $NAS/models--RedHatAI--GLM-5.3-MXFP4 ≈ 403 GB and 141 shards present (an .incomplete file = resume with huggingface-cli download RedHatAI/GLM-5.3-MXFP4). HF_HUB_OFFLINE=1 makes a missing shard a hard load failure.
3. Clock check (nugget #4): per node nvidia-smi --query-gpu=clocks.sm,clocks_event_reasons.active --format=csv,noheader — a SM clock latched below ~1 GHz invalidates every bench number.
4. sm121 health --cluster 4nodes-switch.

P0.1 boot probe — conservative (~60–90 min)

Edit the recipe args: remove --decode-context-parallel-size 4 and --cp-kv-cache-interleave-size 4; set --kv-cache-memory-bytes 4294967296 (4 GiB) and --max-model-len 131072. Then:

sm121 serve --keep --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch

Poll with docker logs --tail 2 sm121_glm53-full-mxfp4-4x_0 every 240–280 s.

• Expect ~94 GiB/rank of weights (~10–20 min via instanttensor).
• ABORT SIGNAL — a second "Loading safetensors" pass: that is the 2026-09-08 double-load pattern that OOM-thrashed all four nodes (sshd down, HA power-cycle). sm121 stop immediately; do not wait to see if it settles.
• Success markers to record verbatim: Model loading took X GiB, the GPU KV cache size: … line, Application startup complete.
• The KV line is the most valuable number of the night — it gives the real bytes/token for the full model (the parked estimate was ~54 KB/token; everything about 512K depends on it).
• If the boot dies on memory: record the numbers and go to P0.4.

P0.2 smoke + capability (~20 min)

• Chat: chat_template_kwargs: {"reasoning_effort":"low"} (glm45 parser, glm47 tool parser). Check coherent content + a tool-call round-trip.
• Token-loop check (the W4A4-on-new-arch risk): generate ~300 tokens and confirm no repeated-token degeneration.
• Long-context sanity if the pool allows: filler + needle at the largest context that fits, temperature 0 (the operator's floor is 512K — this is the capability claim).

P0.3 the 512K attempt (DCP4) — (~90 min + bench)

Restore the DCP4 lines, --max-model-len 524288, KV pin 8 GiB. Boot again.

• If it rejects DCP (the GLM C4 DCP requires cp_kv_cache_interleave_size to be divisible by 4 / page-stride class that blocks DFlash2 on 2x) → the full model is DCP-incompatible; decision point (P0.4).
• Verify the KV line reports concurrency at 524288.
• Bench: sm121 bench --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch --label dev65-full-mxfp4-dcp4 (~2.5 h, under pty automatically).
• Compare against the GLM-5.3-Flash r35 prod anchor: 163/176 (93), 1,554 pp / 49.6 tg / 1,544 ms TTFT @d0 c1, KV 8.06M @512K. A ~740B model beating that on quality would be the story of the week; matching prefill while holding 512K would already make it a capability lane.

P0.4 fallbacks if it does not fit / degrades

• Bench the P0.1 shape (no DCP, 131072, 4 GiB KV) as the honest capability datapoint.
• 5–6 nodes are not available (dgx5 is 10 GbE-only, off the RoCE fabric), so 4x is the ceiling for this model tonight.
• If it thrashes: sm121 stop; if the nodes do not recover, HA power-cycle (sm121 power off/on --cluster 4nodes-switch, ~3–5 min boot).
• Record "full GLM-5.3 on 4x = does not fit / fits with N ctx" — a negative is a valuable result here (it retires the question).

P0.5 teardown (10 min)

sm121 stop; restore prod if wanted (sm121 serve --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch); write the report + history entry + commit locally.

P1 — build the R37 image (background, dgx5, no GPU contention)

The profile is written and dep-checked: profiles/jovian-r37-ds41.env (vLLM c687594b8a8, b12x f1c4e9dd5b1, LMCache/FlashKDA/DeepGEMM unchanged, torch 2.13.0 / FI 0.6.17 same as our GB10 pins).

1. Create the recipe pair (copy, then swap 3 lines each):
   • recipes/dsv41-flash-jovian-r36-4x → recipes/dsv41-flash-jovian-r37-4x (docker_image = "jovian-r37-ds41:local", build_profile = "jovian-r37-ds41"; keeps the composed_model local-Engram hardlinks + 512K + DSpark K7 args).
   • optionally recipes/glm53-flash-r35-4x-dcp4 → recipes/glm53-flash-r37-4x-dcp4 (same image, GLM args) — the image is two-layer, so one build serves both lanes.
2. sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan (~14 min warm buildx cache) → transfer to dgx1–4 (~4 min, 16 GB, parallel docker save | ssh | docker load).
3. Sanity checks in the image (read-only, cheap): GlmMoeDsaForCausalLM in registry.py, mxfp4-pack-quantized scheme, liburing.so.2, the b12x native loader present.
4. Note: the r35/r36 images now read as STALE against the build dir (it gained profiles) — warn-only, but a rebuild makes labels match.

P2 — R37 A/B (needs the 4x cluster; only after P0)

• dsv41-flash-jovian-r37-4x: boot 512K + DSpark K7 + local Engram, bench dev65 → compare with the r36 datapoint (154/176, 3,774 pp / 822 ms TTFT) and the eugr prod (167/176, 1,642 pp / 1,738 ms). Purpose: does R37's +43.65% prefill claim hold on GB10, and does it close the quality gap that kept the lab stack out of prod? This is the only route we have to a faster DS4.1 prod.
• Optional GLM arm: same image with the r35 GLM args (MTP3/DCP4/512K) → does the dispatch fix lift GLM prefill too? Baseline 163/176, 1,554 pp.
• Delete the r36 recipes once R37 is measured (keep both recipes until then).

P3 — eugr nightly-20260913 (cheap, only if ≥3 h remain)

Swap docker_image in dsv41-flash-4x-b12x (or a copy) to eugr/spark-vllm-b12x:nightly-20260913, boot, bench dev65 → A/B against the nightly-20260910 prod numbers (167/176, 1,642 pp). ~2.5–3 h total.

Guardrails

• Never stop / power off a serving cluster without the operator's OK.
• serve --keep for every first/unknown boot (default --rm wipes crash logs); the 09-08/09-13 posts-mortems depended on it.
• Poll in short loops (240–280 s); macOS has no timeout binary.
• Bench convention: dev65 = 88 scenarios, seed 42, --hardmode --perf --concurrency 1,2; reports auto-land in recipes/<lane>/reports/ (manifest + recipe copy included).
• OOM-thrash drill: container stays "Up", workers respawn forever, sshd can die on 2–3 nodes → sm121 stop, then HA power-cycle if needed.
• One serve at a time on the 4x cluster (TP4 uses all four nodes; a stray second serve is an instant OOM).
• Commit locally after every milestone (git server down).

Open questions for the operator

1. OK to take the 4× cluster off prod tonight (agent downtime), and for how long? P0 needs the 4 nodes exclusively.
2. Priority if time is short: (a) the full-model capability datapoint, (b) R37 (build + A/B), (c) both?
3. If the full model cannot hold 512K on 4×, is a smaller context (131K–262K) acceptable for this lane, or is 512K a hard requirement (= the lane is dead on this hardware)?