launcher/history/night-shifts/2026-09-13.md

Night shift 2026-09-13 → 2026-09-14

Prepared 2026-09-13 ~21:30 UTC by the assistant. Operator brief: "посмотри обновления, поищи по форумам, готовь план для ночной смены".

State at handoff

4nodes-switch	ON, serving the eugr DS4.1 prod (dsv41-flash-4x-b12x, started 13:31 UTC, 512K, DSpark k=5, Engram-on-disk + per-rank local sparse). This is the agent's own brain — confirm before stopping.
1node-5 (dgx5)	ON, idle — the build node (no RoCE / 10 GbE only, so it cannot join the 4x fabric: 5-node lanes are out).
Checkpoint	RedHatAI/GLM-5.3-MXFP4 downloading to the NAS: ~110 GB / 403.4 GB at 114 MB/s → ETA ~40 min. Verify before P0.
Kernel	all nodes on 6.17.0-1031-nvidia — do NOT upgrade (see Updates #3).
git	git.kilork.org still unreachable → commit locally; ~10 unpushed commits.

Updates found tonight (external)

1. R37 lock published 2026-09-13 (lab, DS4.1): image localinferencelab/vllm:jovian-judgement-community-20260913-r37. The headline change is a kernel-dispatch overhead removal (repeated CPU arg-list expansion, mutable-schema inspection, scalar dtype allocations, tensor-equality calls) → uncached 32K prefill +43.65% (14,896 → 21,399 tok/s), C1 output +17.7%, verifier +11.9%, Sieve +8.2% (their RTX quartet; integrated, not per-PR). Also corrects the named high reasoning budget 50 → 75. Pins: vLLM c687594b8a8 (tree 3fb88fdf…), b12x f1c4e9dd5b1 (tree 480ea232…), LMCache 29bc5a2e unchanged, DeepGEMM 8b1392b9, torch 2.13.0 / flashinfer 0.6.17 — no dependency resync needed. → GLM stays at R35 (still the latest qualified GLM lock); R37 is the DS4.1 lock. Action: build profile written → docker-images/glm53-jovian-r27/profiles/jovian-r37-ds41.env (P1).
2. eugr nightly-20260913 image built (13:09 UTC, supersedes nightly-20260910 = our DS4.1 prod image). Changelog not at the usual raw path (404) — read the repo/thread during the shift. Cheap A/B (P3).
3. ⚠ Kernel regression (forum #383023): 7.0.0-1019-nvidia breaks NCCL/RoCE — ibv_reg_mr_iova2 failed with error Cannot allocate memory during TP communication. Reproducible kernel-only A/B; root cause traced (2026-09-13) to KHO's default scratch reservation marking ~9.3 GiB as unaccounted MIGRATE_CMA; fix PR NVIDIA/NV-Kernels#590. Working: 6.17.0-1032 (and our 6.17.0-1031). Action: do not upgrade the kernel on any node; if a node ever shows RDMA ENOMEM, check uname -r first.
4. Forum #382897 ("DS4.1-Flash 552B on 4× Spark, 77.2 tok/s c1"): our upstream author (tonyd2wild — the repo our dsv41-flash image is built from). The 77 tok/s is a warmed, streaming, counting-prompt peak with up-to-5.95/6 draft acceptance; the same post lists prose 23, reasoning 39, math 47, code 52 tok/s — i.e. our measured 47–49 tg on agentic traffic is consistent; peak-on-repetition ≠ agent workload (clickbait, as the operator suspected). Two useful nuggets: 203 GB of Engram stays on disk so 552B fits 4x, and "2 GPUs were latched below 1 GHz — unplug the Spark to clear it, a reboot won't" → add a clock check before benches.
5. New lab GLM docs: models/glm-5.3-flash-spark-tp2.md (09-12, Spark **TP2
   • vision** serving & measured capacity — GLM-5.3-Flash is natively multimodal and all our GLM lanes are text-only --language-model-only; a possible capability lane later). Also models/glm-5.1-mxfp4.md and the kld/glm-5.3-flash-* quant-fidelity reports.
6. Other recent threads worth a skim only: #382939 (optimizing NVIDIA's official nvidia/GLM-5.3-Flash-NVFP4 on 2x), #382697 / #382927 (2x vs 4x comparisons), #382584 / #381911 (DS4-Flash-Vision-Exp on 1x).

P0 — GLM-5.3 (full) MXFP4 on 4× — the headline

Recipe recipes/glm53-full-mxfp4-4x (r35 stack, DCP4, 512K, KV 8 GiB pin, GMU 0.85, no spec on the first boot). Read its header for the full memory plan; the arithmetic in one line: 93.9 GiB weights/node + ~12 GiB engine + 8 GiB KV + ~4 GiB activations ≈ 118 of 121.63 GiB.

P0.0 prerequisites (~10 min)

1. Free the cluster — only after confirming with the operator (it may be serving the agent): sm121 stop --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch, wait ~12 s before any serve.
2. Download complete: du -sh $NAS/models--RedHatAI--GLM-5.3-MXFP4 ≈ 403 GB and 141 shards present (an .incomplete file = resume with huggingface-cli download RedHatAI/GLM-5.3-MXFP4). HF_HUB_OFFLINE=1 makes a missing shard a hard load failure.
3. Clock check (nugget #4): per node nvidia-smi --query-gpu=clocks.sm,clocks_event_reasons.active --format=csv,noheader — a SM clock latched below ~1 GHz invalidates every bench number.
4. sm121 health --cluster 4nodes-switch.

P0.1 boot probe — conservative (~60–90 min)

Edit the recipe args: remove --decode-context-parallel-size 4 and --cp-kv-cache-interleave-size 4; set --kv-cache-memory-bytes 4294967296 (4 GiB) and --max-model-len 131072. Then:

sm121 serve --keep --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch

Poll with docker logs --tail 2 sm121_glm53-full-mxfp4-4x_0 every 240–280 s.

• Expect ~94 GiB/rank of weights (~10–20 min via instanttensor).
• ABORT SIGNAL — a second "Loading safetensors" pass: that is the 2026-09-08 double-load pattern that OOM-thrashed all four nodes (sshd down, HA power-cycle). sm121 stop immediately; do not wait to see if it settles.
• Success markers to record verbatim: Model loading took X GiB, the GPU KV cache size: … line, Application startup complete.
• The KV line is the most valuable number of the night — it gives the real bytes/token for the full model (the parked estimate was ~54 KB/token; everything about 512K depends on it).
• If the boot dies on memory: record the numbers and go to P0.4.

P0.2 smoke + capability (~20 min)

• Chat: chat_template_kwargs: {"reasoning_effort":"low"} (glm45 parser, glm47 tool parser). Check coherent content + a tool-call round-trip.
• Token-loop check (the W4A4-on-new-arch risk): generate ~300 tokens and confirm no repeated-token degeneration.
• Long-context sanity if the pool allows: filler + needle at the largest context that fits, temperature 0 (the operator's floor is 512K — this is the capability claim).

P0.3 the 512K attempt (DCP4) — (~90 min + bench)

Restore the DCP4 lines, --max-model-len 524288, KV pin 8 GiB. Boot again.

• If it rejects DCP (the GLM C4 DCP requires cp_kv_cache_interleave_size to be divisible by 4 / page-stride class that blocks DFlash2 on 2x) → the full model is DCP-incompatible; decision point (P0.4).
• Verify the KV line reports concurrency at 524288.
• Bench: sm121 bench --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch --label dev65-full-mxfp4-dcp4 (~2.5 h, under pty automatically).
• Compare against the GLM-5.3-Flash r35 prod anchor: 163/176 (93), 1,554 pp / 49.6 tg / 1,544 ms TTFT @d0 c1, KV 8.06M @512K. A ~740B model beating that on quality would be the story of the week; matching prefill while holding 512K would already make it a capability lane.

P0.4 fallbacks if it does not fit / degrades

• Bench the P0.1 shape (no DCP, 131072, 4 GiB KV) as the honest capability datapoint.
• 5–6 nodes are not available (dgx5 is 10 GbE-only, off the RoCE fabric), so 4x is the ceiling for this model tonight.
• If it thrashes: sm121 stop; if the nodes do not recover, HA power-cycle (sm121 power off/on --cluster 4nodes-switch, ~3–5 min boot).
• Record "full GLM-5.3 on 4x = does not fit / fits with N ctx" — a negative is a valuable result here (it retires the question).

P0.5 teardown (10 min)

sm121 stop; restore prod if wanted (sm121 serve --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch); write the report + history entry + commit locally.

P1 — build the R37 image (background, dgx5, no GPU contention)

The profile is written and dep-checked: profiles/jovian-r37-ds41.env (vLLM c687594b8a8, b12x f1c4e9dd5b1, LMCache/FlashKDA/DeepGEMM unchanged, torch 2.13.0 / FI 0.6.17 same as our GB10 pins).

1. Create the recipe pair (copy, then swap 3 lines each):
   • recipes/dsv41-flash-jovian-r36-4x → recipes/dsv41-flash-jovian-r37-4x (docker_image = "jovian-r37-ds41:local", build_profile = "jovian-r37-ds41"; keeps the composed_model local-Engram hardlinks + 512K + DSpark K7 args).
   • optionally recipes/glm53-flash-r35-4x-dcp4 → recipes/glm53-flash-r37-4x-dcp4 (same image, GLM args) — the image is two-layer, so one build serves both lanes.
2. sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan (~14 min warm buildx cache) → transfer to dgx1–4 (~4 min, 16 GB, parallel docker save | ssh | docker load).
3. Sanity checks in the image (read-only, cheap): GlmMoeDsaForCausalLM in registry.py, mxfp4-pack-quantized scheme, liburing.so.2, the b12x native loader present.
4. Note: the r35/r36 images now read as STALE against the build dir (it gained profiles) — warn-only, but a rebuild makes labels match.

P2 — R37 A/B (needs the 4x cluster; only after P0)

• dsv41-flash-jovian-r37-4x: boot 512K + DSpark K7 + local Engram, bench dev65 → compare with the r36 datapoint (154/176, 3,774 pp / 822 ms TTFT) and the eugr prod (167/176, 1,642 pp / 1,738 ms). Purpose: does R37's +43.65% prefill claim hold on GB10, and does it close the quality gap that kept the lab stack out of prod? This is the only route we have to a faster DS4.1 prod.
• Optional GLM arm: same image with the r35 GLM args (MTP3/DCP4/512K) → does the dispatch fix lift GLM prefill too? Baseline 163/176, 1,554 pp.
• Delete the r36 recipes once R37 is measured (keep both recipes until then).

P3 — eugr nightly-20260913 (cheap, only if ≥3 h remain)

Swap docker_image in dsv41-flash-4x-b12x (or a copy) to eugr/spark-vllm-b12x:nightly-20260913, boot, bench dev65 → A/B against the nightly-20260910 prod numbers (167/176, 1,642 pp). ~2.5–3 h total.

Guardrails

• Never stop / power off a serving cluster without the operator's OK.
• serve --keep for every first/unknown boot (default --rm wipes crash logs); the 09-08/09-13 posts-mortems depended on it.
• Poll in short loops (240–280 s); macOS has no timeout binary.
• Bench convention: dev65 = 88 scenarios, seed 42, --hardmode --perf --concurrency 1,2; reports auto-land in recipes/<lane>/reports/ (manifest + recipe copy included).
• OOM-thrash drill: container stays "Up", workers respawn forever, sshd can die on 2–3 nodes → sm121 stop, then HA power-cycle if needed.
• One serve at a time on the 4x cluster (TP4 uses all four nodes; a stray second serve is an instant OOM).
• Commit locally after every milestone (git server down).

Open questions for the operator

1. OK to take the 4× cluster off prod tonight (agent downtime), and for how long? P0 needs the 4 nodes exclusively.
2. Priority if time is short: (a) the full-model capability datapoint, (b) R37 (build + A/B), (c) both?
3. If the full model cannot hold 512K on 4×, is a smaller context (131K–262K) acceptable for this lane, or is 512K a hard requirement (= the lane is dead on this hardware)?

---

Amendments 2026-09-13 ~22:00 UTC (after reading the lab's GLM docs)

Operator: build R37 first, then read models/glm-5.3-flash-spark-tp2.md and models/glm-5.1-mxfp4.md properly ("GLM-5.3 did not fundamentally change the architecture") and fold the logical consequences into the plan.

R37 build started

sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan (recipe created as a clone of the r36 one with image/profile swapped; the build fetches c687594b8a8 from voipmonitor/vllm + f1c4e9dd5b1 from voipmonitor/b12x with the HEAD/tree gates — both present upstream). Log: /tmp/build-r37.log. Next after it lands: transfer to dgx1–4, then P2.

P0 amendment — the MXFP4 MoE backend must be explicit

The r35/r37 tree's default MoE backend is b12x (r35 changelog). The lab's GLM-5.1 MXFP4 runbook says for MXFP4 GLM experts: --moe-backend flashinfer_cutlass_afp8 → native FlashInfer Cutlass MXFP4×MXFP8; do not set B12X_MOE_FORCE_A16; GLM must get standard SwiGLU (gpt-oss alpha/beta/clamp defaults must not apply). The recipe now sets flashinfer_cutlass_afp8 explicitly.

• Already in our Sept tree (verified by image inspection): oracle/mxfp4.py:297 maps that name to FLASHINFER_CUTLASS_MXFP4_MXFP8; quantization/mxfp4.py routes non-gpt_oss to the native Mxfp4MoEMethod; the gpt-oss SwiGLU gates in experts/flashinfer_cutlass_moe.py:270-280 abstain — i.e. the June GLM-5.1 MXFP4 vLLM changes are merged.
• RedHatAI ships quant_method=compressed-tensors / mxfp4-pack-quantized (CompressedTensorsW4A4Mxfp4), a different ingestion path from the lab's quant_method=mxfp4 — hence the fallback order: flashinfer_cutlass_afp8 → b12x (proven on our DS4.1 MXFP4 experts) → drop the flag and read the engine's selected backend from the log.
• Numerics gate either way: a missing SwiGLU clamp (r33/r34 lesson) and W4A4 token-loop corruption both present as degenerate repetition. Run the P0.2 smoke with ~300 tokens and look at it, before any bench.

P4 — capability lanes worth a logical slot (from the lab docs)

(a) LMCache RAM + filesystem tiers → long context on a tiny GPU KV pool. Their TP2 GLM-5.3-Flash contract is MAX_MODEL_LEN=1048576 with only KV_CACHE_MEMORY_BYTES=4294967296 (4 GiB/rank) and a 1,048,320-token cold text run in 139.7 s, RAM restore 0.808 s, filesystem restore 1.279 s, zero recompute. Options (before the image name): -e CACHE_MODE=lmcache -e LMCACHE_L1_SIZE_GB=64 -e LMCACHE_L2_ROOT=/cache/lmcache-l2. Caveats: the 64 GiB L1 arena is preallocated pinned host shm, so host /dev/shm must be big enough (Docker --shm-size does NOT resize host shm); LMCACHE_L2_ENABLED=0 = RAM-only; the /cache volume persists FS objects across container replacement; objects cover 4,096 global tokens, attention pages 2,048 tokens per DCP rank; checkpoint files are hashed at startup (first log delayed while weights are read — keep them immutable while serving). Why it matters for us: it decouples "≥512K context" from "GPU KV pool size" — exactly the wall the full MXFP4 model hits on 4×. Worth a lane on the r35 GLM Flash profile (cheap: env only, same image).

(b) GLM vision on 2 GPUs. The lab's TP2/DCP2 GLM-5.3-Flash contract includes vision: one image per API request (across all turns), native ~8,000-feature ceiling, video disabled, full-image FlashAttention for the image pass, target KV FP8 + FP32 recurrent state, MTP3 probabilistic. Cold 1,048,320-token image input: correct answer in 143.0 s with min free VRAM 232.31 MiB/GPU. All our GLM lanes are text-only (--language-model-only) — this is a capability upgrade (screenshots/diagrams in agent traffic), and it is a documented, qualified contract rather than guesswork.

(c) Two conventions to copy into our recipes.

• Explicit MAX_MODEL_LEN + explicit KV_CACHE_MEMORY_BYTES (they never let a lane silently shorten context; KV_CACHE_MEMORY_BYTES=auto is for auto-profiling and MAX_MODEL_LEN=-1 is the opt-in for accepting a shorter context; GMU does not resize an explicit KV allocation). We already pin both — keep it that way, and quote the KV line in every run report.
• CUBLAS_WORKSPACE_CONFIG=:4096:1 bounds the cuBLAS workspace to 4 MiB per handle → 140 MiB less live allocation per rank (they measured it; not a confirmed cure for every community cuBLAS crash). Cheap, consider for the full-model lane where every 100 MiB counts.

Important caveat from their doc: "The memory mechanisms can also apply to TP4, but their magnitude depends on sharding and workload. No TP4 memory or performance improvement is qualified by the TP2 measurements. The TP2 NCCL configuration is not a TP4 recommendation." (They use 2 NCCL channels / 1 MiB buffer on TP2; we run 16 channels / 2 MiB on TP4 — do not copy the TP2 values.)

Reference numbers from that doc (2× GB10, TP2/DCP2, MTP3, 4 GiB KV/rank, 1M ctx): 32K prefill 10,648.9 tok/s; C1 MTP3 median output 170.77 tok/s; verifier 67.5 steps/s; R2 vs R1 = +0.41% prefill / +4.52% C1.