# Night shift 2026-09-13 → 2026-09-14

Prepared 2026-09-13 ~21:30 UTC by the assistant. Operator brief: "посмотри
обновления, поищи по форумам, готовь план для ночной смены".

## State at handoff

| | |
|---|---|
| `4nodes-switch` | ON, **serving the eugr DS4.1 prod** (`dsv41-flash-4x-b12x`, started 13:31 UTC, 512K, DSpark k=5, Engram-on-disk + per-rank local sparse). This is the agent's own brain — **confirm before stopping**. |
| `1node-5` (dgx5) | ON, idle — the build node (no RoCE / 10 GbE only, so it cannot join the 4x fabric: 5-node lanes are out). |
| Checkpoint | `RedHatAI/GLM-5.3-MXFP4` downloading to the NAS: ~110 GB / 403.4 GB at **114 MB/s → ETA ~40 min**. Verify before P0. |
| Kernel | all nodes on **6.17.0-1031-nvidia** — do NOT upgrade (see Updates #3). |
| git | `git.kilork.org` still unreachable → commit locally; ~10 unpushed commits. |

## Updates found tonight (external)

1. **R37 lock published 2026-09-13** (lab, DS4.1): image
   `localinferencelab/vllm:jovian-judgement-community-20260913-r37`. The
   headline change is a **kernel-dispatch overhead removal** (repeated CPU
   arg-list expansion, mutable-schema inspection, scalar dtype allocations,
   tensor-equality calls) → **uncached 32K prefill +43.65%** (14,896 → 21,399
   tok/s), C1 output +17.7%, verifier +11.9%, Sieve +8.2% (their RTX quartet;
   integrated, not per-PR). Also corrects the named `high` reasoning budget
   50 → 75. Pins: vLLM `c687594b8a8` (tree `3fb88fdf…`), b12x `f1c4e9dd5b1`
   (tree `480ea232…`), LMCache `29bc5a2e` unchanged, DeepGEMM `8b1392b9`,
   torch 2.13.0 / flashinfer 0.6.17 — **no dependency resync needed**.
   → *GLM* stays at **R35** (still the latest qualified GLM lock); R37 is the
   DS4.1 lock. **Action: build profile written** →
   `docker-images/glm53-jovian-r27/profiles/jovian-r37-ds41.env` (P1).
2. **eugr `nightly-20260913`** image built (13:09 UTC, supersedes
   `nightly-20260910` = our DS4.1 prod image). Changelog not at the usual raw
   path (404) — read the repo/thread during the shift. Cheap A/B (P3).
3. **⚠ Kernel regression (forum #383023):** `7.0.0-1019-nvidia` breaks
   NCCL/RoCE — `ibv_reg_mr_iova2 failed with error Cannot allocate memory`
   during TP communication. Reproducible kernel-only A/B; root cause traced
   (2026-09-13) to **KHO's default scratch reservation marking ~9.3 GiB as
   unaccounted `MIGRATE_CMA`**; fix PR NVIDIA/NV-Kernels#590. Working:
   `6.17.0-1032` (and our `6.17.0-1031`). **Action: do not upgrade the
   kernel on any node; if a node ever shows RDMA ENOMEM, check `uname -r`
   first.**
4. **Forum #382897** ("DS4.1-Flash 552B on 4× Spark, 77.2 tok/s c1"): our
   upstream author (tonyd2wild — the repo our `dsv41-flash` image is built
   from). The 77 tok/s is a **warmed, streaming, counting-prompt peak** with
   up-to-5.95/6 draft acceptance; the same post lists **prose 23, reasoning
   39, math 47, code 52** tok/s — i.e. our measured 47–49 tg on agentic
   traffic is consistent; peak-on-repetition ≠ agent workload (clickbait, as
   the operator suspected). Two useful nuggets: 203 GB of Engram stays on
   disk so 552B fits 4x, and **"2 GPUs were latched below 1 GHz — unplug the
   Spark to clear it, a reboot won't"** → add a clock check before benches.
5. New lab GLM docs: `models/glm-5.3-flash-spark-tp2.md` (09-12, Spark **TP2
   + vision** serving & measured capacity — GLM-5.3-Flash is natively
   multimodal and all our GLM lanes are text-only `--language-model-only`; a
   possible capability lane later). Also `models/glm-5.1-mxfp4.md` and the
   `kld/glm-5.3-flash-*` quant-fidelity reports.
6. Other recent threads worth a skim only: #382939 (optimizing NVIDIA's
   official `nvidia/GLM-5.3-Flash-NVFP4` on 2x), #382697 / #382927 (2x vs 4x
   comparisons), #382584 / #381911 (DS4-Flash-Vision-Exp on 1x).

## P0 — GLM-5.3 (full) MXFP4 on 4× — the headline

Recipe `recipes/glm53-full-mxfp4-4x` (r35 stack, DCP4, 512K, KV 8 GiB pin,
GMU 0.85, no spec on the first boot). Read its header for the full memory
plan; the arithmetic in one line: **93.9 GiB weights/node + ~12 GiB engine +
8 GiB KV + ~4 GiB activations ≈ 118 of 121.63 GiB**.

### P0.0 prerequisites (~10 min)
1. **Free the cluster** — only after confirming with the operator (it may be
   serving the agent):
   `sm121 stop --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch`, wait
   ~12 s before any serve.
2. **Download complete**: `du -sh $NAS/models--RedHatAI--GLM-5.3-MXFP4` ≈
   403 GB and **141 shards** present (an `.incomplete` file = resume with
   `huggingface-cli download RedHatAI/GLM-5.3-MXFP4`). HF_HUB_OFFLINE=1 makes
   a missing shard a hard load failure.
3. **Clock check** (nugget #4): per node
   `nvidia-smi --query-gpu=clocks.sm,clocks_event_reasons.active --format=csv,noheader`
   — a SM clock latched below ~1 GHz invalidates every bench number.
4. `sm121 health --cluster 4nodes-switch`.

### P0.1 boot probe — conservative (~60–90 min)
Edit the recipe args: **remove** `--decode-context-parallel-size 4` and
`--cp-kv-cache-interleave-size 4`; set `--kv-cache-memory-bytes 4294967296`
(4 GiB) and `--max-model-len 131072`. Then:

```
sm121 serve --keep --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch
```

Poll with `docker logs --tail 2 sm121_glm53-full-mxfp4-4x_0` every 240–280 s.

- Expect ~94 GiB/rank of weights (~10–20 min via instanttensor).
- **ABORT SIGNAL — a second "Loading safetensors" pass**: that is the
  2026-09-08 double-load pattern that OOM-thrashed all four nodes (sshd down,
  HA power-cycle). `sm121 stop` immediately; do not wait to see if it settles.
- Success markers to record verbatim: `Model loading took X GiB`, the
  `GPU KV cache size: …` line, `Application startup complete`.
- **The KV line is the most valuable number of the night** — it gives the
  real bytes/token for the full model (the parked estimate was ~54 KB/token;
  everything about 512K depends on it).
- If the boot dies on memory: record the numbers and go to P0.4.

### P0.2 smoke + capability (~20 min)
- Chat: `chat_template_kwargs: {"reasoning_effort":"low"}` (glm45 parser,
  glm47 tool parser). Check coherent content + a tool-call round-trip.
- **Token-loop check** (the W4A4-on-new-arch risk): generate ~300 tokens and
  confirm no repeated-token degeneration.
- Long-context sanity if the pool allows: filler + needle at the largest
  context that fits, temperature 0 (the operator's floor is 512K — this is
  the capability claim).

### P0.3 the 512K attempt (DCP4) — (~90 min + bench)
Restore the DCP4 lines, `--max-model-len 524288`, KV pin 8 GiB. Boot again.
- If it rejects DCP (the `GLM C4 DCP requires cp_kv_cache_interleave_size to
  be divisible by 4` / page-stride class that blocks DFlash2 on 2x) → the
  full model is DCP-incompatible; decision point (P0.4).
- Verify the KV line reports concurrency at 524288.
- Bench: `sm121 bench --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch
  --label dev65-full-mxfp4-dcp4` (~2.5 h, under pty automatically).
- Compare against the **GLM-5.3-Flash r35 prod anchor**: 163/176 (93),
  1,554 pp / 49.6 tg / 1,544 ms TTFT @d0 c1, KV 8.06M @512K. A ~740B model
  beating that on quality would be the story of the week; matching prefill
  while holding 512K would already make it a capability lane.

### P0.4 fallbacks if it does not fit / degrades
- Bench the P0.1 shape (no DCP, 131072, 4 GiB KV) as the honest capability
  datapoint.
- 5–6 nodes are **not** available (dgx5 is 10 GbE-only, off the RoCE fabric),
  so 4x is the ceiling for this model tonight.
- If it thrashes: `sm121 stop`; if the nodes do not recover, HA power-cycle
  (`sm121 power off/on --cluster 4nodes-switch`, ~3–5 min boot).
- Record "full GLM-5.3 on 4x = does not fit / fits with N ctx" — a negative
  is a valuable result here (it retires the question).

### P0.5 teardown (10 min)
`sm121 stop`; restore prod if wanted
(`sm121 serve --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch`);
write the report + history entry + commit locally.

## P1 — build the R37 image (background, dgx5, no GPU contention)

The profile is written and dep-checked: `profiles/jovian-r37-ds41.env`
(vLLM `c687594b8a8`, b12x `f1c4e9dd5b1`, LMCache/FlashKDA/DeepGEMM unchanged,
torch 2.13.0 / FI 0.6.17 same as our GB10 pins).

1. Create the recipe pair (copy, then swap 3 lines each):
   - `recipes/dsv41-flash-jovian-r36-4x` → `recipes/dsv41-flash-jovian-r37-4x`
     (`docker_image = "jovian-r37-ds41:local"`,
     `build_profile = "jovian-r37-ds41"`; keeps the composed_model
     local-Engram hardlinks + 512K + DSpark K7 args).
   - optionally `recipes/glm53-flash-r35-4x-dcp4` →
     `recipes/glm53-flash-r37-4x-dcp4` (same image, GLM args) — the image is
     two-layer, so one build serves both lanes.
2. `sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan`
   (~14 min warm buildx cache) → transfer to dgx1–4 (~4 min, 16 GB, parallel
   `docker save | ssh | docker load`).
3. Sanity checks in the image (read-only, cheap): `GlmMoeDsaForCausalLM` in
   `registry.py`, `mxfp4-pack-quantized` scheme, `liburing.so.2`, the b12x
   native loader present.
4. Note: the r35/r36 images now read as STALE against the build dir (it
   gained profiles) — warn-only, but a rebuild makes labels match.

## P2 — R37 A/B (needs the 4x cluster; only after P0)

- `dsv41-flash-jovian-r37-4x`: boot 512K + DSpark K7 + local Engram, bench
  dev65 → compare with the **r36 datapoint** (154/176, 3,774 pp / 822 ms
  TTFT) and the **eugr prod** (167/176, 1,642 pp / 1,738 ms).
  Purpose: does R37's +43.65% prefill claim hold on GB10, and does it close
  the quality gap that kept the lab stack out of prod? This is the only
  route we have to a *faster* DS4.1 prod.
- Optional GLM arm: same image with the r35 GLM args (MTP3/DCP4/512K) →
  does the dispatch fix lift GLM prefill too? Baseline 163/176, 1,554 pp.
- Delete the r36 recipes once R37 is measured (keep both recipes until then).

## P3 — eugr nightly-20260913 (cheap, only if ≥3 h remain)

Swap `docker_image` in `dsv41-flash-4x-b12x` (or a copy) to
`eugr/spark-vllm-b12x:nightly-20260913`, boot, bench dev65 → A/B against the
nightly-20260910 prod numbers (167/176, 1,642 pp). ~2.5–3 h total.

## Guardrails

- **Never** `stop` / `power off` a serving cluster without the operator's OK.
- `serve --keep` for every first/unknown boot (default `--rm` wipes crash
  logs); the 09-08/09-13 posts-mortems depended on it.
- Poll in short loops (240–280 s); macOS has no `timeout` binary.
- Bench convention: dev65 = 88 scenarios, seed 42, `--hardmode --perf
  --concurrency 1,2`; reports auto-land in `recipes/<lane>/reports/`
  (manifest + recipe copy included).
- OOM-thrash drill: container stays "Up", workers respawn forever, sshd can
  die on 2–3 nodes → `sm121 stop`, then HA power-cycle if needed.
- One serve at a time on the 4x cluster (TP4 uses all four nodes; a stray
  second serve is an instant OOM).
- Commit locally after every milestone (git server down).

## Open questions for the operator

1. OK to take the 4× cluster off prod tonight (agent downtime), and for how
   long? P0 needs the 4 nodes exclusively.
2. Priority if time is short: (a) the full-model capability datapoint,
   (b) R37 (build + A/B), (c) both?
3. If the full model cannot hold 512K on 4×, is a smaller context
   (131K–262K) acceptable for this lane, or is 512K a hard requirement
   (= the lane is dead on this hardware)?

---

## Amendments 2026-09-13 ~22:00 UTC (after reading the lab's GLM docs)

Operator: build R37 first, then read `models/glm-5.3-flash-spark-tp2.md` and
`models/glm-5.1-mxfp4.md` properly ("GLM-5.3 did not fundamentally change the
architecture") and fold the logical consequences into the plan.

### R37 build started
`sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan`
(recipe created as a clone of the r36 one with image/profile swapped; the
build fetches `c687594b8a8` from voipmonitor/vllm + `f1c4e9dd5b1` from
voipmonitor/b12x with the HEAD/tree gates — both present upstream). Log:
`/tmp/build-r37.log`. Next after it lands: transfer to dgx1–4, then P2.

### P0 amendment — the MXFP4 MoE backend must be explicit
The r35/r37 tree's **default MoE backend is b12x** (r35 changelog). The lab's
GLM-5.1 MXFP4 runbook says for MXFP4 GLM experts:
`--moe-backend flashinfer_cutlass_afp8` → native **FlashInfer Cutlass
MXFP4×MXFP8**; do **not** set `B12X_MOE_FORCE_A16`; GLM must get **standard
SwiGLU** (gpt-oss alpha/beta/clamp defaults must not apply).
The recipe now sets `flashinfer_cutlass_afp8` explicitly.
- Already in our Sept tree (verified by image inspection): `oracle/mxfp4.py:297`
  maps that name to `FLASHINFER_CUTLASS_MXFP4_MXFP8`; `quantization/mxfp4.py`
  routes non-gpt_oss to the native `Mxfp4MoEMethod`; the gpt-oss SwiGLU gates
  in `experts/flashinfer_cutlass_moe.py:270-280` abstain — i.e. the June
  GLM-5.1 MXFP4 vLLM changes are merged.
- RedHatAI ships `quant_method=compressed-tensors` / `mxfp4-pack-quantized`
  (`CompressedTensorsW4A4Mxfp4`), a *different* ingestion path from the lab's
  `quant_method=mxfp4` — hence the fallback order: `flashinfer_cutlass_afp8`
  → `b12x` (proven on our DS4.1 MXFP4 experts) → drop the flag and read the
  engine's selected backend from the log.
- **Numerics gate either way**: a missing SwiGLU clamp (r33/r34 lesson) and
  W4A4 token-loop corruption both present as degenerate repetition. Run the
  P0.2 smoke with ~300 tokens and look at it, before any bench.

### P4 — capability lanes worth a logical slot (from the lab docs)

**(a) LMCache RAM + filesystem tiers → long context on a tiny GPU KV pool.**
Their TP2 GLM-5.3-Flash contract is `MAX_MODEL_LEN=1048576` with only
`KV_CACHE_MEMORY_BYTES=4294967296` (4 GiB/rank) and a 1,048,320-token cold
text run in 139.7 s, RAM restore 0.808 s, filesystem restore 1.279 s, zero
recompute. Options (before the image name):
`-e CACHE_MODE=lmcache -e LMCACHE_L1_SIZE_GB=64 -e LMCACHE_L2_ROOT=/cache/lmcache-l2`.
Caveats: the 64 GiB L1 arena is **preallocated pinned host shm**, so host
`/dev/shm` must be big enough (Docker `--shm-size` does NOT resize host shm);
`LMCACHE_L2_ENABLED=0` = RAM-only; the `/cache` volume persists FS objects
across container replacement; objects cover 4,096 global tokens, attention
pages 2,048 tokens per DCP rank; checkpoint files are hashed at startup (first
log delayed while weights are read — keep them immutable while serving).
Why it matters for us: it decouples "≥512K context" from "GPU KV pool size" —
exactly the wall the full MXFP4 model hits on 4×. Worth a lane on the r35 GLM
Flash profile (cheap: env only, same image).

**(b) GLM vision on 2 GPUs.** The lab's TP2/DCP2 GLM-5.3-Flash contract
includes vision: one image per API request (across all turns), native
~8,000-feature ceiling, video disabled, full-image FlashAttention for the
image pass, target KV FP8 + FP32 recurrent state, MTP3 probabilistic. Cold
1,048,320-token image input: correct answer in 143.0 s with **min free VRAM
232.31 MiB/GPU**. All our GLM lanes are text-only (`--language-model-only`) —
this is a capability upgrade (screenshots/diagrams in agent traffic), and it
is a documented, qualified contract rather than guesswork.

**(c) Two conventions to copy into our recipes.**
- Explicit `MAX_MODEL_LEN` + explicit `KV_CACHE_MEMORY_BYTES` (they never let
  a lane silently shorten context; `KV_CACHE_MEMORY_BYTES=auto` is for
  auto-profiling and `MAX_MODEL_LEN=-1` is the opt-in for accepting a shorter
  context; GMU does not resize an explicit KV allocation). We already pin both
  — keep it that way, and quote the KV line in every run report.
- `CUBLAS_WORKSPACE_CONFIG=:4096:1` bounds the cuBLAS workspace to 4 MiB per
  handle → 140 MiB less live allocation per rank (they measured it; not a
  confirmed cure for every community cuBLAS crash). Cheap, consider for the
  full-model lane where every 100 MiB counts.

**Important caveat from their doc:** "The memory mechanisms can also apply to
TP4, but their magnitude depends on sharding and workload. **No TP4 memory or
performance improvement is qualified by the TP2 measurements.** The TP2 NCCL
configuration is not a TP4 recommendation." (They use 2 NCCL channels / 1 MiB
buffer on TP2; we run 16 channels / 2 MiB on TP4 — do not copy the TP2 values.)

**Reference numbers from that doc** (2× GB10, TP2/DCP2, MTP3, 4 GiB KV/rank,
1M ctx): 32K prefill 10,648.9 tok/s; C1 MTP3 median output 170.77 tok/s;
verifier 67.5 steps/s; R2 vs R1 = +0.41% prefill / +4.52% C1.

---

## Progress 2026-09-13 ~22:20 UTC — both lanes are READY

**P1 DONE — R37 image built and distributed.**
`jovian-r37-ds41:local`, checksum label
`9350429b719b85df3009ad4c8ac86d58f16bd0ede942b7ad67a0f4baa2d1aeba`,
built on dgx5 in ~37 min (the deps layer was cold this time; the r36 build was
13m47s warm), transferred to all four nodes in ~4 min (4 parallel
`docker save | ssh | docker load`), verified on dgx1:
- vLLM `0.26.1rc0+glm53.r37.vllmc687594b` (the r37 composed pin)
- b12x 1.3.0, `liburing.so.2 → 2.5`
- `GlmMoeDsaForCausalLM` registered; `vllm/models/deepseek_v32/nvidia/` ships
  `model.py`, `mtp.py`, `b12x.py`, `glm52_low_latency_gemm.py` — i.e. the
  full-model arch, its MTP head AND a b12x path are all in the image.

**P0 prerequisite DONE — the checkpoint is complete on the NAS.**
`models--RedHatAI--GLM-5.3-MXFP4`: **141/141 shards, 0 `.incomplete`,
376 GiB** (= 403.4 GB decimal ✓ matches the HF index exactly).

**Switch is the only thing left, and it needs a human decision.** `sm121
health --cluster 4nodes-switch` → all four nodes online, **0 free**, the eugr
DS4.1 prod occupying them (`sm121_dsv41-flash-4x-b12x_0..3`, up 9 h). Both
night lanes (P0 and P2) need the whole quartet, and stopping the prod
interrupts whatever it is serving — possibly this agent's own model. So:
confirm, then

```
sm121 stop --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch   # ~10 s
# wait ~12 s
sm121 serve --keep --recipe dsv41-flash-jovian-r37-4x --cluster 4nodes-switch   # P2
# or the P0 ladder on glm53-full-mxfp4-4x
```

Suggested order now that both are ready: **P2 (R37) first** — it is a
known-good configuration class (a boot is low-risk, ~50 min) and answers the
+43% prefill claim on GB10; then **P0 (MXFP4)** with the whole remaining
night, whose probe boot decides the capability question. Teardown/restore is
P0.5.

---

## P0 in flight (interim, ~23:55 UTC) — full MXFP4 on 4x: PROBE PASSED, DCP4 ladder

Boot ladder for `glm53-full-mxfp4-4x-probe` (no DCP, 131K, r35 img):
1. `--moe-backend flashinfer_cutlass_afp8` → argparse rejects it (not a CLI
   choice on r35; the CLI family is `flashinfer_cutlass`, the `_afp8` key is
   an oracle-level thing).
2. `--moe-backend flashinfer_cutlass` → accepted, but the flag is INERT for
   this checkpoint: the compressed-tensors W4A4-MXFP4 MoE scheme picks
   **MarlinExperts** on its own. The boot then died in the mxfp4 LINEAR
   kernel: the default priority [FlashInfer, Marlin, Humming, B12x] starts
   with the FlashInfer kernel, whose `is_supported()` wrongly accepts
   capability 121 (numeric `has_device_capability(100)` passes on GB10)
   while the cute-dsl mm_fp4 GEMM rejects it at runtime:
   `mm_fp4 does not support backend 'cute-dsl' with capability 121`
   (kv_b_proj dequant, mla_attention.py).
3. `--linear-backend b12x` → correct selection ("Using
   B12xMxFp4LinearKernel for MXFP4 GEMM"; verified per-component: family(120)
   True, b12x blockscaled OK, cute-dsl input-quantize works on 121) — but the
   UNCONDITIONAL b12x warmup crashes on this checkpoint's shapes:
   "Expected source and destination tensors to have the same rank, but got 3
   and 4" (b12x dense_gemm via synthetic zero source). Filed-worthy for the lab.
4. **`--linear-backend marlin` + 8 GiB KV → BOOTED.** (A 4 GiB pin was
   rejected: the engine needs 6.58 GiB/rank for ONE 131072-token request →
   the full model's KV = **51.6 KB/token/rank**, fp8_ds_mla.)
   Pool: **159,232 tokens per 8 GiB pin** (52.7 KB/token), max concurrency
   1.21x at 131K. Smoke: 27×43=1161 ✓, tool call exact in 4.2 s ✓,
   35,157-token needle answered "73451" in 5.1 s (≈7.8K tok/s prefill) ✓,
   no degenerate repetition. **The 740B model works on 4× GB10.**
5. DCP4 first boot: `DCP sparse indexer currently supports only
   cp_kv_cache_interleave_size=1 (got 4)` — the x4 interleave is a Flash
   dense-MLA rule, not the DSA top-k path. Recipe fixed to interleave 1,
   rebooting.

DCP4 512K pool estimate: 8 GiB/rank ÷ (52.7 KB/token ÷ 4) ≈ 650K tokens
≥ 524288 ✓. Then smoke + dev65 bench `dev65-full-mxfp4-dcp4`, then P2.

---

## Night shift results (final, 2026-09-14 ~02:45 UTC)

### P0 — full GLM-5.3 (RedHatAI MXFP4, ~740B) on 4× GB10: WORKS (131K lane)

Booted, smoke-passed, benched (label `dev65-full-mxfp4-131k`, report in
`recipes/glm53-full-mxfp4-4x-probe/reports/`):
- **Quality 158/176 (90**, their 100-scale "91 Excellent", 1 safety warning
  TC-51, 87/88 scored).
- **Perf pp2048/tg128**: d0 c1 **934 pp / 19.0 tg / 2,124 ms**, c2 867/26.5/
  3,185; d4096 c1 896/22.6/6,285; d8192 c1 877/22.8/10,406.
- KV: **52.7 KB/token/rank** (fp8_ds_mla), pool 159,232 tokens per 8 GiB pin,
  max concurrency 1.21× at 131K. Load ~10 min at ~600 MB/s from NAS.
- MoE = MarlinExperts (scheme's own pick; `--moe-backend` is inert for the
  compressed-tensors path), linear = Marlin via `--linear-backend marlin`
  (the b12x warmup crashes on this checkpoint's shapes; the flashinfer kernel
  wrongly claims sm_12x support — both documented in the recipe).
- **512K is BLOCKED on 4×**: DCP4 is the only way 512K fits (26.3 GiB
  aggregate KV), and the DCP>1 sparse-decode path crashes identically on BOTH
  r35 and r37: after "DCP sparse indexer currently supports only
  cp_kv_cache_interleave_size=1 (got 4)" → interleave=1 → graph-capture
  dummy run dies in deepseek_v32/attention.py:596 with
  "shape '[16, 16, 512]' is invalid for input of size 524288" (the ×4 DCP
  factor; the dcp_manager.combine path is gated on use_pcp, which we do not
  set). Upstream bug to file with the lab. The 512K route for this model is
  the LMCache RAM/FS tiers (P4a below) or the DCP fix.

### P2 — R37 DS4.1 on 4×: prefill champion, quality regressed

(Report `recipes/dsv41-flash-jovian-r37-4x/reports/`, label
`dev65-r37-ds41-4x`; image label 9350429b, built on dgx5, 37 min.)
- **Perf**: d0 c1 **3,830 pp / 46.4 tg / 832 ms** vs prod 1,642/47.0/1,738 →
  **prefill +133% (2.3×), TTFT 2.1× better, decode flat** (the dispatch
  removal is a prefill win; the lab measured it on RTX, GB10 matches and
  beats). c2: 2,688 pp / 71.8 tg.
- **Quality 150/176 (85), safety-capped** (4 warnings: TC-51/74/82 + one more
  — the same long-horizon cluster as the r36 K7 run).
- **Verdict: eugr DS4.1 (nightly-20260910) stays prod of record for agent
  quality; R37 is the prefill/batch champion** (long-context, low-interactivity
  workloads, or until the quality regression is fixed).

### P3 skipped (eugr nightly-20260913 A/B)
The marginal nightly diff is not worth pulling an unverified external image in
the last hours before teardown; R37 already answers the "newer tree" question
decisively.

### Teardown
- dgx5 powered off ~22:33 UTC (operator authorized, at shift start).
- 4nodes-switch: R37 lane stopped, then `sm121 power off --cluster
  4nodes-switch` (operator's instruction for end of shift). No prod restore —
  the operator re-serves manually in the morning.

### Morning recommendations (priority order)
1. **File the DCP>1 sparse-decode bug** (repro: full GLM-5.3 MXFP4, DCP4,
   interleave 1, both r35 and r37 images; exact messages above) — this is the
   only thing standing between the full model and 512K on 4×.
2. **512K via LMCache tiers** on the full model (the lab's own pattern:
   small GPU KV + 64 GiB L1 pinned shm + /cache FS; check host /dev/shm
   capacity first — Docker --shm-size does not resize host shm).
3. **R37 as the prefill lane** for batch/long-context; keep eugr for agent
   traffic. A/B an eugr nightly that inherits the r37 dispatch work if/when
   the lab merges it.
4. The b12x-mxfp4-linear warmup rank bug + the flashinfer is_supported
   sm_12x false-positive (both workarounds are in the recipe; file both).
5. Push all local commits when git.kilork.org recovers.

---

## Morning shift (2026-09-14, ~05:50-07:00 UTC)

### Operator request: bring prod up; R37 makes suspicious typos
"как будто оно не попадает в лексику" — check that nothing was exported/quantized/
serialized wrong.

### Finding 1: R37 tree runtime-quantizes the LM head by default (NOT the cause, but real)
- `envs.py`: `VLLM_MXFP8_LM_HEAD` default **"1"** — "Quantize eligible unquantized
  LM heads on b12x by default; =0 opts out." Implementation:
  `model_executor/layers/vocab_parallel_embedding.py` (sets
  `runtime_lm_head_quantization = "mxfp8"` for unquantized, untied LM heads).
- The eugr/tonyd2wild prod image (nightly-20260910 base) has NO such feature —
  prod runs a BF16 head. Added `VLLM_MXFP8_LM_HEAD = "0"` to the R37 recipe
  (commit 902e989). Boot without the "Quantizing LM head shards to MXFP8" line
  = opt-out confirmed; KV pool shrank 10.39M -> 10.19M tokens (head now BF16 =
  more memory) — indirect confirmation.
- **But the typos persist** with the BF16 head (verbatim/greedy probes below).

### Typo evidence (R37 lane, temp=0 greedy, ~06:20 UTC)
Long 300-word Russian generation: "данних. Етот цыфровой оазис среди вечной
мерзлоти... навикации... иследований... критically важной точкой опори...
видлели колосальное количество енергии" (10+ mangled words in ~150 words).
Code sample: "возврощает", "валідный" (Ukrainian і), "должин быть словарём".
Identifiers check: all 5 near-identical config keys emitted correctly (code
structure intact; it's the vocab surface layer that wobbles).

### Bisect (in progress at 06:30)
Flag diff vs every clean lane: R37 contract uniquely adds `--linear-backend
b12x` (prod: b12x MoE only, default linear; GLM r35 clean lane: B12X attention
+ instanttensor, default linear). Boot A = r37 image WITHOUT `--linear-backend
b12x`. If typos persist -> Boot B: prod's spec config (K5 probabilistic/block,
adaptive off) instead of K7 greedy/standard/adaptive-on; Boot C: full prod arg
set on the r37 image (tree vs contract).

### Naming correction (operator question)
Our prod 4x image is NOT eugr: `dsv41-flash-b12x:local` = thin wrapper over
`dsv41-flash:local` = digest-pinned `vllm/vllm-openai` arm64 nightly +
tonyd2wild overlay chain (dsv41-feat@e47aa780 + 7 boot-10 patches + FlashInfer
0.7.0rc1, all built in this repo). "eugr" was my sloppy label inherited from
the `deepseek-v4-flash-eugr*`/`glm53-flash-*eugr*` lane naming; from now on:
"DS4.1 prod" for the tonyd2wild lane, "eugr" only for the eugr-image lanes.

### Boot D interrupted: cluster-wide outage (~07:27 UTC)
During Boot D (r37 + prod contract, engram off), all 4 nodes went unreachable
mid-load (~3 min after container start): "Host is down" / "No route to host"
from the live gateway (192.168.88.1 up), HA (192.168.88.232) alive but
returning 401 to the known token. Earlier in the window one ping to dgx1 got
through (1/2, 2 ms) - the nodes died abruptly, not degraded. All 4 boxes at
once, in the middle of a 4x parallel NFS load. Suspects: PDU/segment power
trip, or a physical event; nothing sm121 did can take a node's management NIC
down (the serve had already passed container start and was at the API-wait
stage). Polling for recovery; Boot D test is the only remaining bisect step.
HA check (~07:38): switch.dgx1-4 = ON, sw6_mt/sw7_fan ON (dgx5-8 off as
expected). So the nodes are powered but the 88.x management path is dead -
either the nodes hung/crashed under load (powered but kernel dead) or the
management segment/switch failed. Cannot distinguish from the Mac (no route
to 200.x). Physical check needed: power-cycle the nodes or check the
management switch.

## GLM-5.3 eugr 2x re-bench (2026-09-14 ~09:20 UTC, dev66-glm53-eugr-2x-0910)

Operator redeployed `glm53-flash-eugr-2x` on dgx1+dgx2 (2nodes-dac) after the
outage (he powered on only 1+2; dgx3/4/5 stay off by choice). Same image as
the 09-11 run: eugr/spark-vllm-b12x:nightly-20260910, model
local-inference-lab/GLM-5.3-Flash-NVFP4-Spark, 1M ctx, KV pool 1,162,277
tokens.

Result: **161/176 (91, ★★★★★)**, 1 safety warning (TC-51, same as the other
lanes). Perf d0 c1: **1,326 pp / 28.1 tg / 1,458 ms TTFT** (c2: 1,308/33.9/
2,157); d4096 c1 1,663/30.5/3,369; d8192 c1 1,762/31.8/5,297.

vs anchors: eugr 2x 09-11 166/176 (same stack, -5 = variance); r28 2x (our
image) 158/176, 1,475 pp / 27.2 tg / 1,362 ms — eugr wins quality, r28 wins
prefill by ~11%, decode/TTFT near-tie. Verdict unchanged: eugr 2x = quality
lane, r28 = prefill lane (2x).

Also this morning: report/web "running" status was stale for the r37 lane —
the run-ledger (.lock) `running` record from the killed Boot D serve was never
closed (sm121 stop can't verify with dgx3/4 offline). Closed the record per
mark_stop semantics + docker rm'd the exited r37 containers on dgx1/2.
Systemic options on the table: (1) report cross-checks ledger against the
live probe, (2) `sm121 runlock close <recipe>` offline command.

## Report stale-status + sm121 runlock (2026-09-14, ~11:40 UTC)

Operator asked for both fixes after the r37 false-"up":
1. `sm121 report --serve` now cross-checks the run ledger against a live
   fleet probe every refresh tick (30 s, only when an open run exists): an
   open run whose recipe has no running container on any reachable node is
   rendered orange "stale" with a `sm121 runlock close <recipe>` tooltip
   instead of green "up". Inconclusive (unknown cluster, unreachable node)
   stays green. tmux deploys are never probed. `src/report/stale.rs`
   (mark_stale + 4 unit tests), serve.rs refresh hook, page.html state cell.
2. `sm121 runlock list|close <recipe>` — offline ledger management (the
   `mark_stop` file op without node verification). RunlockCommands lives in
   runlock.rs (cli.rs at the 500-line cap).
All gates green (310 tests). Live-verified: eugr 2x shows up (probe sees the
containers), r37 shows stopped.