Night shift 2026-09-10 (CEST) — r31 + eugr campaign

Window: ~01:45–08:00 local. Operator approval: "Начинаем через 1.5 часа. Остановишь прод и вперед." (00:20 local, 2026-09-10).

Context

Production at window start: glm53-flash-r28-4x-dcp4 on 4nodes-switch (r28 stack: vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a; 155/159 hardmode, ~46.8 tok/s decode, RoCEnante on).
Update scan (2026-09-09/10) found a full new release line R28.1 → R29 → R30 → R31 (qualified) in local-inference-lab/rtx6kpro source locks (models/glm-5.3-flash/validation/*):
- R31 = vLLM 55769270 (integration/jovian-warmup-buffer-reuse-20260909)
- b12x 3edbcbce (frozen since 09-08) + LMCache f868580a.
- Carries the GLM Flash MTP projection loader fix (our upstream eh_proj.weight O_DIRECT issue), FlashInfer GDN prefill on SM12x, b12x TP2 graph peer-push fix, scheduler hardening (fixed prefill compute share 0.4), engine-driven async SHM LMCache + atomic recurrent checkpoints, 16-channel NCCL.
- Published image for cross-check: localinferencelab/vllm:jovian-judgement-community-20260909-r31.
eugr lane (operator: "eugr — это тоже b12x вариант, тоже интересно для glm53"): eugr/spark-vllm-docker (MIT) ships a cluster_only dual-Spark GLM-5.3-Flash recipe on their experimental b12x stack (--load-format b12x loader, MTP k=5 "humming", 1M ctx, 8 GiB KV, GMU 0.87). Prebuilt image eugr/spark-vllm-b12x:nightly-20260909 (Docker Hub, ~11 GB). Their vLLM ref = local-inference-lab/vllm @ dev/jovian-judgement (mainline line, 41ea64ae — diverged from the integration release cuts: +16/-39 vs r28).

docker-images/glm53-jovian-r27/profiles/glm53-r31.env — named version profile (clone-at-pin: vLLM 55769270/tree 6a2da27b, b12x 3edbcbce/tree c4bfeee9, LMCache f868580a/tree 6e5f080f). Build-dep gates verified against the r31 tree: deepgemm.cmake still 8b1392b9, cuda.txt still torch 2.13.0 / flashinfer 0.6.17 — the r27 image dir's gates hold.
recipes/glm53-flash-r31-4x-dcp4/ — fork of the r28 4x recipe (model GLM-5.3-Flash-NVFP4, 16 GiB KV pin); delta: --prefill-compute-share 0.4 + NCCL_NCHANNELS=16.
recipes/glm53-flash-r31-2x/ — fork of r28-spark 2x (model GLM-5.3-Flash-NVFP4-Spark, 10 GiB KV pin); same delta.
recipes/glm53-flash-eugr-2x/ — verbatim mirror of eugr's recipes/glm-5.3-flash.yaml on the prebuilt nightly-20260909 image (no build: no docker_image_dir).
eugr image pull started on dgx1+dgx2 pre-window.
LMCache engine-driven SHM deliberately NOT enabled in the first A/B: the r31-vs-r28 comparison must isolate the engine line; the LMCache connector (kv_transfer lmcache offload) is a separate axis for a later window. --recurrent-checkpoint-policy defaults to auto in r31 (no flag needed).

#	Step	Command / note
1	Stop prod	`sm121 stop --recipe glm53-flash-r28-4x-dcp4` + 15 s settle + verify no containers on 4 nodes
2	Build r31	`sm121 build --recipe glm53-flash-r31-4x-dcp4 --build-node dgx1.lan` (~1 h, clone-at-pin; image syncs to the other nodes)
3	r31 4x A/B	`sm121 serve --recipe glm53-flash-r31-4x-dcp4` → API gate 88.14:30000 → `sm121 bench --recipe glm53-flash-r31-4x-dcp4 --cluster 4nodes-switch --label r31-4x`
4	r31 2x A/B	stop → `sm121 serve --recipe glm53-flash-r31-2x` → bench label `r31-2x` on 2nodes-dac
5	eugr 2x	stop → `sm121 serve --recipe glm53-flash-eugr-2x` → bench label `eugr-2x` on 2nodes-dac
6	Verdict + restore	best 4x-candidate to prod (r31 vs r28); `sm121 serve` + API verify
7	Report	this file + glm53-recipe-state.md score table + commit/push

Baselines (seed 42, hardmode, tool-eval-bench 2.6.1.dev46, 88 scenarios):

Score: 159/176 (90/100), all 88 scenarios completed, 0 errors — recipes/glm53-flash-r31-4x-dcp4/reports/2026-09-10_063616_r31-4x.*
Perf (d0 c1): 1,484 pp / 51.6 tg / 1,785 ms TTFT — decode +10% over the r28 4x baseline (~46.8 tg), prefill at parity (1,484 vs ~1,500-1,600). d8192 c1: 50.4 tg.
Scenario A/B vs r28 4x (2026-09-08 full run, same seed 42):
- fixed in r31: TC-40, TC-47, TC-49, TC-51, TC-88 (5)
- regressed: TC-33 (safety: hallucination resistance), TC-50, TC-68, TC-80, TC-82 (5)
- common fails: 8 (TC-43/45/46/57/61/63/67/85) — quality is a wash
Verdict: r31 4x ≥ r28 4x → r31 4x promoted to production (09:50 UTC, window close requested by operator). Rollback to r28 4x = one stop+serve if the operator prefers the old image.

Score: 154/176 (88/100) — recipes/glm53-flash-r31-2x/reports/2026-09-10_070856_r31-2x.*
Perf (d0 c1): 1,544 pp / 25.9 tg / 1,289 ms TTFT (vs r28 2x baseline 1,475 pp / 27.2 tg / 1,362 ms — decode -5%, prefill +5%, TTFT -6%).
17 non-pass scenarios (incl. 4 fails: TC-43/45/51/80/81) — 4 points below the r28 2x baseline (158/176). The r31 delta (fixed prefill compute share 0.4 + 16 NCCL channels) does not help the 2x lane; the 2x lane stays on r28 (glm53-flash-r28-spark) unless the 2x recipe is re-tuned (drop the r31 delta flags and re-bench).

Not benched — the window closed (operator: "пора останавливать, поднять прод"). The recipe (glm53-flash-eugr-2x) and the image (eugr/spark-vllm-b12x:nightly-20260909, pulled on dgx1+dgx2) are ready; next window: serve + bench label eugr-2x on 2nodes-dac.

Build: sm121 build on dgx1 failed once on the r31 FlashKDA API gate ("FlashKDA sources lack the packed checkpoint API") — fixed by making FLASH_KDA_SRC_DIR a Dockerfile ARG (default /deps2/FlashKDA, empty in the r31 profile) so the r31 tree uses its own pinned 3b225bf2 + in-tree flashkda-packed-checkpoints.patch fallback. Rebuild: ~40 min (ccache warm), image glm53-jovian-r31:local stamped d05629731f0539ccf176b70d30bfebef59738200b0a4a9d39762adf2ca745c97.
The eugr lane and the sparkring new-image re-run (4774a7c5, guarded 2 GiB memory floor) carry over to the next window.