Night shift 2026-09-10 (CEST) — r31 + eugr campaign

Window: ~01:45–08:00 local. Operator approval: "Начинаем через 1.5 часа. Остановишь прод и вперед." (00:20 local, 2026-09-10).

Context

Production at window start: glm53-flash-r28-4x-dcp4 on 4nodes-switch (r28 stack: vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a; 155/159 hardmode, ~46.8 tok/s decode, RoCEnante on).
Update scan (2026-09-09/10) found a full new release line R28.1 → R29 → R30 → R31 (qualified) in local-inference-lab/rtx6kpro source locks (models/glm-5.3-flash/validation/*):
- R31 = vLLM 55769270 (integration/jovian-warmup-buffer-reuse-20260909)
- b12x 3edbcbce (frozen since 09-08) + LMCache f868580a.
- Carries the GLM Flash MTP projection loader fix (our upstream eh_proj.weight O_DIRECT issue), FlashInfer GDN prefill on SM12x, b12x TP2 graph peer-push fix, scheduler hardening (fixed prefill compute share 0.4), engine-driven async SHM LMCache + atomic recurrent checkpoints, 16-channel NCCL.
- Published image for cross-check: localinferencelab/vllm:jovian-judgement-community-20260909-r31.
eugr lane (operator: "eugr — это тоже b12x вариант, тоже интересно для glm53"): eugr/spark-vllm-docker (MIT) ships a cluster_only dual-Spark GLM-5.3-Flash recipe on their experimental b12x stack (--load-format b12x loader, MTP k=5 "humming", 1M ctx, 8 GiB KV, GMU 0.87). Prebuilt image eugr/spark-vllm-b12x:nightly-20260909 (Docker Hub, ~11 GB). Their vLLM ref = local-inference-lab/vllm @ dev/jovian-judgement (mainline line, 41ea64ae — diverged from the integration release cuts: +16/-39 vs r28).

docker-images/glm53-jovian-r27/profiles/glm53-r31.env — named version profile (clone-at-pin: vLLM 55769270/tree 6a2da27b, b12x 3edbcbce/tree c4bfeee9, LMCache f868580a/tree 6e5f080f). Build-dep gates verified against the r31 tree: deepgemm.cmake still 8b1392b9, cuda.txt still torch 2.13.0 / flashinfer 0.6.17 — the r27 image dir's gates hold.
recipes/glm53-flash-r31-4x-dcp4/ — fork of the r28 4x recipe (model GLM-5.3-Flash-NVFP4, 16 GiB KV pin); delta: --prefill-compute-share 0.4 + NCCL_NCHANNELS=16.
recipes/glm53-flash-r31-2x/ — fork of r28-spark 2x (model GLM-5.3-Flash-NVFP4-Spark, 10 GiB KV pin); same delta.
recipes/glm53-flash-eugr-2x/ — verbatim mirror of eugr's recipes/glm-5.3-flash.yaml on the prebuilt nightly-20260909 image (no build: no docker_image_dir).
eugr image pull started on dgx1+dgx2 pre-window.
LMCache engine-driven SHM deliberately NOT enabled in the first A/B: the r31-vs-r28 comparison must isolate the engine line; the LMCache connector (kv_transfer lmcache offload) is a separate axis for a later window. --recurrent-checkpoint-policy defaults to auto in r31 (no flag needed).

#	Step	Command / note
1	Stop prod	`sm121 stop --recipe glm53-flash-r28-4x-dcp4` + 15 s settle + verify no containers on 4 nodes
2	Build r31	`sm121 build --recipe glm53-flash-r31-4x-dcp4 --build-node dgx1.lan` (~1 h, clone-at-pin; image syncs to the other nodes)
3	r31 4x A/B	`sm121 serve --recipe glm53-flash-r31-4x-dcp4` → API gate 88.14:30000 → `sm121 bench --recipe glm53-flash-r31-4x-dcp4 --cluster 4nodes-switch --label r31-4x`
4	r31 2x A/B	stop → `sm121 serve --recipe glm53-flash-r31-2x` → bench label `r31-2x` on 2nodes-dac
5	eugr 2x	stop → `sm121 serve --recipe glm53-flash-eugr-2x` → bench label `eugr-2x` on 2nodes-dac
6	Verdict + restore	best 4x-candidate to prod (r31 vs r28); `sm121 serve` + API verify
7	Report	this file + glm53-recipe-state.md score table + commit/push

Baselines (seed 42, hardmode, tool-eval-bench 2.6.1.dev46, 88 scenarios):

(pending — filled as the shift progresses)