# Night shift 2026-09-10 (CEST) — r31 + eugr campaign

Window: ~01:45–08:00 local. Operator approval: "Начинаем через 1.5 часа.
Остановишь прод и вперед." (00:20 local, 2026-09-10).

## Context

- Production at window start: `glm53-flash-r28-4x-dcp4` on 4nodes-switch
  (r28 stack: vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a; 155/159
  hardmode, ~46.8 tok/s decode, RoCEnante on).
- Update scan (2026-09-09/10) found a full new release line R28.1 → R29 →
  R30 → **R31 (qualified)** in `local-inference-lab/rtx6kpro` source locks
  (models/glm-5.3-flash/validation/*):
  - R31 = vLLM `55769270` (integration/jovian-warmup-buffer-reuse-20260909)
    + b12x `3edbcbce` (frozen since 09-08) + LMCache `f868580a`.
  - Carries the **GLM Flash MTP projection loader fix** (our upstream
    eh_proj.weight O_DIRECT issue), FlashInfer GDN prefill on SM12x,
    b12x TP2 graph peer-push fix, scheduler hardening (fixed prefill
    compute share 0.4), engine-driven async SHM LMCache + atomic
    recurrent checkpoints, 16-channel NCCL.
  - Published image for cross-check:
    `localinferencelab/vllm:jovian-judgement-community-20260909-r31`.
- **eugr lane** (operator: "eugr — это тоже b12x вариант, тоже интересно
  для glm53"): `eugr/spark-vllm-docker` (MIT) ships a cluster_only
  dual-Spark GLM-5.3-Flash recipe on their experimental b12x stack
  (`--load-format b12x` loader, MTP k=5 "humming", 1M ctx, 8 GiB KV,
  GMU 0.87). Prebuilt image `eugr/spark-vllm-b12x:nightly-20260909`
  (Docker Hub, ~11 GB). Their vLLM ref = `local-inference-lab/vllm` @
  `dev/jovian-judgement` (mainline line, 41ea64ae — diverged from the
  integration release cuts: +16/-39 vs r28).

## Artifacts (prepared before the window, no prod impact)

- `docker-images/glm53-jovian-r27/profiles/glm53-r31.env` — named version
  profile (clone-at-pin: vLLM 55769270/tree 6a2da27b, b12x 3edbcbce/tree
  c4bfeee9, LMCache f868580a/tree 6e5f080f). Build-dep gates verified
  against the r31 tree: deepgemm.cmake still 8b1392b9, cuda.txt still
  torch 2.13.0 / flashinfer 0.6.17 — the r27 image dir's gates hold.
- `recipes/glm53-flash-r31-4x-dcp4/` — fork of the r28 4x recipe (model
  GLM-5.3-Flash-NVFP4, 16 GiB KV pin); delta: `--prefill-compute-share
  0.4` + `NCCL_NCHANNELS=16`.
- `recipes/glm53-flash-r31-2x/` — fork of r28-spark 2x (model
  GLM-5.3-Flash-NVFP4-Spark, 10 GiB KV pin); same delta.
- `recipes/glm53-flash-eugr-2x/` — verbatim mirror of eugr's
  `recipes/glm-5.3-flash.yaml` on the prebuilt nightly-20260909 image
  (no build: no docker_image_dir).
- eugr image pull started on dgx1+dgx2 pre-window.
- LMCache engine-driven SHM deliberately NOT enabled in the first A/B:
  the r31-vs-r28 comparison must isolate the engine line; the LMCache
  connector (kv_transfer lmcache offload) is a separate axis for a later
  window. `--recurrent-checkpoint-policy` defaults to `auto` in r31 (no
  flag needed).

## Plan

| # | Step | Command / note |
|---|------|----------------|
| 1 | Stop prod | `sm121 stop --recipe glm53-flash-r28-4x-dcp4` + 15 s settle + verify no containers on 4 nodes |
| 2 | Build r31 | `sm121 build --recipe glm53-flash-r31-4x-dcp4 --build-node dgx1.lan` (~1 h, clone-at-pin; image syncs to the other nodes) |
| 3 | r31 4x A/B | `sm121 serve --recipe glm53-flash-r31-4x-dcp4` → API gate 88.14:30000 → `sm121 bench --recipe glm53-flash-r31-4x-dcp4 --cluster 4nodes-switch --label r31-4x` |
| 4 | r31 2x A/B | stop → `sm121 serve --recipe glm53-flash-r31-2x` → bench label `r31-2x` on 2nodes-dac |
| 5 | eugr 2x | stop → `sm121 serve --recipe glm53-flash-eugr-2x` → bench label `eugr-2x` on 2nodes-dac |
| 6 | Verdict + restore | best 4x-candidate to prod (r31 vs r28); `sm121 serve` + API verify |
| 7 | Report | this file + glm53-recipe-state.md score table + commit/push |

Baselines (seed 42, hardmode, tool-eval-bench 2.6.1.dev46, 88 scenarios):

| Lane | Score | Perf (pp/tg/TTFT, d0 c1) |
|------|-------|---------------------------|
| r28 4x DCP4 (PROD) | 155/159 (90) | ~1,500-1,600 / ~46.8 / — |
| r28 2x DAC (spark) | 158/176 (90) | 1,475 / 27.2 / 1,362 |
| eugr 2x (their yaml) | — | — |

## Results

### r31 4x DCP4 (4nodes-switch, model GLM-5.3-Flash-NVFP4, 16 GiB KV)

- **Score: 159/176 (90/100), all 88 scenarios completed, 0 errors** —
  `recipes/glm53-flash-r31-4x-dcp4/reports/2026-09-10_063616_r31-4x.*`
- Perf (d0 c1): **1,484 pp / 51.6 tg / 1,785 ms TTFT** — decode **+10%**
  over the r28 4x baseline (~46.8 tg), prefill at parity (1,484 vs
  ~1,500-1,600). d8192 c1: 50.4 tg.
- Scenario A/B vs r28 4x (2026-09-08 full run, same seed 42):
  - fixed in r31: TC-40, TC-47, TC-49, TC-51, TC-88 (5)
  - regressed: TC-33 (safety: hallucination resistance), TC-50, TC-68,
    TC-80, TC-82 (5)
  - common fails: 8 (TC-43/45/46/57/61/63/67/85) — quality is a wash
- Verdict: r31 4x ≥ r28 4x → **r31 4x promoted to production** (09:50 UTC,
  window close requested by operator). Rollback to r28 4x = one
  stop+serve if the operator prefers the old image.

### r31 2x DAC (2nodes-dac, model GLM-5.3-Flash-NVFP4-Spark, 10 GiB KV)

- **Score: 154/176 (88/100)** — `recipes/glm53-flash-r31-2x/reports/2026-09-10_070856_r31-2x.*`
- Perf (d0 c1): 1,544 pp / 25.9 tg / 1,289 ms TTFT (vs r28 2x baseline
  1,475 pp / 27.2 tg / 1,362 ms — decode -5%, prefill +5%, TTFT -6%).
- 17 non-pass scenarios (incl. 4 fails: TC-43/45/51/80/81) — **4 points
  below the r28 2x baseline (158/176)**. The r31 delta (fixed prefill
  compute share 0.4 + 16 NCCL channels) does not help the 2x lane; the
  2x lane stays on r28 (`glm53-flash-r28-spark`) unless the 2x recipe is
  re-tuned (drop the r31 delta flags and re-bench).

### eugr 2x

- **Not benched** — the window closed (operator: "пора останавливать,
  поднять прод"). The recipe (`glm53-flash-eugr-2x`) and the image
  (`eugr/spark-vllm-b12x:nightly-20260909`, pulled on dgx1+dgx2) are ready;
  next window: serve + bench label `eugr-2x` on 2nodes-dac.

## Campaign notes

- Build: `sm121 build` on dgx1 failed once on the r31 FlashKDA API gate
  ("FlashKDA sources lack the packed checkpoint API") — fixed by making
  `FLASH_KDA_SRC_DIR` a Dockerfile ARG (default `/deps2/FlashKDA`, empty in
  the r31 profile) so the r31 tree uses its own pinned 3b225bf2 +
  in-tree `flashkda-packed-checkpoints.patch` fallback. Rebuild: ~40 min
  (ccache warm), image `glm53-jovian-r31:local` stamped
  `d05629731f0539ccf176b70d30bfebef59738200b0a4a9d39762adf2ca745c97`.
- The eugr lane and the sparkring new-image re-run (`4774a7c5`, guarded
  2 GiB memory floor) carry over to the next window.