# Night shift 2026-09-11 (local) — DSV4.1 first deployment + eugr lane refresh Window opened by operator at ~21:50 local ("прод до утра не нужен"). Priority: DeepSeek-V4.1-Flash campaign. ## Production - Stopped `glm53-flash-r31-4x-dcp4` (21:50 local). - **End state: `dsv41-flash-4x` (4nodes-switch) is the live 4x config** — operator to decide in the morning: keep DSV4.1 up or switch back to r31 (one stop+serve). ## DSV4.1-Flash (the campaign) ### Image - Built on **dgx5** (powered on via HA, buildx docker-container builder): `dsv41-flash:local` 23.8GB, checksum label `91b2523a46afae80279d7ef8ddb2e7308192f70097e4eca32b8d714a98aa199d` (build log: `/tmp/dsv41-flash-4x-build.log` on dgx5). - Build fixes (commit dd86b8a): cutlass extraction into `build/_deps/cutlass-src`, cmake/ninja/g++ install, NVRTC resolution via `-DCUDA_nvrtc_LIBRARY`, stable-ext validation switched to file-existence (no GPU driver in the build container). - Transferred to dgx1–dgx4 via `docker save | ssh docker load` (single-transfer pattern; the first 4-parallel batch failed on nested host-key race, single transfers are reliable). - **Boot-1 bug**: the base `vllm/vllm-openai` image bakes `ENTRYPOINT ["vllm","serve"]`; our recipe's `entrypoint = ["vllm","serve"]` doubled the command (`vllm serve vllm serve ` → instant exit). Fix: **no `entrypoint` in the recipe** — the launcher's CMD (model + flags) completes the baked entrypoint. No image rebuild needed. ### Engram node-local copies - First boot (ENGRAM_DISK, NFS) printed the per-rank row ranges: - layer 1: r0 [0, 96000564) / r1 [96000564, 192001740) / r2 [192001740, 288003654) / r3 [288003654, 384006168) - layer 14: r0 [0, 96003054) / r1 [96003054, 192007016) / r2 [192007016, 288011564) / r3 [288011564, 384016682) - `tools/engram_local.py` (theirs, sparse same-offset, 600 MB/s cap, page-cache drops, byte-for-byte verify) run per node against the corresponding rank's ranges → `/var/tmp/engram-local/DeepSeek-V4.1-Flash` on all 4 nodes: **47.2 GiB each, verify 0 mismatches**. - Recipe: `DSV41_ENGRAM_LOCAL=1`, `DSV41_ENGRAM_DIR=/engram-local`, ro bind mount `/var/tmp/engram-local/DeepSeek-V4.1-Flash → /engram-local`. Boot log confirms: `Engram layer N rows [...] read from node-local /engram-local`. ### Boot notes - Weight load: 475 GiB NFS4, 48 shards fast pass ~5.5 min, then a **second per-expert pass over all 48 shards** (W13/W2 per-expert copies, ~200 MB/s from NFS, ~30 min). Total boot ≈ 45 min (boot 1), faster for boot 2 with the local Engram + warm NFS page cache. - **CUDA graphs worked from the first boot** (prestage patch + exact capture sizes — no eager fallback needed). - KV pool: 7.34 GiB (their boot-10: 7.89 GiB), 300K ctx, DSpark k=5. ### Smoke (boot 1) — all pass - chat: "2+2" → "Четыре" (0 reasoning tokens, thinking off by default); - tools: math tool call `add(a=0, b=2)` correct; - vision: 64×64 red png → "Красного". ### Bench (boot 2, local Engram) — `2026-09-10_231741_dsv41-local-0910` **156/176 (89/100), ★★★★ "Good", 88/88 scenarios completed, 0 errors/crashes, 0 safety warnings.** 70 pass / 16 partial / 2 fail (TC-61, TC-88). Categories (earned/max): A 5/6, B 6/6, C 6/8, D 6/6, E 5/6, F 6/6, G 6/6, H 10/10, I 14/20 (70%), J 6/6, K 24/26, L 8/8, **M 3/6 (50% — all partial)**, N 6/6, **O 12/12 (structured output 100%)**, P 33/38. Weak spots vs our GLM: autonomous planning (M 50%), context & state (I 70%). Strong: structured output, toolset scale, localization (RU), code patterns — all 100%. Perf (pp t/s / tg t/s / TTFT ms / total ms, pp2048 tg128): | depth | c1 | c2 | |-------|-------------------------------|-------------------------------| | d0 | 1,090 / 36.1 / 1,876 / 5,295 | 1,052 / 36.5 / 3,086 / 8,858 | | d4096 | 1,124 / 32.2 / 5,418 / 9,272 | 1,172 / 26.0 / 8,278 / 15,723 | | d8192 | 1,411 / 31.2 / 7,421 / 11,398 | 1,371 / 20.9 / 10,973 / 19,633 | (vs GLM r31 4x d0 c1: 1,484 pp / 51.6 tg / 1,785 ms — DSV4.1 decode is ~30% slower, TTFT on par. 16B-active MoE vs GLM-5.3-Flash. Their own claims: prefill 902–1,539 t/s — our 1,052–1,411 matches; their C6 agg 131.9 t/s is a concurrency-6 aggregate, not directly comparable.) Verdict: quality within 3 points of our GLM prod (159/176), reliability clean (no death under the full 88-scenario load — unlike sparkring), structured output 100%. Slower decode/TTFT. A serious candidate for a second production lane. ## eugr lane refresh - New image `eugr/spark-vllm-b12x:nightly-20260910` (published today 13:06 UTC) pulled to dgx1 + dgx2. Recipe `glm53-flash-eugr-2x` needs its `docker_image` bumped next window; bench vs r28 2x (158/176) still queued. ## Carried - dgx5 left powered on (used for the DSV4.1 build). - sparkring 2x re-run (memory-guard image), 386B infra decision, gpuflip probe — all still queued for a window.