Night shift 2026-09-11 (local) — DSV4.1 first deployment + eugr lane refresh

Window opened by operator at ~21:50 local ("прод до утра не нужен"). Priority: DeepSeek-V4.1-Flash campaign.

Production

Stopped glm53-flash-r31-4x-dcp4 (21:50 local).
End state: dsv41-flash-4x (4nodes-switch) is the live 4x config — operator to decide in the morning: keep DSV4.1 up or switch back to r31 (one stop+serve).

DSV4.1-Flash (the campaign)

Image

Built on dgx5 (powered on via HA, buildx docker-container builder): dsv41-flash:local 23.8GB, checksum label 91b2523a46afae80279d7ef8ddb2e7308192f70097e4eca32b8d714a98aa199d (build log: /tmp/dsv41-flash-4x-build.log on dgx5).
Build fixes (commit dd86b8a): cutlass extraction into build/_deps/cutlass-src, cmake/ninja/g++ install, NVRTC resolution via -DCUDA_nvrtc_LIBRARY, stable-ext validation switched to file-existence (no GPU driver in the build container).
Transferred to dgx1–dgx4 via docker save | ssh docker load (single-transfer pattern; the first 4-parallel batch failed on nested host-key race, single transfers are reliable).
Boot-1 bug: the base vllm/vllm-openai image bakes ENTRYPOINT ["vllm","serve"]; our recipe's entrypoint = ["vllm","serve"] doubled the command (vllm serve vllm serve <model> → instant exit). Fix: no entrypoint in the recipe — the launcher's CMD (model + flags) completes the baked entrypoint. No image rebuild needed.

Engram node-local copies

First boot (ENGRAM_DISK, NFS) printed the per-rank row ranges:
- layer 1: r0 [0, 96000564) / r1 [96000564, 192001740) / r2 [192001740, 288003654) / r3 [288003654, 384006168)
- layer 14: r0 [0, 96003054) / r1 [96003054, 192007016) / r2 [192007016, 288011564) / r3 [288011564, 384016682)
tools/engram_local.py (theirs, sparse same-offset, 600 MB/s cap, page-cache drops, byte-for-byte verify) run per node against the corresponding rank's ranges → /var/tmp/engram-local/DeepSeek-V4.1-Flash on all 4 nodes: 47.2 GiB each, verify 0 mismatches.
Recipe: DSV41_ENGRAM_LOCAL=1, DSV41_ENGRAM_DIR=/engram-local, ro bind mount /var/tmp/engram-local/DeepSeek-V4.1-Flash → /engram-local. Boot log confirms: Engram layer N rows [...] read from node-local /engram-local.

Boot notes

Weight load: 475 GiB NFS4, 48 shards fast pass ~5.5 min, then a second per-expert pass over all 48 shards (W13/W2 per-expert copies, ~200 MB/s from NFS, ~30 min). Total boot ≈ 45 min (boot 1), faster for boot 2 with the local Engram + warm NFS page cache.
CUDA graphs worked from the first boot (prestage patch + exact capture sizes — no eager fallback needed).
KV pool: 7.34 GiB (their boot-10: 7.89 GiB), 300K ctx, DSpark k=5.

Smoke (boot 1) — all pass

chat: "2+2" → "Четыре" (0 reasoning tokens, thinking off by default);
tools: math tool call add(a=0, b=2) correct;
vision: 64×64 red png → "Красного".

Bench (boot 2, local Engram) — `2026-09-10_231741_dsv41-local-0910`

156/176 (89/100), ★★★★ "Good", 88/88 scenarios completed, 0 errors/crashes, 0 safety warnings. 70 pass / 16 partial / 2 fail (TC-61, TC-88).

Categories (earned/max): A 5/6, B 6/6, C 6/8, D 6/6, E 5/6, F 6/6, G 6/6, H 10/10, I 14/20 (70%), J 6/6, K 24/26, L 8/8, M 3/6 (50% — all partial), N 6/6, O 12/12 (structured output 100%), P 33/38.

Weak spots vs our GLM: autonomous planning (M 50%), context & state (I 70%). Strong: structured output, toolset scale, localization (RU), code patterns — all 100%.

Perf (pp t/s / tg t/s / TTFT ms / total ms, pp2048 tg128):

depth	c1	c2
d0	1,090 / 36.1 / 1,876 / 5,295	1,052 / 36.5 / 3,086 / 8,858
d4096	1,124 / 32.2 / 5,418 / 9,272	1,172 / 26.0 / 8,278 / 15,723
d8192	1,411 / 31.2 / 7,421 / 11,398	1,371 / 20.9 / 10,973 / 19,633

(vs GLM r31 4x d0 c1: 1,484 pp / 51.6 tg / 1,785 ms — DSV4.1 decode is ~30% slower, TTFT on par. 16B-active MoE vs GLM-5.3-Flash. Their own claims: prefill 902–1,539 t/s — our 1,052–1,411 matches; their C6 agg 131.9 t/s is a concurrency-6 aggregate, not directly comparable.)

Verdict: quality within 3 points of our GLM prod (159/176), reliability clean (no death under the full 88-scenario load — unlike sparkring), structured output 100%. Slower decode/TTFT. A serious candidate for a second production lane.

eugr lane refresh

New image eugr/spark-vllm-b12x:nightly-20260910 (published today 13:06 UTC) pulled to dgx1 + dgx2. Recipe glm53-flash-eugr-2x needs its docker_image bumped next window; bench vs r28 2x (158/176) still queued.

Carried

dgx5 left powered on (used for the DSV4.1 build).
sparkring 2x re-run (memory-guard image), 386B infra decision, gpuflip probe — all still queued for a window.

DSV4.1 reconfig per operator (06:42 local, boot #4)

Requested: 512K/1M context "как везде", +5GB KV (they saw ~10GB free), thinking ON + reasoning_effort=max by default, served alias deepseek-v4-flash (gateway config stays untouched).

--max-model-len 1048576 (1M; their boot7-go proof = 1,078,380-token pool at gmu 0.80/eager — ours at graphs), --gpu-memory-utilization 0.85 (+2.9 GiB KV: 8.45 → 11.39 GiB pool ≈ 1.6M tokens = 1.6x the 1M seq). Caveat from their RECIPE.md: the top-k fix was validated at row widths up to 300K, "the full 1M-wide case has not been re-run" — even for them.
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "max"}' (verified accepted per-request on the running server first).
--served-model-name deepseek-v4-flash.
Boot #4: clean. KV 11.39 GiB. Thinking on by default (16 reasoning tokens on a simple query). Needle test: 377,063-token prompt, secret word at the very end → "KILORK-42" exact (3m56s total, ~1.6K t/s effective prefill) — 377K-wide decode path verified working.

Note: the 156/176 bench above was measured at thinking OFF (their default); thinking-on + max-effort scoring is unmeasured — offer a re-bench if the operator wants the apples-to-apples number for the new default.

2026-09-11.md 6.2 KB История Исходник