# Night shift 2026-09-11 (local) — DSV4.1 first deployment + eugr lane refresh

Window opened by operator at ~21:50 local ("прод до утра не нужен"). Priority:
DeepSeek-V4.1-Flash campaign.

## Production

- Stopped `glm53-flash-r31-4x-dcp4` (21:50 local).
- **End state: `dsv41-flash-4x` (4nodes-switch) is the live 4x config** —
  operator to decide in the morning: keep DSV4.1 up or switch back to r31
  (one stop+serve).

## DSV4.1-Flash (the campaign)

### Image

- Built on **dgx5** (powered on via HA, buildx docker-container builder):
  `dsv41-flash:local` 23.8GB, checksum label
  `91b2523a46afae80279d7ef8ddb2e7308192f70097e4eca32b8d714a98aa199d`
  (build log: `/tmp/dsv41-flash-4x-build.log` on dgx5).
- Build fixes (commit dd86b8a): cutlass extraction into
  `build/_deps/cutlass-src`, cmake/ninja/g++ install, NVRTC resolution via
  `-DCUDA_nvrtc_LIBRARY`, stable-ext validation switched to file-existence
  (no GPU driver in the build container).
- Transferred to dgx1–dgx4 via `docker save | ssh docker load`
  (single-transfer pattern; the first 4-parallel batch failed on nested
  host-key race, single transfers are reliable).
- **Boot-1 bug**: the base `vllm/vllm-openai` image bakes
  `ENTRYPOINT ["vllm","serve"]`; our recipe's `entrypoint = ["vllm","serve"]`
  doubled the command (`vllm serve vllm serve <model>` → instant exit).
  Fix: **no `entrypoint` in the recipe** — the launcher's CMD (model + flags)
  completes the baked entrypoint. No image rebuild needed.

### Engram node-local copies

- First boot (ENGRAM_DISK, NFS) printed the per-rank row ranges:
  - layer 1:  r0 [0, 96000564) / r1 [96000564, 192001740) /
    r2 [192001740, 288003654) / r3 [288003654, 384006168)
  - layer 14: r0 [0, 96003054) / r1 [96003054, 192007016) /
    r2 [192007016, 288011564) / r3 [288011564, 384016682)
- `tools/engram_local.py` (theirs, sparse same-offset, 600 MB/s cap,
  page-cache drops, byte-for-byte verify) run per node against the
  corresponding rank's ranges → `/var/tmp/engram-local/DeepSeek-V4.1-Flash`
  on all 4 nodes: **47.2 GiB each, verify 0 mismatches**.
- Recipe: `DSV41_ENGRAM_LOCAL=1`, `DSV41_ENGRAM_DIR=/engram-local`,
  ro bind mount `/var/tmp/engram-local/DeepSeek-V4.1-Flash → /engram-local`.
  Boot log confirms: `Engram layer N rows [...] read from node-local /engram-local`.

### Boot notes

- Weight load: 475 GiB NFS4, 48 shards fast pass ~5.5 min, then a **second
  per-expert pass over all 48 shards** (W13/W2 per-expert copies, ~200 MB/s
  from NFS, ~30 min). Total boot ≈ 45 min (boot 1), faster for boot 2 with
  the local Engram + warm NFS page cache.
- **CUDA graphs worked from the first boot** (prestage patch + exact capture
  sizes — no eager fallback needed).
- KV pool: 7.34 GiB (their boot-10: 7.89 GiB), 300K ctx, DSpark k=5.

### Smoke (boot 1) — all pass

- chat: "2+2" → "Четыре" (0 reasoning tokens, thinking off by default);
- tools: math tool call `add(a=0, b=2)` correct;
- vision: 64×64 red png → "Красного".

### Bench (boot 2, local Engram) — `2026-09-10_231741_dsv41-local-0910`

**156/176 (89/100), ★★★★ "Good", 88/88 scenarios completed, 0 errors/crashes,
0 safety warnings.** 70 pass / 16 partial / 2 fail (TC-61, TC-88).

Categories (earned/max): A 5/6, B 6/6, C 6/8, D 6/6, E 5/6, F 6/6, G 6/6,
H 10/10, I 14/20 (70%), J 6/6, K 24/26, L 8/8, **M 3/6 (50% — all partial)**,
N 6/6, **O 12/12 (structured output 100%)**, P 33/38.

Weak spots vs our GLM: autonomous planning (M 50%), context & state (I 70%).
Strong: structured output, toolset scale, localization (RU), code patterns —
all 100%.

Perf (pp t/s / tg t/s / TTFT ms / total ms, pp2048 tg128):

| depth | c1                            | c2                            |
|-------|-------------------------------|-------------------------------|
| d0    | 1,090 / 36.1 / 1,876 / 5,295  | 1,052 / 36.5 / 3,086 / 8,858  |
| d4096 | 1,124 / 32.2 / 5,418 / 9,272  | 1,172 / 26.0 / 8,278 / 15,723 |
| d8192 | 1,411 / 31.2 / 7,421 / 11,398 | 1,371 / 20.9 / 10,973 / 19,633 |

(vs GLM r31 4x d0 c1: 1,484 pp / 51.6 tg / 1,785 ms — DSV4.1 decode is
~30% slower, TTFT on par. 16B-active MoE vs GLM-5.3-Flash. Their own
claims: prefill 902–1,539 t/s — our 1,052–1,411 matches; their C6 agg
131.9 t/s is a concurrency-6 aggregate, not directly comparable.)

Verdict: quality within 3 points of our GLM prod (159/176), reliability
clean (no death under the full 88-scenario load — unlike sparkring),
structured output 100%. Slower decode/TTFT. A serious candidate for a
second production lane.

## eugr lane refresh

- New image `eugr/spark-vllm-b12x:nightly-20260910` (published today 13:06
  UTC) pulled to dgx1 + dgx2. Recipe `glm53-flash-eugr-2x` needs its
  `docker_image` bumped next window; bench vs r28 2x (158/176) still queued.

## Carried

- dgx5 left powered on (used for the DSV4.1 build).
- sparkring 2x re-run (memory-guard image), 386B infra decision, gpuflip
  probe — all still queued for a window.

## DSV4.1 reconfig per operator (06:42 local, boot #4)

Requested: 512K/1M context "как везде", +5GB KV (they saw ~10GB free),
thinking ON + reasoning_effort=max by default, served alias
`deepseek-v4-flash` (gateway config stays untouched).

- `--max-model-len 1048576` (1M; their boot7-go proof = 1,078,380-token pool
  at gmu 0.80/eager — ours at graphs), `--gpu-memory-utilization 0.85`
  (+2.9 GiB KV: 8.45 → 11.39 GiB pool ≈ 1.6M tokens = 1.6x the 1M seq).
  Caveat from their RECIPE.md: the top-k fix was validated at row widths up
  to 300K, "the full 1M-wide case has not been re-run" — even for them.
- `--default-chat-template-kwargs '{"thinking": true, "reasoning_effort":
  "max"}'` (verified accepted per-request on the running server first).
- `--served-model-name deepseek-v4-flash`.
- Boot #4: clean. KV 11.39 GiB. Thinking on by default (16 reasoning tokens
  on a simple query). **Needle test: 377,063-token prompt, secret word at
  the very end → "KILORK-42" exact** (3m56s total, ~1.6K t/s effective
  prefill) — 377K-wide decode path verified working.

Note: the 156/176 bench above was measured at thinking OFF (their default);
thinking-on + max-effort scoring is unmeasured — offer a re-bench if the
operator wants the apples-to-apples number for the new default.

## DSV4.1 ctx 1M → 512K (boot #5, operator decision) + Aiden base analysis

Operator: "контекст в 512К" + correction: Aiden's page says
production-1.1 where it should say **production-1.2** (the DCP2-baked
image; the page is not updated to the end). Recipe now: 524288 ctx, gmu
0.85, recipe header cites the AidenLab write-up.

Boot #5 results (512K, gmu 0.85):
- KV pool **13.52 GiB** (vs 11.39 at 1M — the indexer prefill buffer
  halves, ~2.6 vs ~5.2 GiB).
- **Counting decode (thinking off): 47.9 → 71.5 t/s (+50%)** — the 1M
  context was costing ~35% of decode throughput (per-step indexer
  scoring grows with max context). Thinking on (default): 65.8 t/s.
- vs Aiden's boot-1 (300K, same stack, 84 t/s counting): gap closed from
  1.75× to 1.17× — the remainder is exactly their B12X MoE (+9%) and DCP2
  (+9%) ≈ 18% they ship. Our stack at 512K is at parity with their
  pre-b12x baseline.

Aiden base (aidenle.com, extracted from the SPA JS bundle):
- Boot 1 = tonyd2wild boot-10 config = our recipe; +3 changes: B12X MoE
  backend (b12x==1.3.0 wheel, `--moe-backend b12x`: decode 81→89-93,
  prefill 9K +22%), explicit `--kv-cache-memory-bytes` (8 GiB = 1.26M
  tokens, head 9 GB free; 10 GiB = 2 GB free — too close to earlyoom),
  day-0 official `_C_stable_libtorch` reuse (their build seconds vs our
  hours — provenance choice, ours is clone-at-pin from source).
- DCP2 (21→23 patch files, `PROFILE=dcp2`, 5 env switches, image
  production-1.1/1.2): KV 1,888,800 tokens at 300K (6.3×), decode
  93.6/64.4/31.5 (1.2), 128K prefill 60.5 s (2,150 t/s), 8-way soak clean,
  their prod since 08:58. DCP halves KV bytes per rank (window+compressed
  caches sharded over compressed states), decode cost 5-9%.
- Docker Hub `aidendle94/sparkrun-vllm-dsv41-gb10`: **only
  production-1.0 published** (9 GB arm64, b12x + 8GiB KV, no DCP2) —
  1.1/1.2 not pushed; their DCP2 image A/B is blocked until Aiden
  publishes it (or the 23-file set lands for our tree).

Next candidates: (a) A/B bench of our 512K config (thinking on, the new
default) for the apples-to-apples number; (b) b12x into our image;
(c) DCP2 23-file set from Aiden into our clone-at-pin build.