# Night shift 2026-09-11 (local) — DSV4.1 first deployment + eugr lane refresh

Window opened by operator at ~21:50 local ("прод до утра не нужен"). Priority:
DeepSeek-V4.1-Flash campaign.

## Production

- Stopped `glm53-flash-r31-4x-dcp4` (21:50 local).
- **End state: `dsv41-flash-4x` (4nodes-switch) is the live 4x config** —
  operator to decide in the morning: keep DSV4.1 up or switch back to r31
  (one stop+serve).

## DSV4.1-Flash (the campaign)

### Image

- Built on **dgx5** (powered on via HA, buildx docker-container builder):
  `dsv41-flash:local` 23.8GB, checksum label
  `91b2523a46afae80279d7ef8ddb2e7308192f70097e4eca32b8d714a98aa199d`
  (build log: `/tmp/dsv41-flash-4x-build.log` on dgx5).
- Build fixes (commit dd86b8a): cutlass extraction into
  `build/_deps/cutlass-src`, cmake/ninja/g++ install, NVRTC resolution via
  `-DCUDA_nvrtc_LIBRARY`, stable-ext validation switched to file-existence
  (no GPU driver in the build container).
- Transferred to dgx1–dgx4 via `docker save | ssh docker load`
  (single-transfer pattern; the first 4-parallel batch failed on nested
  host-key race, single transfers are reliable).
- **Boot-1 bug**: the base `vllm/vllm-openai` image bakes
  `ENTRYPOINT ["vllm","serve"]`; our recipe's `entrypoint = ["vllm","serve"]`
  doubled the command (`vllm serve vllm serve <model>` → instant exit).
  Fix: **no `entrypoint` in the recipe** — the launcher's CMD (model + flags)
  completes the baked entrypoint. No image rebuild needed.

### Engram node-local copies

- First boot (ENGRAM_DISK, NFS) printed the per-rank row ranges:
  - layer 1:  r0 [0, 96000564) / r1 [96000564, 192001740) /
    r2 [192001740, 288003654) / r3 [288003654, 384006168)
  - layer 14: r0 [0, 96003054) / r1 [96003054, 192007016) /
    r2 [192007016, 288011564) / r3 [288011564, 384016682)
- `tools/engram_local.py` (theirs, sparse same-offset, 600 MB/s cap,
  page-cache drops, byte-for-byte verify) run per node against the
  corresponding rank's ranges → `/var/tmp/engram-local/DeepSeek-V4.1-Flash`
  on all 4 nodes: **47.2 GiB each, verify 0 mismatches**.
- Recipe: `DSV41_ENGRAM_LOCAL=1`, `DSV41_ENGRAM_DIR=/engram-local`,
  ro bind mount `/var/tmp/engram-local/DeepSeek-V4.1-Flash → /engram-local`.
  Boot log confirms: `Engram layer N rows [...] read from node-local /engram-local`.

### Boot notes

- Weight load: 475 GiB NFS4, 48 shards fast pass ~5.5 min, then a **second
  per-expert pass over all 48 shards** (W13/W2 per-expert copies, ~200 MB/s
  from NFS, ~30 min). Total boot ≈ 45 min (boot 1), faster for boot 2 with
  the local Engram + warm NFS page cache.
- **CUDA graphs worked from the first boot** (prestage patch + exact capture
  sizes — no eager fallback needed).
- KV pool: 7.34 GiB (their boot-10: 7.89 GiB), 300K ctx, DSpark k=5.

### Smoke (boot 1) — all pass

- chat: "2+2" → "Четыре" (0 reasoning tokens, thinking off by default);
- tools: math tool call `add(a=0, b=2)` correct;
- vision: 64×64 red png → "Красного".

### Bench (boot 2, local Engram) — `2026-09-10_231741_dsv41-local-0910`

**156/176 (89/100), ★★★★ "Good", 88/88 scenarios completed, 0 errors/crashes,
0 safety warnings.** 70 pass / 16 partial / 2 fail (TC-61, TC-88).

Categories (earned/max): A 5/6, B 6/6, C 6/8, D 6/6, E 5/6, F 6/6, G 6/6,
H 10/10, I 14/20 (70%), J 6/6, K 24/26, L 8/8, **M 3/6 (50% — all partial)**,
N 6/6, **O 12/12 (structured output 100%)**, P 33/38.

Weak spots vs our GLM: autonomous planning (M 50%), context & state (I 70%).
Strong: structured output, toolset scale, localization (RU), code patterns —
all 100%.

Perf (pp t/s / tg t/s / TTFT ms / total ms, pp2048 tg128):

| depth | c1                            | c2                            |
|-------|-------------------------------|-------------------------------|
| d0    | 1,090 / 36.1 / 1,876 / 5,295  | 1,052 / 36.5 / 3,086 / 8,858  |
| d4096 | 1,124 / 32.2 / 5,418 / 9,272  | 1,172 / 26.0 / 8,278 / 15,723 |
| d8192 | 1,411 / 31.2 / 7,421 / 11,398 | 1,371 / 20.9 / 10,973 / 19,633 |

(vs GLM r31 4x d0 c1: 1,484 pp / 51.6 tg / 1,785 ms — DSV4.1 decode is
~30% slower, TTFT on par. 16B-active MoE vs GLM-5.3-Flash. Their own
claims: prefill 902–1,539 t/s — our 1,052–1,411 matches; their C6 agg
131.9 t/s is a concurrency-6 aggregate, not directly comparable.)

Verdict: quality within 3 points of our GLM prod (159/176), reliability
clean (no death under the full 88-scenario load — unlike sparkring),
structured output 100%. Slower decode/TTFT. A serious candidate for a
second production lane.

## eugr lane refresh

- New image `eugr/spark-vllm-b12x:nightly-20260910` (published today 13:06
  UTC) pulled to dgx1 + dgx2. Recipe `glm53-flash-eugr-2x` needs its
  `docker_image` bumped next window; bench vs r28 2x (158/176) still queued.

## Carried

- dgx5 left powered on (used for the DSV4.1 build).
- sparkring 2x re-run (memory-guard image), 386B infra decision, gpuflip
  probe — all still queued for a window.

## DSV4.1 reconfig per operator (06:42 local, boot #4)

Requested: 512K/1M context "как везде", +5GB KV (they saw ~10GB free),
thinking ON + reasoning_effort=max by default, served alias
`deepseek-v4-flash` (gateway config stays untouched).

- `--max-model-len 1048576` (1M; their boot7-go proof = 1,078,380-token pool
  at gmu 0.80/eager — ours at graphs), `--gpu-memory-utilization 0.85`
  (+2.9 GiB KV: 8.45 → 11.39 GiB pool ≈ 1.6M tokens = 1.6x the 1M seq).
  Caveat from their RECIPE.md: the top-k fix was validated at row widths up
  to 300K, "the full 1M-wide case has not been re-run" — even for them.
- `--default-chat-template-kwargs '{"thinking": true, "reasoning_effort":
  "max"}'` (verified accepted per-request on the running server first).
- `--served-model-name deepseek-v4-flash`.
- Boot #4: clean. KV 11.39 GiB. Thinking on by default (16 reasoning tokens
  on a simple query). **Needle test: 377,063-token prompt, secret word at
  the very end → "KILORK-42" exact** (3m56s total, ~1.6K t/s effective
  prefill) — 377K-wide decode path verified working.

Note: the 156/176 bench above was measured at thinking OFF (their default);
thinking-on + max-effort scoring is unmeasured — offer a re-bench if the
operator wants the apples-to-apples number for the new default.

## DSV4.1 ctx 1M → 512K (boot #5, operator decision) + Aiden base analysis

Operator: "контекст в 512К" + correction: Aiden's page says
production-1.1 where it should say **production-1.2** (the DCP2-baked
image; the page is not updated to the end). Recipe now: 524288 ctx, gmu
0.85, recipe header cites the AidenLab write-up.

Boot #5 results (512K, gmu 0.85):
- KV pool **13.52 GiB** (vs 11.39 at 1M — the indexer prefill buffer
  halves, ~2.6 vs ~5.2 GiB).
- **Counting decode (thinking off): 47.9 → 71.5 t/s (+50%)** — the 1M
  context was costing ~35% of decode throughput (per-step indexer
  scoring grows with max context). Thinking on (default): 65.8 t/s.
- vs Aiden's boot-1 (300K, same stack, 84 t/s counting): gap closed from
  1.75× to 1.17× — the remainder is exactly their B12X MoE (+9%) and DCP2
  (+9%) ≈ 18% they ship. Our stack at 512K is at parity with their
  pre-b12x baseline.

Aiden base (aidenle.com, extracted from the SPA JS bundle):
- Boot 1 = tonyd2wild boot-10 config = our recipe; +3 changes: B12X MoE
  backend (b12x==1.3.0 wheel, `--moe-backend b12x`: decode 81→89-93,
  prefill 9K +22%), explicit `--kv-cache-memory-bytes` (8 GiB = 1.26M
  tokens, head 9 GB free; 10 GiB = 2 GB free — too close to earlyoom),
  day-0 official `_C_stable_libtorch` reuse (their build seconds vs our
  hours — provenance choice, ours is clone-at-pin from source).
- DCP2 (21→23 patch files, `PROFILE=dcp2`, 5 env switches, image
  production-1.1/1.2): KV 1,888,800 tokens at 300K (6.3×), decode
  93.6/64.4/31.5 (1.2), 128K prefill 60.5 s (2,150 t/s), 8-way soak clean,
  their prod since 08:58. DCP halves KV bytes per rank (window+compressed
  caches sharded over compressed states), decode cost 5-9%.
- Docker Hub `aidendle94/sparkrun-vllm-dsv41-gb10`: **only
  production-1.0 published** (9 GB arm64, b12x + 8GiB KV, no DCP2) —
  1.1/1.2 not pushed; their DCP2 image A/B is blocked until Aiden
  publishes it (or the 23-file set lands for our tree).

Next candidates: (a) A/B bench of our 512K config (thinking on, the new
default) for the apples-to-apples number; (b) b12x into our image;
(c) DCP2 23-file set from Aiden into our clone-at-pin build.

## DSV4.1 full bench at 512K + thinking (boot #5 config, 10:06–12:49 local)

`2026-09-11_100632_dsv41-512k-think-0911` — **158/176 = 90/100,
★★★★★ "Excellent"**, 74 pass / 10 partial / 4 fail (TC-43, TC-68, TC-84,
TC-88), safety gate passed, 0 warnings. (The thinking-off 1M-context run
was 156/176 "Good".)

Thinking ON deltas vs thinking OFF: A +1 (100%), C +2 (multi-step 100%),
E +1 (100%), I +3 (context 85%); K −2 (safety 85%), O −2 (structured
output 83% — thinking leaks prose into JSON-only outputs), P −1. M stays
50% (autonomous planning — the model's structural weak spot).
TC-61 (fail at thinking-off) now passes; TC-88 fails in both runs.

Perf (512K, thinking off in the raw perf table):

| depth | c1                          | c2                          |
|-------|-----------------------------|-----------------------------|
| d0    | 1,387 pp / 39.9 tg / 1,886 ms | 1,450 / 58.9 / 2,620       |
| d4096 | 1,605 / 41.7 / 4,302        | 1,568 / 35.3 / 6,651        |
| d8192 | 1,783 / 41.7 / 6,057        | 1,505 / 28.2 / 10,783       |

vs the 1M run (1,090/36.1/1,876 at d0 c1): prefill +27%, decode +10% —
512K helps both. 448K→801K total tokens (thinking on).

## b12x A/B prep (built on dgx5 while the bench ran)

- `docker-images/dsv41-flash-b12x/` thin wrapper (FROM dsv41-flash:local +
  pip b12x==1.3.0 + B12X_CACHE_DIR) — classic docker driver in build.sh
  (local-only FROM breaks the buildx container driver), 30 s build,
  label 4d6136bb..., transferred to dgx1–4.
- `recipes/dsv41-flash-4x-b12x` = 512K recipe + `--moe-backend b12x`.
  Ready for the next window; expected decode +9% / prefill 9K +22%
  (Aiden's numbers).
- Aiden: operator published the request for production-1.2 / the 23-file
  DCP2 set; hub check confirmed only production-1.0 is public.

## reasoning_effort contract (operator question, verified on the live API)

The model's chat template is the DeepSeek "harmony" format: it renders a
system line `Reasoning Effort: N (range 1-100, the higher the value, the
more thorough the reasoning)` before the user message. Contract (verified
via /tokenize + /detokenize, and the template's own 400 message):

- `low` → 25, `high` → 50, `xhigh` → 75, `max` → 100 (string aliases)
- `none` → no effort line at all
- any JSON **integer** 1–100 → the number verbatim (strings "50" are
  rejected: must be a JSON number)
- anything else (`medium`, `min`, `off`, `minimal`, 101, -5, 1.5, "50")
  → HTTP 400: "DeepSeek V4.1 reasoning_effort must be low, high, xhigh,
  max, or an integer within [1, 100] in chat_template_kwargs"

Why the operator's curl "didn't look like it worked": `low` vs default
renders identically except one number (25 vs 100) — same token count.
Behavioral check (medium sheep problem): effort low = 180 reasoning
tokens, max = 252, both answer correctly — the knob is live.

Our recipe default `{"thinking": true, "reasoning_effort": "max"}` = 100 —
exactly as intended, no change. Gateway note: forward low/high/xhigh/max
or integers; `medium` is not a valid alias on this model.

## b12x A/B (window until dinner, 14:40–17:30 local) — WINNER

Boot: `--moe-backend b12x` accepted, log `Using 'B12X_MXFP4_MXFP8' Mxfp4
MoE backend`; weights 48/48 in 1:36 (warm NFS page cache); b12x JIT
1.4 MB cached on the per-node mount (warm for future boots).

Quick A/B (counting, thinking off): 71.5 → 75.0/95.4 t/s (+33% run 2).

Full bench `2026-09-11_145916_dsv41-512k-b12x-think-0911`:
**167/176 = 95/100 ★★★★★** (vs 158/176 without b12x, +9 points; 81 pass /
5 partial / 2 fail). MoE kernel numerics (MXFP4/MXFP8, clamped SwiGLU)
shift borderline scenario outcomes: P Hard Mode 32→**37/38 (+5)**, K
Safety 22→**25/26 (+3)**, I Context 17→18. Fixed: TC-43, TC-84, TC-88
(the three persistent fails); regressed: TC-51 (called send_email before
observing the calendar result — the only new safety note). TC-68 fails in
all four runs (model weakness, config-independent).

Perf (pp/tg/TTFT ms, vs no-b12x 512K):
d0 c1 1,642/47.0/1,738 (was 1,387/39.9/1,886: pp +18%, tg +18%)
d4096 c1 1,964/45.5/3,427 (was 1,605/41.7/4,302: pp +22% — Aiden's claimed
+22% at 9K, TTFT −20%)
d8192 c1 1,948/55.0/5,560 (was 1,783/41.7/6,057: tg +32%)

**Verdict: b12x wins on quality AND speed. `dsv41-flash-4x-b12x` (live
now) is the new DSV4.1 baseline.** Still pending from Aiden:
production-1.2 / DCP2 23-file set (hub check 14:40: only production-1.0).