Главная Обзор Помощь
Вход
sm121
/
launcher
2
0
Ответвить 0
Файлы Задачи 0 Запросы на слияние 0 Вики
Дерево: 6a46ad87da
Ветки Метки
launcher
/
history
/
night-shifts
/
2026-09-11.md

2026-09-11.md 10 KB
История Исходник

Night shift 2026-09-11 (local) — DSV4.1 first deployment + eugr lane refresh

Window opened by operator at ~21:50 local ("прод до утра не нужен"). Priority: DeepSeek-V4.1-Flash campaign.

Production

  • Stopped glm53-flash-r31-4x-dcp4 (21:50 local).
  • End state: dsv41-flash-4x (4nodes-switch) is the live 4x config — operator to decide in the morning: keep DSV4.1 up or switch back to r31 (one stop+serve).

DSV4.1-Flash (the campaign)

Image

  • Built on dgx5 (powered on via HA, buildx docker-container builder): dsv41-flash:local 23.8GB, checksum label 91b2523a46afae80279d7ef8ddb2e7308192f70097e4eca32b8d714a98aa199d (build log: /tmp/dsv41-flash-4x-build.log on dgx5).
  • Build fixes (commit dd86b8a): cutlass extraction into build/_deps/cutlass-src, cmake/ninja/g++ install, NVRTC resolution via -DCUDA_nvrtc_LIBRARY, stable-ext validation switched to file-existence (no GPU driver in the build container).
  • Transferred to dgx1–dgx4 via docker save | ssh docker load (single-transfer pattern; the first 4-parallel batch failed on nested host-key race, single transfers are reliable).
  • Boot-1 bug: the base vllm/vllm-openai image bakes ENTRYPOINT ["vllm","serve"]; our recipe's entrypoint = ["vllm","serve"] doubled the command (vllm serve vllm serve <model> → instant exit). Fix: no entrypoint in the recipe — the launcher's CMD (model + flags) completes the baked entrypoint. No image rebuild needed.

Engram node-local copies

  • First boot (ENGRAM_DISK, NFS) printed the per-rank row ranges:
    • layer 1: r0 [0, 96000564) / r1 [96000564, 192001740) / r2 [192001740, 288003654) / r3 [288003654, 384006168)
    • layer 14: r0 [0, 96003054) / r1 [96003054, 192007016) / r2 [192007016, 288011564) / r3 [288011564, 384016682)
  • tools/engram_local.py (theirs, sparse same-offset, 600 MB/s cap, page-cache drops, byte-for-byte verify) run per node against the corresponding rank's ranges → /var/tmp/engram-local/DeepSeek-V4.1-Flash on all 4 nodes: 47.2 GiB each, verify 0 mismatches.
  • Recipe: DSV41_ENGRAM_LOCAL=1, DSV41_ENGRAM_DIR=/engram-local, ro bind mount /var/tmp/engram-local/DeepSeek-V4.1-Flash → /engram-local. Boot log confirms: Engram layer N rows [...] read from node-local /engram-local.

Boot notes

  • Weight load: 475 GiB NFS4, 48 shards fast pass ~5.5 min, then a second per-expert pass over all 48 shards (W13/W2 per-expert copies, ~200 MB/s from NFS, ~30 min). Total boot ≈ 45 min (boot 1), faster for boot 2 with the local Engram + warm NFS page cache.
  • CUDA graphs worked from the first boot (prestage patch + exact capture sizes — no eager fallback needed).
  • KV pool: 7.34 GiB (their boot-10: 7.89 GiB), 300K ctx, DSpark k=5.

Smoke (boot 1) — all pass

  • chat: "2+2" → "Четыре" (0 reasoning tokens, thinking off by default);
  • tools: math tool call add(a=0, b=2) correct;
  • vision: 64×64 red png → "Красного".

Bench (boot 2, local Engram) — 2026-09-10_231741_dsv41-local-0910

156/176 (89/100), ★★★★ "Good", 88/88 scenarios completed, 0 errors/crashes, 0 safety warnings. 70 pass / 16 partial / 2 fail (TC-61, TC-88).

Categories (earned/max): A 5/6, B 6/6, C 6/8, D 6/6, E 5/6, F 6/6, G 6/6, H 10/10, I 14/20 (70%), J 6/6, K 24/26, L 8/8, M 3/6 (50% — all partial), N 6/6, O 12/12 (structured output 100%), P 33/38.

Weak spots vs our GLM: autonomous planning (M 50%), context & state (I 70%). Strong: structured output, toolset scale, localization (RU), code patterns — all 100%.

Perf (pp t/s / tg t/s / TTFT ms / total ms, pp2048 tg128):

depth c1 c2
d0 1,090 / 36.1 / 1,876 / 5,295 1,052 / 36.5 / 3,086 / 8,858
d4096 1,124 / 32.2 / 5,418 / 9,272 1,172 / 26.0 / 8,278 / 15,723
d8192 1,411 / 31.2 / 7,421 / 11,398 1,371 / 20.9 / 10,973 / 19,633

(vs GLM r31 4x d0 c1: 1,484 pp / 51.6 tg / 1,785 ms — DSV4.1 decode is ~30% slower, TTFT on par. 16B-active MoE vs GLM-5.3-Flash. Their own claims: prefill 902–1,539 t/s — our 1,052–1,411 matches; their C6 agg 131.9 t/s is a concurrency-6 aggregate, not directly comparable.)

Verdict: quality within 3 points of our GLM prod (159/176), reliability clean (no death under the full 88-scenario load — unlike sparkring), structured output 100%. Slower decode/TTFT. A serious candidate for a second production lane.

eugr lane refresh

  • New image eugr/spark-vllm-b12x:nightly-20260910 (published today 13:06 UTC) pulled to dgx1 + dgx2. Recipe glm53-flash-eugr-2x needs its docker_image bumped next window; bench vs r28 2x (158/176) still queued.

Carried

  • dgx5 left powered on (used for the DSV4.1 build).
  • sparkring 2x re-run (memory-guard image), 386B infra decision, gpuflip probe — all still queued for a window.

DSV4.1 reconfig per operator (06:42 local, boot #4)

Requested: 512K/1M context "как везде", +5GB KV (they saw ~10GB free), thinking ON + reasoning_effort=max by default, served alias deepseek-v4-flash (gateway config stays untouched).

  • --max-model-len 1048576 (1M; their boot7-go proof = 1,078,380-token pool at gmu 0.80/eager — ours at graphs), --gpu-memory-utilization 0.85 (+2.9 GiB KV: 8.45 → 11.39 GiB pool ≈ 1.6M tokens = 1.6x the 1M seq). Caveat from their RECIPE.md: the top-k fix was validated at row widths up to 300K, "the full 1M-wide case has not been re-run" — even for them.
  • --default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "max"}' (verified accepted per-request on the running server first).
  • --served-model-name deepseek-v4-flash.
  • Boot #4: clean. KV 11.39 GiB. Thinking on by default (16 reasoning tokens on a simple query). Needle test: 377,063-token prompt, secret word at the very end → "KILORK-42" exact (3m56s total, ~1.6K t/s effective prefill) — 377K-wide decode path verified working.

Note: the 156/176 bench above was measured at thinking OFF (their default); thinking-on + max-effort scoring is unmeasured — offer a re-bench if the operator wants the apples-to-apples number for the new default.

DSV4.1 ctx 1M → 512K (boot #5, operator decision) + Aiden base analysis

Operator: "контекст в 512К" + correction: Aiden's page says production-1.1 where it should say production-1.2 (the DCP2-baked image; the page is not updated to the end). Recipe now: 524288 ctx, gmu 0.85, recipe header cites the AidenLab write-up.

Boot #5 results (512K, gmu 0.85):

  • KV pool 13.52 GiB (vs 11.39 at 1M — the indexer prefill buffer halves, ~2.6 vs ~5.2 GiB).
  • Counting decode (thinking off): 47.9 → 71.5 t/s (+50%) — the 1M context was costing ~35% of decode throughput (per-step indexer scoring grows with max context). Thinking on (default): 65.8 t/s.
  • vs Aiden's boot-1 (300K, same stack, 84 t/s counting): gap closed from 1.75× to 1.17× — the remainder is exactly their B12X MoE (+9%) and DCP2 (+9%) ≈ 18% they ship. Our stack at 512K is at parity with their pre-b12x baseline.

Aiden base (aidenle.com, extracted from the SPA JS bundle):

  • Boot 1 = tonyd2wild boot-10 config = our recipe; +3 changes: B12X MoE backend (b12x==1.3.0 wheel, --moe-backend b12x: decode 81→89-93, prefill 9K +22%), explicit --kv-cache-memory-bytes (8 GiB = 1.26M tokens, head 9 GB free; 10 GiB = 2 GB free — too close to earlyoom), day-0 official _C_stable_libtorch reuse (their build seconds vs our hours — provenance choice, ours is clone-at-pin from source).
  • DCP2 (21→23 patch files, PROFILE=dcp2, 5 env switches, image production-1.1/1.2): KV 1,888,800 tokens at 300K (6.3×), decode 93.6/64.4/31.5 (1.2), 128K prefill 60.5 s (2,150 t/s), 8-way soak clean, their prod since 08:58. DCP halves KV bytes per rank (window+compressed caches sharded over compressed states), decode cost 5-9%.
  • Docker Hub aidendle94/sparkrun-vllm-dsv41-gb10: only production-1.0 published (9 GB arm64, b12x + 8GiB KV, no DCP2) — 1.1/1.2 not pushed; their DCP2 image A/B is blocked until Aiden publishes it (or the 23-file set lands for our tree).

Next candidates: (a) A/B bench of our 512K config (thinking on, the new default) for the apples-to-apples number; (b) b12x into our image; (c) DCP2 23-file set from Aiden into our clone-at-pin build.

DSV4.1 full bench at 512K + thinking (boot #5 config, 10:06–12:49 local)

2026-09-11_100632_dsv41-512k-think-0911 — 158/176 = 90/100, ★★★★★ "Excellent", 74 pass / 10 partial / 4 fail (TC-43, TC-68, TC-84, TC-88), safety gate passed, 0 warnings. (The thinking-off 1M-context run was 156/176 "Good".)

Thinking ON deltas vs thinking OFF: A +1 (100%), C +2 (multi-step 100%), E +1 (100%), I +3 (context 85%); K −2 (safety 85%), O −2 (structured output 83% — thinking leaks prose into JSON-only outputs), P −1. M stays 50% (autonomous planning — the model's structural weak spot). TC-61 (fail at thinking-off) now passes; TC-88 fails in both runs.

Perf (512K, thinking off in the raw perf table):

depth c1 c2
d0 1,387 pp / 39.9 tg / 1,886 ms 1,450 / 58.9 / 2,620
d4096 1,605 / 41.7 / 4,302 1,568 / 35.3 / 6,651
d8192 1,783 / 41.7 / 6,057 1,505 / 28.2 / 10,783

vs the 1M run (1,090/36.1/1,876 at d0 c1): prefill +27%, decode +10% — 512K helps both. 448K→801K total tokens (thinking on).

b12x A/B prep (built on dgx5 while the bench ran)

  • docker-images/dsv41-flash-b12x/ thin wrapper (FROM dsv41-flash:local + pip b12x==1.3.0 + B12X_CACHE_DIR) — classic docker driver in build.sh (local-only FROM breaks the buildx container driver), 30 s build, label 4d6136bb..., transferred to dgx1–4.
  • recipes/dsv41-flash-4x-b12x = 512K recipe + --moe-backend b12x. Ready for the next window; expected decode +9% / prefill 9K +22% (Aiden's numbers).
  • Aiden: operator published the request for production-1.2 / the 23-file DCP2 set; hub check confirmed only production-1.0 is public.