launcher/history/night-shifts/2026-09-12.md

Night shift 2026-09-12 (local) — maximum shift: eugr 2x, sparkring re-run, Aiden DCP2 A/B

Operator plan (before start): "максимальная смена" — first update-check of all sources, then 2x window (eugr + sparkring), morning: Aiden image re-check, r31 back to production, dgx5 power-off consideration.

Phase 0 — source update-check (22:15 UTC)

• Aiden published production-1.1 at 16:54 UTC (Docker Hub) — the event of the shift. Pulled to all 4 nodes + dgx5 (23.4 GB, digest ede55d46).
• vLLM main is 338 commits ahead of our pin e47aa780 (09-10, "[Model] Support DeepSeek-V4.1-Flash", still not on main) and DSV4 fixes started flowing into main (2 today: indexer gather kernel, dflash attention padding) → next-rebase campaign material.
• FlashInfer: today's feat(sm120): runtime KV row stride + canonical 528B GLM53_NOPE rows + NaN-safe masked gather — candidate for the GLM lanes.
• DeepGEMM: new "Public Release 26/09" (09-10).
• SparkRing upstream: new r33-arm64-c8646b0 tag (R33 TP4/DCP4 SparkCache profile) + the memory-guard rebuild we needed tonight: glm53-shared-8b32ef2 = digest 4774a7c5 (confirmed via ghcr digest lookup; glm53-shared-74275757 = f25cdb6 = our current pin).
• tonyd2wild: new EXL3 TP4 context lane (3.3M-token KV pool on four Sparks), now their default serving config — note for the next DSV4.1 campaign.
• eugr: no fresh nightly (20260910 still the newest, already on the nodes).
• LMCache: quiet since 09-04.

eugr 2x — nightly-20260910 (report 2026-09-11_223443_eugr-2x-0910)

• 4x DSV4.1 stopped at 22:18 UTC; eugr 2x on dgx1+dgx2.
• 166/176 = 94/100 (81 pass / 4 partial TC-51/57/62/63 / 3 fail TC-45/68/80) vs r28 2x baseline 158/176 (90) — their b12x + humming-MTP + 1M-ctx stack is +8 points on 2 nodes.
• Perf d0 c1 (pp/tg/TTFT ms): 1,198 / 29.0 / 2,113 vs r28 1,475 / 27.2 / 1,362 — slower prefill and a 55% higher TTFT (1M-ctx prefill buffer, the same effect we measured on DSV4.1 at 1M).
• ⚠ tool-eval-bench drifted to v2.6.1.dev65+g6be685f0e this shift (all earlier runs: dev46+g9d412464a) — cross-run comparisons spanning the drift are approximate.

sparkring 2x re-run on the memory-guard image (report 2026-09-11_235045_...)

• Pulled ghcr.io/fujitsupolycom/sparkring:glm53-shared-8b32ef2 (= 4774a7c5, the 2 GB memory-guard rebuild) to dgx1+dgx2.
• Boot-1 bug (fixed in recipe, commit 13720ae): the new image's b12x loader JIT-compiles its native module at first boot and the linker cannot find libcudart (the old image prebaked it). Verified the fix in a scratch container: LIBRARY_PATH=/usr/local/cuda/lib64 (+ CUDA_HOME), plus a persistent B12X_CACHE_DIR mount so the compile survives reboots.
• Reliability verdict: FAIL, again. The engine died at scenario ~62 (TC-62 ran 123 s, then all 26 remaining scenarios infra-failed: timeout/connection/5xx). 112/124, 70.5% completion — the "90/100" header is a partial score, not comparable to full runs. No system OOM (114 GB free at inspection) — the death is container-internal (their liveness/memory guard, per the 09-08 forensics). The 2 GB guard fix did not fix the sustained-load death.

Aiden production-1.1 — DCP2 A/B (report 2026-09-12_003640_aiden11-dcp2-0912)

Image internals (verified on dgx5): vllm 0.28.1rc1.dev388+g8a728663c (base vllm/vllm-openai:nightly-8a728663, not our e47aa780 pin) + b12x 1.3.0 + FI 0.7.0rc1 + torch 2.13.0+cu130; the 24-file DCP2 patch set is baked (manifest at /opt/dsv41-dcp2-mounts.txt) + FI source with trtllm_dcp_alltoall.cu / cake_dcp.py. DCP env defaults are OFF (DSV41_DCP_STAGE=3, DCP_COMBINE=ag_rs, DCP_PREFILL_GATHER=0, ENGRAM_MMAP/PREFETCH=0); CLI: --decode-context-parallel-size 2 (-dcp), --dcp-comm-backend {ag_rs|a2a}.

Recipe dsv41-flash-4x-aiden11 = their DCP2 core (TP4 + dcp=2 + a2a + 8 GiB pinned KV, their validated 300K row) on our serving config (DSpark k=5 exact capture, SEQS 8, thinking+max, engram-local, deepseek-v4-flash alias).

• Boot: clean, Worker_TP0_DCP0, weights 48/48 in 92 s (warm NFS), "DSA indexer: DCP2 with compress_ratio=2 (state-sharded K cache)", CUDA graphs 11 s, KV 2,518,386 tokens @ 300K (better than their claimed 1.89M), 8.39× 300K concurrency.
• Result: 158/176 = 90/100 (74 pass / 10 partial / 4 fail TC-43/51/68/84) vs our b12x baseline 167/176 (95) @ 512K.
• Perf d0 c1 (pp/tg/TTFT ms): 1,517 / 41.8 / 2,446 vs ours 1,642 / 47.0 / 1,738 — we win prefill, decode, and TTFT.
• Verdict: the DCP2 A/B is ours to keep. DCP2 buys KV capacity (2.5M tokens) / high-concurrency production, not quality or single-stream speed. TC-43/84 (fixed by our b12x numerics) fail on their stack again.
• Option left open: the 24-file DCP2 set is extractable from the image (docker cp of the manifest's targets) if the operator ever wants DCP2 on our clone-at-pin tree — no need to wait for Aiden's file dump.

End state

• Production: glm53-flash-r31-4x-dcp4 (4nodes-switch) — restarted as morning prod per operator plan.
• DSV4.1 lane fully staged: dsv41-flash-4x-b12x recipe + images on all nodes + engram-local copies; Aiden 1.1 image on all 4 nodes for re-runs.
• dgx5: powered off (nothing pending on it; power on via HA when a build is needed).
• Gist (published daytime): https://gist.github.com/kilork/1ba224bfe28779c571d096cf954fff31 — base recipe card + live b12x overlay instructions in the README.

Production state after the daytime window

• Production: glm53-flash-r35-4x-dcp4 (4nodes-switch, promoted 09-12 per operator decision — r35 stays; r31 recipe remains as the rollback lane, image tagged on all nodes).
• Follow-ups added: (a) r31 re-bench on dev65 at the next 4x window for a harness-clean delta (and to settle TC-51 variance-vs-regression); (b) 32K-prefill probe if the split-MoE prefill gain on GB10 ever matters (outside the current bench matrix).

Morning follow-ups (ranked)

1. Aiden production-1.2 on Hub (DCP2-1.2 with the launcher env values) — re-check; the 1.1 A/B already answers the direction.
2. vLLM rebase campaign: DSV4.1 support is landing in main (338 commits ahead of our pin) — re-pin + re-patch when the branch stabilizes.
3. FlashInfer sm120 update (today's GLM53_NOPE rows commit) for the GLM lanes.
4. SparkRing r33-arm64-c8646b0 (R33 profile) — if the reliability story ever changes.
5. tonyd2wild EXL3 TP4 lane (3.3M KV pool) — next DSV4.1 quant option.
6. gpuflip.py GB10 slow-state probe on our 4 nodes (read-only, not done).

Daytime follow-up: R35 adoption + A/B (09-12, operator window)

• Adopted R35 (latest lab rev, qualified 2026-09-11, swiglu-reviewed- composition-r35.source.lock): full 3-fork cut — vLLM de982a50 (tree 6d5cc349), b12x 98086604 (tree d963f030), LMCache 29bc5a2e (tree 5a88a1ea). All pins verified fetchable + tree-matched; build-dep set r31↔r35 identical (no fetch-deps resync). profiles/glm53-r35.env + recipe glm53-flash-r35-4x-dcp4.
• Image built on dgx5 (71 m 18 s, checksum 8bb87348), boot gates green (vllm string 0.1.dev0+glm53.r35.de982a50, b12x/lmcache/flashkda OK); pre-transferred dgx5→dgx1 (docker save|ssh, ~11 min) before the window.
• A/B (4nodes-switch, r31 stopped, dev65 harness, seed 42, hardmode, c1/c2): | | r31 (09-10, dev46) | r35 (09-12, dev65) | |---|---|---| | score | 159/176 (90) | 163/176 (93) | | pp/tg/TTFT d0 c1 (ms) | 1,519 / 50.0 / 1,524 | 1,554 / 49.6 / 1,544 | | KV | 16 GiB pin | 8,060,928 tokens @512K (15.4x) | Scenario delta: r31's TC-33/TC-80 fails and TC-46/61/67 partials are GONE on r35 (TC-68 partial→pass); NEW on r35: TC-51 fail (step-ordering: sent email before observing create_calendar_event result) + TC-35/TC-40 partials. Cold inductor recompile cost ~25 min first boot (JIT warm since).
• Interpretation: perf FLAT on GB10 (their +8.8% prefill gain was a 32K-prefill cell on RTX PRO 6000; our matrix maxes at 8192 prefix — unmeasured here), quality +4 but harness-confounded (dev46→dev65 credit fixes lean up). r35's real value: SwiGLU-clamp-safe MoE split path (the R33/R34 runaway- generation defect does not exist in r35), LMCache fs-eviction accounting, DFlash2 K7 qualified, latest lab line. TC-51 = variance vs regression unresolved (needs an r31 dev65 re-bench or a second r35 run).