# Night shift 2026-09-12 (local) — maximum shift: eugr 2x, sparkring re-run, Aiden DCP2 A/B

Operator plan (before start): "максимальная смена" — first update-check of all
sources, then 2x window (eugr + sparkring), morning: Aiden image re-check,
r31 back to production, dgx5 power-off consideration.

## Phase 0 — source update-check (22:15 UTC)

- **Aiden published `production-1.1` at 16:54 UTC** (Docker Hub) — the event
  of the shift. Pulled to all 4 nodes + dgx5 (23.4 GB, digest ede55d46).
- vLLM `main` is 338 commits ahead of our pin `e47aa780` (09-10, "[Model]
  Support DeepSeek-V4.1-Flash", still not on main) and DSV4 fixes started
  flowing into main (2 today: indexer gather kernel, dflash attention
  padding) → next-rebase campaign material.
- FlashInfer: today's `feat(sm120): runtime KV row stride + canonical 528B
  GLM53_NOPE rows + NaN-safe masked gather` — candidate for the GLM lanes.
- DeepGEMM: new "Public Release 26/09" (09-10).
- SparkRing upstream: new `r33-arm64-c8646b0` tag (R33 TP4/DCP4 SparkCache
  profile) + the memory-guard rebuild we needed tonight:
  `glm53-shared-8b32ef2` = digest **4774a7c5** (confirmed via ghcr digest
  lookup; `glm53-shared-74275757` = f25cdb6 = our current pin).
- tonyd2wild: new EXL3 TP4 context lane (3.3M-token KV pool on four Sparks),
  now their default serving config — note for the next DSV4.1 campaign.
- eugr: no fresh nightly (20260910 still the newest, already on the nodes).
- LMCache: quiet since 09-04.

## eugr 2x — nightly-20260910 (report `2026-09-11_223443_eugr-2x-0910`)

- 4x DSV4.1 stopped at 22:18 UTC; eugr 2x on dgx1+dgx2.
- **166/176 = 94/100** (81 pass / 4 partial TC-51/57/62/63 / 3 fail TC-45/68/80)
  vs r28 2x baseline **158/176 (90)** — their b12x + humming-MTP + 1M-ctx
  stack is +8 points on 2 nodes.
- Perf d0 c1 (pp/tg/TTFT ms): **1,198 / 29.0 / 2,113** vs r28 1,475 / 27.2 /
  1,362 — slower prefill and a 55% higher TTFT (1M-ctx prefill buffer, the
  same effect we measured on DSV4.1 at 1M).
- ⚠ tool-eval-bench drifted to **v2.6.1.dev65+g6be685f0e** this shift (all
  earlier runs: dev46+g9d412464a) — cross-run comparisons spanning the
  drift are approximate.

## sparkring 2x re-run on the memory-guard image (report `2026-09-11_235045_...`)

- Pulled `ghcr.io/fujitsupolycom/sparkring:glm53-shared-8b32ef2`
  (= 4774a7c5, the 2 GB memory-guard rebuild) to dgx1+dgx2.
- **Boot-1 bug (fixed in recipe, commit 13720ae)**: the new image's b12x
  loader JIT-compiles its native module at first boot and the linker cannot
  find `libcudart` (the old image prebaked it). Verified the fix in a
  scratch container: `LIBRARY_PATH=/usr/local/cuda/lib64` (+ `CUDA_HOME`),
  plus a persistent `B12X_CACHE_DIR` mount so the compile survives reboots.
- **Reliability verdict: FAIL, again.** The engine died at scenario ~62
  (TC-62 ran 123 s, then all 26 remaining scenarios infra-failed:
  timeout/connection/5xx). 112/124, 70.5% completion — the "90/100" header
  is a partial score, not comparable to full runs. No system OOM (114 GB
  free at inspection) — the death is container-internal (their
  liveness/memory guard, per the 09-08 forensics). The 2 GB guard fix did
  **not** fix the sustained-load death.

## Aiden production-1.1 — DCP2 A/B (report `2026-09-12_003640_aiden11-dcp2-0912`)

Image internals (verified on dgx5): vllm 0.28.1rc1.dev388+g8a728663c (base
`vllm/vllm-openai:nightly-8a728663`, not our e47aa780 pin) + b12x 1.3.0 +
FI 0.7.0rc1 + torch 2.13.0+cu130; the **24-file DCP2 patch set is baked**
(manifest at `/opt/dsv41-dcp2-mounts.txt`) + FI source with
`trtllm_dcp_alltoall.cu` / `cake_dcp.py`. DCP env defaults are OFF
(`DSV41_DCP_STAGE=3`, `DCP_COMBINE=ag_rs`, `DCP_PREFILL_GATHER=0`,
`ENGRAM_MMAP/PREFETCH=0`); CLI: `--decode-context-parallel-size 2`
(`-dcp`), `--dcp-comm-backend {ag_rs|a2a}`.

Recipe `dsv41-flash-4x-aiden11` = their DCP2 core (TP4 + dcp=2 + a2a + 8 GiB
pinned KV, their validated 300K row) on our serving config (DSpark k=5
exact capture, SEQS 8, thinking+max, engram-local, deepseek-v4-flash alias).

- Boot: clean, `Worker_TP0_DCP0`, weights 48/48 in 92 s (warm NFS),
  "DSA indexer: DCP2 with compress_ratio=2 (state-sharded K cache)",
  CUDA graphs 11 s, **KV 2,518,386 tokens** @ 300K (better than their
  claimed 1.89M), 8.39× 300K concurrency.
- **Result: 158/176 = 90/100** (74 pass / 10 partial / 4 fail TC-43/51/68/84)
  vs our b12x baseline **167/176 (95)** @ 512K.
- Perf d0 c1 (pp/tg/TTFT ms): **1,517 / 41.8 / 2,446** vs ours
  **1,642 / 47.0 / 1,738** — we win prefill, decode, and TTFT.
- **Verdict: the DCP2 A/B is ours to keep.** DCP2 buys KV capacity
  (2.5M tokens) / high-concurrency production, not quality or single-stream
  speed. TC-43/84 (fixed by our b12x numerics) fail on their stack again.
- Option left open: the 24-file DCP2 set is extractable from the image
  (`docker cp` of the manifest's targets) if the operator ever wants DCP2
  on our clone-at-pin tree — no need to wait for Aiden's file dump.

## End state

- **Production: `glm53-flash-r31-4x-dcp4`** (4nodes-switch) — restarted as
  morning prod per operator plan.
- DSV4.1 lane fully staged: `dsv41-flash-4x-b12x` recipe + images on all
  nodes + engram-local copies; Aiden 1.1 image on all 4 nodes for re-runs.
- dgx5: powered off (nothing pending on it; power on via HA when a build
  is needed).
- Gist (published daytime): https://gist.github.com/kilork/1ba224bfe28779c571d096cf954fff31
  — base recipe card + live b12x overlay instructions in the README.

## Morning follow-ups (ranked)

1. Aiden `production-1.2` on Hub (DCP2-1.2 with the launcher env values) —
   re-check; the 1.1 A/B already answers the direction.
2. vLLM rebase campaign: DSV4.1 support is landing in `main` (338 commits
   ahead of our pin) — re-pin + re-patch when the branch stabilizes.
3. FlashInfer sm120 update (today's GLM53_NOPE rows commit) for the GLM lanes.
4. SparkRing `r33-arm64-c8646b0` (R33 profile) — if the reliability story
   ever changes.
5. tonyd2wild EXL3 TP4 lane (3.3M KV pool) — next DSV4.1 quant option.
6. gpuflip.py GB10 slow-state probe on our 4 nodes (read-only, not done).