# Night shift 2026-09-12 (local) — maximum shift: eugr 2x, sparkring re-run, Aiden DCP2 A/B

Operator plan (before start): "максимальная смена" — first update-check of all
sources, then 2x window (eugr + sparkring), morning: Aiden image re-check,
r31 back to production, dgx5 power-off consideration.

## Phase 0 — source update-check (22:15 UTC)

- **Aiden published `production-1.1` at 16:54 UTC** (Docker Hub) — the event
  of the shift. Pulled to all 4 nodes + dgx5 (23.4 GB, digest ede55d46).
- vLLM `main` is 338 commits ahead of our pin `e47aa780` (09-10, "[Model]
  Support DeepSeek-V4.1-Flash", still not on main) and DSV4 fixes started
  flowing into main (2 today: indexer gather kernel, dflash attention
  padding) → next-rebase campaign material.
- FlashInfer: today's `feat(sm120): runtime KV row stride + canonical 528B
  GLM53_NOPE rows + NaN-safe masked gather` — candidate for the GLM lanes.
- DeepGEMM: new "Public Release 26/09" (09-10).
- SparkRing upstream: new `r33-arm64-c8646b0` tag (R33 TP4/DCP4 SparkCache
  profile) + the memory-guard rebuild we needed tonight:
  `glm53-shared-8b32ef2` = digest **4774a7c5** (confirmed via ghcr digest
  lookup; `glm53-shared-74275757` = f25cdb6 = our current pin).
- tonyd2wild: new EXL3 TP4 context lane (3.3M-token KV pool on four Sparks),
  now their default serving config — note for the next DSV4.1 campaign.
- eugr: no fresh nightly (20260910 still the newest, already on the nodes).
- LMCache: quiet since 09-04.

## eugr 2x — nightly-20260910 (report `2026-09-11_223443_eugr-2x-0910`)

- 4x DSV4.1 stopped at 22:18 UTC; eugr 2x on dgx1+dgx2.
- **166/176 = 94/100** (81 pass / 4 partial TC-51/57/62/63 / 3 fail TC-45/68/80)
  vs r28 2x baseline **158/176 (90)** — their b12x + humming-MTP + 1M-ctx
  stack is +8 points on 2 nodes.
- Perf d0 c1 (pp/tg/TTFT ms): **1,198 / 29.0 / 2,113** vs r28 1,475 / 27.2 /
  1,362 — slower prefill and a 55% higher TTFT (1M-ctx prefill buffer, the
  same effect we measured on DSV4.1 at 1M).
- ⚠ tool-eval-bench drifted to **v2.6.1.dev65+g6be685f0e** this shift (all
  earlier runs: dev46+g9d412464a) — cross-run comparisons spanning the
  drift are approximate.

## sparkring 2x re-run on the memory-guard image (report `2026-09-11_235045_...`)

- Pulled `ghcr.io/fujitsupolycom/sparkring:glm53-shared-8b32ef2`
  (= 4774a7c5, the 2 GB memory-guard rebuild) to dgx1+dgx2.
- **Boot-1 bug (fixed in recipe, commit 13720ae)**: the new image's b12x
  loader JIT-compiles its native module at first boot and the linker cannot
  find `libcudart` (the old image prebaked it). Verified the fix in a
  scratch container: `LIBRARY_PATH=/usr/local/cuda/lib64` (+ `CUDA_HOME`),
  plus a persistent `B12X_CACHE_DIR` mount so the compile survives reboots.
- **Reliability verdict: FAIL, again.** The engine died at scenario ~62
  (TC-62 ran 123 s, then all 26 remaining scenarios infra-failed:
  timeout/connection/5xx). 112/124, 70.5% completion — the "90/100" header
  is a partial score, not comparable to full runs. No system OOM (114 GB
  free at inspection) — the death is container-internal (their
  liveness/memory guard, per the 09-08 forensics). The 2 GB guard fix did
  **not** fix the sustained-load death.

## Aiden production-1.1 — DCP2 A/B (report `2026-09-12_003640_aiden11-dcp2-0912`)

Image internals (verified on dgx5): vllm 0.28.1rc1.dev388+g8a728663c (base
`vllm/vllm-openai:nightly-8a728663`, not our e47aa780 pin) + b12x 1.3.0 +
FI 0.7.0rc1 + torch 2.13.0+cu130; the **24-file DCP2 patch set is baked**
(manifest at `/opt/dsv41-dcp2-mounts.txt`) + FI source with
`trtllm_dcp_alltoall.cu` / `cake_dcp.py`. DCP env defaults are OFF
(`DSV41_DCP_STAGE=3`, `DCP_COMBINE=ag_rs`, `DCP_PREFILL_GATHER=0`,
`ENGRAM_MMAP/PREFETCH=0`); CLI: `--decode-context-parallel-size 2`
(`-dcp`), `--dcp-comm-backend {ag_rs|a2a}`.

Recipe `dsv41-flash-4x-aiden11` = their DCP2 core (TP4 + dcp=2 + a2a + 8 GiB
pinned KV, their validated 300K row) on our serving config (DSpark k=5
exact capture, SEQS 8, thinking+max, engram-local, deepseek-v4-flash alias).

- Boot: clean, `Worker_TP0_DCP0`, weights 48/48 in 92 s (warm NFS),
  "DSA indexer: DCP2 with compress_ratio=2 (state-sharded K cache)",
  CUDA graphs 11 s, **KV 2,518,386 tokens** @ 300K (better than their
  claimed 1.89M), 8.39× 300K concurrency.
- **Result: 158/176 = 90/100** (74 pass / 10 partial / 4 fail TC-43/51/68/84)
  vs our b12x baseline **167/176 (95)** @ 512K.
- Perf d0 c1 (pp/tg/TTFT ms): **1,517 / 41.8 / 2,446** vs ours
  **1,642 / 47.0 / 1,738** — we win prefill, decode, and TTFT.
- **Verdict: the DCP2 A/B is ours to keep.** DCP2 buys KV capacity
  (2.5M tokens) / high-concurrency production, not quality or single-stream
  speed. TC-43/84 (fixed by our b12x numerics) fail on their stack again.
- Option left open: the 24-file DCP2 set is extractable from the image
  (`docker cp` of the manifest's targets) if the operator ever wants DCP2
  on our clone-at-pin tree — no need to wait for Aiden's file dump.

## End state

- **Production: `glm53-flash-r31-4x-dcp4`** (4nodes-switch) — restarted as
  morning prod per operator plan.
- DSV4.1 lane fully staged: `dsv41-flash-4x-b12x` recipe + images on all
  nodes + engram-local copies; Aiden 1.1 image on all 4 nodes for re-runs.
- dgx5: powered off (nothing pending on it; power on via HA when a build
  is needed).
- Gist (published daytime): https://gist.github.com/kilork/1ba224bfe28779c571d096cf954fff31
  — base recipe card + live b12x overlay instructions in the README.

## Production state after the daytime window

- **Production: `glm53-flash-r35-4x-dcp4`** (4nodes-switch, promoted 09-12
  per operator decision — r35 stays; r31 recipe remains as the rollback
  lane, image tagged on all nodes).
- Follow-ups added: (a) r31 re-bench on dev65 at the next 4x window for a
  harness-clean delta (and to settle TC-51 variance-vs-regression);
  (b) 32K-prefill probe if the split-MoE prefill gain on GB10 ever matters
  (outside the current bench matrix).

## Morning follow-ups (ranked)

1. Aiden `production-1.2` on Hub (DCP2-1.2 with the launcher env values) —
   re-check; the 1.1 A/B already answers the direction.
2. vLLM rebase campaign: DSV4.1 support is landing in `main` (338 commits
   ahead of our pin) — re-pin + re-patch when the branch stabilizes.
3. FlashInfer sm120 update (today's GLM53_NOPE rows commit) for the GLM lanes.
4. SparkRing `r33-arm64-c8646b0` (R33 profile) — if the reliability story
   ever changes.
5. tonyd2wild EXL3 TP4 lane (3.3M KV pool) — next DSV4.1 quant option.
6. gpuflip.py GB10 slow-state probe on our 4 nodes (read-only, not done).

## Daytime follow-up: R35 adoption + A/B (09-12, operator window)

- Adopted **R35** (latest lab rev, qualified 2026-09-11, `swiglu-reviewed-
  composition-r35.source.lock`): full 3-fork cut — vLLM `de982a50` (tree
  6d5cc349), b12x `98086604` (tree d963f030), LMCache `29bc5a2e` (tree
  5a88a1ea). All pins verified fetchable + tree-matched; build-dep set
  r31↔r35 identical (no fetch-deps resync). `profiles/glm53-r35.env` +
  recipe `glm53-flash-r35-4x-dcp4`.
- Image built on dgx5 (71 m 18 s, checksum 8bb87348), boot gates green
  (vllm string 0.1.dev0+glm53.r35.de982a50, b12x/lmcache/flashkda OK);
  pre-transferred dgx5→dgx1 (docker save|ssh, ~11 min) before the window.
- A/B (4nodes-switch, r31 stopped, dev65 harness, seed 42, hardmode, c1/c2):
  | | r31 (09-10, dev46) | **r35 (09-12, dev65)** |
  |---|---|---|
  | score | 159/176 (90) | **163/176 (93)** |
  | pp/tg/TTFT d0 c1 (ms) | 1,519 / 50.0 / 1,524 | 1,554 / 49.6 / 1,544 |
  | KV | 16 GiB pin | 8,060,928 tokens @512K (15.4x) |
  Scenario delta: r31's TC-33/TC-80 fails and TC-46/61/67 partials are GONE
  on r35 (TC-68 partial→pass); NEW on r35: TC-51 fail (step-ordering: sent
  email before observing create_calendar_event result) + TC-35/TC-40
  partials. Cold inductor recompile cost ~25 min first boot (JIT warm since).
- Interpretation: perf FLAT on GB10 (their +8.8% prefill gain was a 32K-prefill
  cell on RTX PRO 6000; our matrix maxes at 8192 prefix — unmeasured here),
  quality +4 but harness-confounded (dev46→dev65 credit fixes lean up).
  r35's real value: SwiGLU-clamp-safe MoE split path (the R33/R34 runaway-
  generation defect does not exist in r35), LMCache fs-eviction accounting,
  DFlash2 K7 qualified, latest lab line. TC-51 = variance vs regression
  unresolved (needs an r31 dev65 re-bench or a second r35 run).

## Night shift (22:00–00:10 UTC+2 local, unattended ops + DFlash2 A/B)

### DFlash2 4x A/B — WINNER on latency, +2 points

Recipe `glm53-flash-r35-4x-dflash2` (clone of the r35 MTP3 prod, ONLY the
speculative-config swapped: MTP3 → `{"method":"dflash","model":"...DFlash2-MXFP8",
"num_speculative_tokens":7,"kv_cache_dtype":"auto"}`). No build, no download:
the r35 image ships the DFlash2 machinery (DFlash2DraftModel, dedicated
dflash_attn graph-safe split-KV backend, FlashInferCutlassMxfp8LinearKernel
for the fused draft K/V projection) and the 1.2 GB MXFP8 draft was already on
the shared NAS.

Boot gates green: `Resolved architecture: DFlash2DraftModel`, method=dflash
K7, and — the 08-29 blocker is DEAD — the r35 tree pads the mamba page size
(+0.70%) so draft/target page geometry unifies instead of the old
`4096 % 660 != 0` abort. KV: 6,333,399 tokens @512K (12.08x) vs MTP3's
8,060,928 — the draft's 5 sliding-attention layers + BF16 draft pages eat part
of the 16 GiB pin.

| | r35 MTP3 (prod, 11:31) | **r35 DFlash2 K7 (22:20)** |
|---|---|---|
| score (dev65, seed 42) | 163/176 (93) | **165/176 (94)** |
| pp / tg / TTFT, d0 c1 | 1,554 / 49.6 / 1,544 ms | **1,977 / 47.3 / 1,071 ms** |
| pp / tg, d0 c2 | 1,489 / 53.0 | 1,475 / **55.8** |
| KV | 8.06M tokens | 6.33M tokens |

Same harness both runs (clean A/B). Prefill **+27%**, TTFT **−31%**,
decode −4.6% @c1 / +5.3% @c2, score +2 (TC-40/50/63 now pass; new partials
TC-60 + one more; shared fails TC-43/45/51/68). The run package (manifest +
recipe copy, the 081a0e6 feature) landed on its first live run; found and
fixed a utc_now_iso bug in the same commit (civil_date got seconds, produced
year 4900778).

Verdict (operator, 09-12 night): the DFlash2 lane is the better 4x config on
this bench — but the operator keeps r35 MTP3 as prod for now (the day's
decision) and logs DFlash2 as the 4x candidate: +2 points, big TTFT/prefill
gain, −22% KV (6.3M vs 8.06M — matters for the 512K concurrency story).
Restore path: `sm121 stop + serve` the MTP3 recipe.

### DFlash2 2x Spark — BLOCKED on the engine side (definitive)

`glm53-flash-r35-2x-dflash2` (r31-2x shape, r35 image, NVFP4-Spark target,
same draft). First attempt (bare TP2, no DCP): both ranks abort 5 min into
boot with `ValueError: GLM MLA parent-page stride must be an exact number of
C4 pages` — the r35-tree successor of the 08-29 page-unification blocker.
Second attempt (+DCP2, interleave 2): `ValueError: GLM C4 DCP requires
cp_kv_cache_interleave_size to be divisible by 4, got 2`. The interleave must
be a multiple of 4 → the only working DFlash2 shapes are the lab's certified
TP4/DCP1 and TP4/DCP4 (4x only). This matches the lab's qualification
exactly (their 2x Spark profile is MTP3-only). **2x stays MTP3; DFlash2 is
a 4x-only lane on this tree.** The recipe is kept as a regression canary
(BLOCKED notes in the file).

Ops note: sm121's `docker run` carries `--rm` by default — a crashed rank-0
container vanished with its logs. `sm121 serve --keep` suppresses `--rm`
and was used for the second attempt (logs survived, root cause captured).

### Power-off (operator instruction)

Cluster 4nodes-switch powered off via HA after the A/Bs (prod state: down
until the next serve; MTP3 r35 = prod of record, DFlash2 = candidate).

## Day follow-up 09-13: DeepSeek-V4.1 lab r36 — built, works, LOSES on GB10

The lab published their FIRST DS4.1 lock (09-12): image
`jovian-judgement-community-20260912-r36` (vLLM 202a11a98eb / JJ b40673cd +
#738/#740, b12x 323107ff with DS4.1 native kernels, LMCache 29bc5a2e,
FlashKDA 3b225bf2, DeepGEMM 8b1392b9, FI 0.6.17, + liburing 2.5 for the
Engram disk-mode io_uring reads). DS4.1 profile: TP4/DCP1, native DSpark K7
greedy + standard rejection + adaptive verification (draft inside the
checkpoint), B12X attn/MoE/linear, 131K ctx, 4 seqs / 4096 batched, block
256, FULL_AND_PIECEWISE ≤128, Engram RAM(188.83 GiB)/disk. Their RTX quartet
numbers: 32K prefill ~14K t/s, C1 244–261 t/s.

We built the same source composition for GB10 (the r35 build dir + new
profile `jovian-r36-ds41`, docker-images/glm53-jovian-r27):

- Build 1: failed at our final gate — the b12x marker
  `B12X_PCIE_ONESHOT_PDL` no longer exists in b12x 323107ff (gate updated to
  `_PREPARED_ONESHOT_LAUNCHERS`, same file).
- Build 2: image built, but first serve hard-errored at boot:
  `RuntimeError: io_uring PLE support is unavailable: install liburing
  development headers and pkg-config, then rebuild the b12x native loader`.
  The b12x loader is C sources (`_ple_reader.c` et al.) compiled at RUNTIME
  by `b12x/loader/_native.py` (cc, `-Werror`, liburing optional via
  pkg-config probe) — the image needs `liburing-dev` (headers), not just the
  `liburing2` runtime lib. Build 3 (13m47s) with both: OK.
- Boot gates on 4nodes-switch (warm cache transfers ~4 min/node, label
  49b368a0): DSpark K7 `SpeculativeConfig(method='dspark', num_spec_tokens=7)`,
  `EngramConfig(table_memory='disk')` (no io_uring error), KV 3,377,679
  tokens (25.77x @131K), `Application startup complete`, smoke "четыре" ✅.
  One engine warning: "torch.compile is turned on, but the model ... does
  not support it" — the DS4.1 model class lacks `@support_torch_compile` in
  the r36 tree (true for their RTX runs too; piecewise cudagraphs, not
  inductor fusion).

A/B vs the current dsv41 prod (eugr image, DSpark K5 probabilistic, same
dev65 harness/seed, 4nodes-switch):

| | dsv41 prod (eugr) | **lab r36 (this run)** |
|---|---|---|
| score | 167/176 (95) | **158/176 (90)** |
| pp / tg / TTFT @d0 c1 | 1,642 / 47.0 / 1,738 ms | **401 / 33.7 / 5,061 ms** |
| tg @d4096 c2 | (b12x lane 1,642/47 anchor) | 12.0 |
| KV | 512K pin | 3.38M @131K |

Verdict: **r36 is a clear loser on GB10** — prefill 4.1x slower, TTFT 3x
worse, decode −28%, quality −9. Prime suspect for the prefill collapse: the
Engram PLE reader's io_uring row reads over our NFS (their RTX runs use a
local NVMe checkpoint; 510 GB won't fit our 132 GB local free space),
compounded by RTX-tuned scheduler/compile settings (131K ctx, 4 seqs, block
256, DCP1). The lab's stack is optimized for the RTX PRO 6000 (1.8 TB/s
GDDR7); on GB10 the eugr lane stays production. Follow-ups (in order of
value): (1) diagnose the Engram read path (log PLE read latency; try the
r36 stack with our prod scheduler flags — 512K, DCP4, block 128, seqs 8 —
to separate storage from tuning); (2) wait for the lab's sm_121 port if
they ever tune for Spark; (3) the built image `jovian-r36-ds41:local`
(label 49b6...) + recipe `dsv41-flash-jovian-r36-4x` stay in the repo as a
known-working (losing) lane.

### Day follow-up 2 (09-13): Engram on local NVMe — the NFS suspect CONFIRMED

Operator: "use engram equivalently from NVMe — NAS is a guaranteed loss."
The eugr/aiden dsv41 lanes already stage per-rank sparse Engram copies on
node NVMe (`/var/tmp/engram-local/DeepSeek-V4.1-Flash`, 48 GB real per node —
two full-size 101.5 GB files, only the rank's ceil-row TP shard materialized,
verified per-node row starts 0 / 96,000,564 / 192,001,740 / 288,003,654).
The r36 engine's io_uring PLE reader reads (file, offset+rows) from the model
dir — so it works unchanged against those files: same filename (the engine
looks shards up by name via the index.json), per-node individual content.

**Launcher feature: `composed_model.local_files`** (commit eb63d88) — the
standard composed-dir mechanism extended: entries are HARDLINKED (not copied;
sparse stays sparse, zero disk) from a per-node host path into the per-node
/tmp composed dir, after the repo symlink pass (local wins at the same name).
Recipe `dsv41-flash-jovian-r36-4x` now uses `[composed_model]`: whole DSV4.1
snapshot symlinked into the HF cache + the two Engram shards hardlinked from
the eugr lane's sparse copies. (A hand-rolled ~/models farm first failed in
the container — host-side /mnt/nas symlink targets dangle inside the
container; composed-dir symlinks use container-side paths by design, and
hardlinks are path-independent — the failure is the feature validation.)

**Result (4nodes-switch, dev65/seed 42, vs NFS run 158/176 and prod 167/176):**

| | prod (eugr K5) | r36 + Engram NFS | **r36 + Engram NVMe** |
|---|---|---|---|
| score | 167/176 (95) | 158/176 (90) | **149/176 (85)** |
| pp / tg / TTFT @d0 c1 | 1,642 / 47.0 / 1,738 | 401 / 33.7 / 5,061 | **3,654 / 43.9 / 826** |
| pp / tg @d4096 c2 | — | 12.0 tg | 1,711 / 34.9 |
| KV | 512K pin | 3.38M @131K | 3.34M @131K |

**The storage hypothesis is CONFIRMED**: prefill 401 → 3,654 tok/s (9.1x,
now 2.2x PROD), TTFT 5,061 → 826 ms (6.1x, 2.1x better than prod),
d4096c2 decode 12.0 → 34.9 tg. The io_uring row reads over NFS were the
entire prefill collapse — with the table on NVMe the r36 stack
(K7 DSpark + B12X + V2 runner) is the fastest prefill lane we have.

Quality: 149 vs 158 (NFS) is run variance, not storage — the 11 differing
scenarios are tool-discipline wobbles (skipped steps, empty query, extra
calls; two improved: TC-35, TC-50), plus 2 new safety warnings on
long-horizon P scenarios (TC-82/85). The r36-vs-prod gap (−9…−18) is the
engine-stack delta (K7 greedy+adaptive vs K5 probabilistic), present in both
r36 runs. r36 + local Engram is now a legitimate CANDIDATE lane: best
prefill/TTFT we have, −18 quality vs prod. Next levers (if pursued):
r36 with our prod scheduler flags (512K/DCP4/block 128/seqs 8) to separate
stack from tuning; or K7 speculative-config tuning.

### Day follow-up 3 (09-13): r36 @512K (our floor) + forum 381350 post #521

Operator: "128K is not usable — we need ≥512K." Recipe: `--max-model-len`
524288 (rest of the lab profile unchanged: DCP1, 4 seqs, 4096 batched,
block 256, GMU 0.88, B12X, DSpark K7). Boot green, KV pool GREW to
9,616,957 tokens (18.34x concurrency @512K) — the larger max-len changes
the engine's pool computation (vs 3.34M @131K), a free win.

**dev65 (seed 42, same bench):**

| lane (4x) | pp / tg / TTFT @d0c1 | score |
|---|---|---|
| dsv41 prod (eugr K5, 512K) | 1,642 / 47.0 / 1,738 | 167/176 (95) |
| r36 + Engram NFS (131K) | 401 / 33.7 / 5,061 | 158/176 |
| r36 + Engram NVMe (131K) | 3,654 / 43.9 / 826 | 149/176 |
| **r36 + Engram NVMe (512K)** | **3,774 / 45.7 / 822** | **154/176 (88)** |

d4096c2: 2,554 pp / 58.8 tg (vs 1,711/34.9 at 131K — the bigger pool
shows under load). Quality: 154 — the recurring failures are the same
long-horizon cluster across every r36 run (TC-82/85 fail-or-partial every
time, TC-60/74 flapping) — systematic K7-stack behavior on multi-turn
P scenarios, not storage, not variance.

**Verdict: r36 + local Engram @512K is a serious candidate** — best
prefill (2.3x prod) and TTFT (2.1x better) in the fleet, decode within 3%
of prod, −13 quality (the K7 stack's long-horizon gap). The storage story
is settled: Engram must live on node-local NVMe (the composed_model
local_files mechanism + the eugr lane's per-rank sparse copies).

**Forum 381350 (#521, tea24864, 09-13, verified): `--load-format`
matters.** On 4x Spark GB10 (same fleet), GLM-5.3-Flash 320B/18B
NVFP4-Spark, TP4: single-stream prefill ~2,950 tok/s with
`--load-format instanttensor` vs ~1,900 with `--load-format b12x`
(−35%, flat ratio 0.63–0.65 across 8k–128k ctx — a constant per-token
cost; no memory difference). The b12x loader targets 2x-Spark memory
pressure (O_DIRECT into CPU-addressable CUDA storage) — irrelevant at TP4,
cost remains. ALSO: the b12x loader cannot load a separate DFlash2 drafter
(`checkpoint destination was not allocated as a weight:
candidate_selector.predecessor_codebook`). All our lanes use
instanttensor or the default loader (dsv41 prod has no --load-format) —
no action needed, but the b12x loader is now a known anti-pattern at TP4.

**320B/18B verified from the checkpoint itself**
(local-inference-lab/GLM-5.3-Flash-NVFP4-Spark config.json): 45 layers,
hidden 4096, 288 routed experts + 1 shared, top-k 8, moe_inter 2048 →
~315–320B total / ~18–19B active. The thread title is real.