# Night shift plan — 2026-09-18

## Context since the 09-13 shift

- Prod (DS4.1 4x tonyd2wild) was restored the morning of 09-14, then stopped
  for the R37 investigation; the R37 typo bisect (A: no linear-b12x, B: prod
  spec K5, C: full prod contract) all **garbled** → cause is in the r37 tree
  code, not the flags. Boot D (engram off) never ran: the 4x cluster went
  dark mid-load (~07:27 UTC 09-14); operator later power-cycled dgx1+2 only
  and deployed `glm53-flash-eugr-2x` (eugr b12x nightly-20260910) on the
  2nodes-dac pair. **This lane is running and must NOT be stopped** until
  the operator says so (explicit instruction 09-17).
- dev66 bench (09-14): eugr 2x = **161/176 (91)**, d0 c1 1,326 pp / 28.1 tg /
  1,458 ms — quality lane verdict unchanged (r28 2x = prefill lane).
- 09-18 morning: the operator stopped the 2x eugr lane and powered the whole
  cluster down while he provisioned the four new nodes (dgx5–8). As of this
  plan there is **nothing serving** and **all 8 nodes are powered on** with
  `sw6_mt` on (prepared for the 8x work, see below).
- R37 lane status: unqualified for DS4.1 (typos) — pending Boot D + lab
  issue. The bug report draft exists:
  `specs/bugs/BUG-2026-09-14-r37-ds41-text-garbling.md`.
- New tooling shipped 09-14: `sm121 runlock list|close` + report stale
  cross-check (commits `2f71763`, `706b7a5`).

## What updated upstream (scanned 09-17)

- **RedHatAI/GLM-5.3-speculator.dspark** (09-16, v1.0): external DSpark K8
  drafter for the FULL GLM-5.3 (zai-org base, GlmMoeDsaForCausalLM). 3-layer
  Qwen3-style backbone (hidden 6144, sliding window 2048), 2.5 GB bf16,
  aux hidden states from target layers 2/20/39/58/75, Markov head (vanilla,
  rank 256) + confidence head, greedy proposal, verifier_accept_k=1, 1M ctx
  native. Card acceptance: tool_call 4.01, math 6.13, HumanEval 4.76 tokens/
  step; MRCR 524K–1M 41.16% / 4.293 avg. Deploy contract (from the card):
  `--speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark",
  "num_speculative_tokens":8,"method":"dspark"}'`. Trained on vllm
  0.28.0 / speculators 0.8.0.dev196 — our r37 tree reports 0.26.1rc0 but
  **has the integration** (grep-verified: DSparkDraftModel method
  resolution in `config/speculative.py:1333/1404`, speculators config
  loader at `transformers_utils/configs/speculators`, draft_model field,
  model registered in `models/registry.py`). Needs `--trust-remote-code`
  (repo carries config.py custom code) — the full recipe already sets it.
  Drafter trained on BF16-target hidden states; we serve MXFP4 — acceptance
  may shift a bit; that's part of what the A/B measures.
- **lil-catalog updated (snapshot bab43db6, 09-15)** — the lab's qualified
  launch manifests. Only change: `Qwen3.8-Flash-Next-NVFP4` dropped
  `limit_per_prompt: image: 1`. Extracts that matter:
  - `DeepSeek-V4-Flash` (rev 60d8d707): lab policy = **MTP k=2** (moe mxfp4,
    probabilistic/standard), GMU **0.91**, seqs **64**, batched tokens 8192 —
    very different from our prod contract (dspark K5, GMU 0.85, seqs 8).
    NOTE: this is V4-Flash, not our V4.1-Flash; the 0731 variant carries the
    DSpark head.
  - `DeepSeek-V4-Flash-Vision-Exp`: dspark K7 + adaptive_verification — the
    lab qualified the vision lane we have an image for.
  - `GLM-5.3-NVFP4` (lab's FULL GLM-5.3, 433 GB): NVFP4 routed experts +
    BF16 MTP expert layer, **MTP k=3 model:target**, B12X attention,
    long_prefill 2048, async_scheduling, FULL_AND_PIECEWISE + custom_ops
    all, CUDA_DEVICE_MAX_CONNECTIONS 32, no GMU pin. Their qualified
    full-model lane — we never ran spec on the full model.
  - `GLM-5.3-Flash-DFlash2-MXFP8` (1.2 GB, kind: draft): MXFP8 DFlash2
    drafter variant, compatible with the NVFP4/Spark Flash models.
- **eugr**: b12x line's latest tag is still `nightly-20260913` (built from
  the lab's **dev/infernal-invocation** — a NEWER lab branch than our r37
  tree 3fb88fdf); non-b12x line pushed `nightly-20260917`. Changelog
  highlights: Qwen3.8-Flash-Next solo recipe with `VLLM_PLE_TABLE_MEMORY=
  disk` (1M+ KV at GMU 0.8), Vision-Exp B12X + DSpark, GLM Flash b12x
  loader.
- **vLLM 0.28.0** (08-26): DeepSeek V4 sparse MLA end-to-end for decode/MTP/
  DSpark (#51538) — candidate upstream fix for our DCP>1 sparse-decode
  crash (512K blocker); DFlash2 loading from speculators format (#53797);
  DSpark confidence-scheduled verification (#47808); tiered KV offload
  with a disk tier; per-request spec-decode metrics (#48915). Our lab tree
  (0.26.1rc0) predates all of this — rebasing is still the lab's call.
- **tonyd2wild** expanded to GLM-5.3-Flash: `ghcr.io/tonyd2wild/
  vllm-glm53-flash:sm121-v11-dflash2` (+ `sm121-v8` base): 5 SM121 fixes
  (NoPE-MLA sparse backend, FlashInfer 0.6.17→0.6.18 FA2 NaN, dep re-pins,
  PDL gating, indexer top-k init) + fp8 KV unlock. Measured TP2 @262K:
  DFlash2 46.9 tg/s (74.1% acceptance, 2.15×), TP4/1M sibling 68.5 tg/s,
  pool 2.62M. A third independent stack for the GLM Flash lane.
- Forum #382120 (EXL3/TR3 kit, 4bpw uncensored): 2.2M-token pool @900K ctx
  on 2x; their host-sync findings (per-layer `torch.unique` gather →
  ~700 ms stalls; fixed-shape no-dedup gather + raised fused temp rows)
  are good context for the lab issues doc.

## Prep done (09-18)

- `RedHatAI/GLM-5.3-speculator.dspark` downloaded to the shared HF cache
  (2.3 GB, snapshot 198b3ced) ✓
- r37-image grep: external-dspark support confirmed ✓
- lil-catalog read + diffed ✓
- git.kilork.org recovered — the ~20 queued local commits pushed ✓

### 8x readiness (done 09-18 daytime, verified live)

- All 8 nodes up (dgx5–8 fresh), `sw6_mt` turned on via the HA relay —
  the 200G fabric is invisible without the switch (operator: the switch
  relay is a separate HA entity, not implied by node power).
- **The "two IPs on the IB interface" question is solved**: every Spark
  has TWO dual-port 200G NICs (`enp1s0` + `enP2p1s0`); port0 of each carries
  a 200.x IP and both are on ONE L2 domain (a single 16-port 200G switch,
  cross-ping verified with sudo ping). Dual-rail 2×200G per node is already
  the current `4nodes-switch` convention (`NCCL_IB_HCA=rocep1s0f0,
  roceP2p1s0f0`). Per-node `ib_ip` = NIC A port0: dgx5=200.11, dgx6=200.10,
  dgx7=200.23, dgx8=200.25 (plus the 100.x/101.x DAC pair on dgx1/2 port1,
  untouched). All 16 switch ports link UP.
- dgx5/6: `/mnt/nas` (nfs4 `nas.lan:/sna`, nconnect=8) validated ✓. dgx7/8:
  **fixed** — the fstab line + `/mnt/nas` were missing; added (sudo
  available, NAS exports `/sna *`), mounted, 65 HF repos visible ✓.
- `8nodes-switch` cluster added to `config.toml` (dgx1–8, ranks 0–7,
  entities `switch.dgxN`, `sw6_mt` blocking + `sw7_fan` non-blocking) with
  `[nccl."8nodes-switch"]` = the dual-rail copy of `4nodes-switch`. All 8
  nodes share one ED25519 host key (cloned Ubuntu image) — same pinned
  `SHA256:aUXh…` as dgx1–4; (security footnote: per-node key regen later).
- `sm121 health --cluster 8nodes-switch`: **8/8 online, all free** ✓.
- Dry-run `serve --cluster 8nodes-switch --fit 8` renders correctly:
  TP8/nnodes 8/ranks 0–7, GID auto-detect on all 8, before_serve hooks ×8,
  MASTER_ADDR=200.15 ✓.
- Images: dgx5 is fully pre-seeded (r37/r36/r35 + both dsv41); dgx6–8 have
  no vLLM image yet — **the launcher syncs them natively at serve**
  (build on `--build-node` + checksum-label sync; no manual
  `docker save|ssh|docker load` — an abandoned manual copy attempt left no
  residue, verified).

## Night program

### P0 — Full GLM-5.3 MXFP4 + DSpark K8 on 4x @131K (operator's pick)
STATUS (09-18 ~13:45 UTC): recipe `glm53-full-mxfp4-4x-dspark8` up on
4nodes-switch, quality gate PASSED (RU 300-word text clean, 6/6 identifier
probe), bench `dspark8` running.

Boot log (6 boots, 12:21 → 13:34):
- Boot 1-2: the draft model's non-MLA DFlash attention (head 64) got the
  target's B12X sparse-MLA `fp8_ds_mla` KV dtype — the target's MLA layer
  rewrites the shared `cache_config.cache_dtype` (mla_attention
  `_canonicalize_sparse_mla_kv_cache_dtype`) and the selector has no backend
  for (head 64, bf16, fp8_ds_mla, non-MLA). Explicit `--kv-cache-dtype
  fp8_e4m3` did NOT help (overridden back to fp8_ds_mla).
- Boot 3: `--kv-cache-dtype-skip-layers sliding_window` (the r37 flag for
  exactly this: skipped layers use auto/bf16 KV; the target's DSA layers are
  not sliding-window so they keep fp8_ds_mla) → attention init passed, but
  the draft SlidingWindowSpec inherited a second pollution: the target's
  `cache_config.skip_page_size_padded` (shared-page padding) < the draft's
  bf16 page → `assert page_size_padded >= unpadded_page_size_bytes` in
  kv_cache_interface.py.
- Fix: `patches/patch_dspark_draft_page.py` (anchor patcher, in-container
  mod via the shared HF cache `sm121-patches/`; the draft spec passes
  `page_size_padded=None` — its KV group is independent and must use its
  own page size). Tested throwaway-container first (check→apply→check).
- Boot 5: KV budget — the 8 GiB pin was 80 MB short of the 8.08 GiB the
  engine needs for one 131072 request + the drafter's KV group → pin raised
  to 9 GiB.
- Boot 6 (13:34): up. KV pool 145,664 tokens @131K (concurrency 1.11x —
  the drafter's KV group eats ~9% of the pin vs the no-spec 159,232),
  PIECEWISE+FULL target graphs + separate dspark graph capture (54 s),
  Application startup complete.

1. [done] Recipe + spec config (K8, method dspark, drafter on NAS
   snapshot 198b3ced; the repo is in the shared HF cache, HF_HUB_OFFLINE).
2. [done] Boot: drafter resolved as Qwen3DSparkModel, graphs captured.
3. [done] Quality gate: typo probe clean (the 09-14 r37 DS4.1 garbling did
   NOT appear here), identifiers 6/6. Note: this recipe's glm47 template
   does not honor chat_template_kwargs thinking:false (the model writes its
   plan into content; reasoning_tokens stays 0) — cosmetic, bench handles it.
4. [done] Bench `dev67-full-glm53-dspark8` (13:43 → 14:31 UTC, dev65+g6be685f0e,
   seed 42, hardmode, c1+c2): **153/176 (87/100, ★★★★)** — within run variance
   of the no-spec anchor 158/176 (−5, same as eugr 161 vs 166). 8 fails, all
   wrong_args/missing_step, no garbling: TC-22/TC-68 invalid JSON, TC-43 empty
   query, TC-45 tool_choice, TC-38 crowded namespace 2/4, TC-51 (safety, the
   same send_email-before-confirm quirk eugr 2x had), TC-80 + TC-87 (★★★★★
   preconditioned update / cursor integrity).
   Perf vs anchor (no-spec 158/176, 934 pp / 19.0 tg / 2,124 ms):
   - d0 c1: **892 pp / 33.2 tg / 2,193 ms — decode 1.75×**, prefill −4.5% (spec
     overhead), TTFT +3% (flat).
   - d4096 c1: 847 pp / 28.4 tg / 6,662 ms; d8192 c1: 863 / 27.3 / 10,979 —
     decode holds 27–28 tg at depth.
   Verdict: the K8 drafter is a real +75% decode win on the full model at 131K
   with quality held (card numbers were BF16-target / MRCR; MXFP4 target +
   131K context land below them, as expected). Report:
   `recipes/glm53-full-mxfp4-4x-dspark8/reports/2026-09-18_134322_dspark8.*`.
5. Teardown: stop, then the P1/P2 sequence.

### P1 — r37 typo verdict (closes the morning's bisect)
DONE (2026-09-18 night): **verdict = the garbling is a lab-tree trait.**
Boot D INFEASIBLE on 4x (OOM incident 14:37 UTC, nodes recovered).
1. **Boot D — INFEASIBLE ON 4X, OOM incident 14:37 UTC.** The r37 recipe
   state has no `--engram-config` (the DS4.1 engram table is part of the
   model weights, not a toggle) and no instanttensor/lazy → the full 510 GB
   loads eagerly: 127.6 GB/node + process > 121.6 GiB → all 4 nodes OOM'd
   simultaneously (journal: NVRM `_memdescAllocInternal` NoMemory burst +
   "Under memory pressure, flushing caches" at 14:37 UTC; ping alive, sshd
   dead = classic thrash). Power-cycled dgx1-4 via HA, back up 15:10 UTC.
   The engram-off hypothesis can only be tested on 8x (510/8 = 64 GB/node
   fits) — P2.3 is the natural test. Morning A/B/C (same tree, engram disk
   mode + lazy load) booted fine — the OOM is specific to the D state.
2. **eugr b12x nightly-20260913 A/B (done 15:45 UTC):** recipe
   `dsv41-flash-eugr-0913-4x` = the measured r37 contract (GMU 0.88, 512K
   DCP1, block 256, B12X trio, K7 greedy/adaptive, engram disk,
   instanttensor/lazy, MXFP8_LM_HEAD=0) on the eugr 0913 image (the lab's
   dev/infernal-invocation, `b40673cd0`, newer than r37's 3fb88fdf tree).
   Result: **GARBLED, same pattern** (300-word RU: ~30 close-vocabulary
   corruptions in 2.4 KB — "абслуживание", "парамитры", "мадель", "тензары",
   "маштабировать", "упровляемой"...; identifiers 6/6 PASS, structure clean).
   Both independent 09-13 lab builds garble, the tonyd2wild prod chain is
   clean, MXFP8-LM-head opted out in all runs → the defect is in code common
   to the lab's DS4.1 line as of 09-13 (`deepseek_v32`/DS4.1 model code,
   b12x MoE, or the engram disk-read path). DS4.1 stays on tonyd2wild prod;
   the lab line stays proven only on the full GLM-5.3 lane (P0 quality gate
   passed there). Bug report updated with the A/B section.
   Note: the launcher's 1800 s readiness wait is too short for a cold NFS
   load of 510 GB at ~130 MB/s (boot = ~40 min); the serve failed the wait
   while the engine kept booting — probes went straight at the API.

### P2 — 8x trials (cables on site; nodes provisioned, cluster configured)
Gate: CLEARED — all 8 nodes up + validated, `8nodes-switch` in config,
health green. The launcher syncs missing images at serve time.
STATUS (22:10 UTC): P2.3 done (DS4.1 8x, 161/176). P2.1 bench killed by
operator (partial, results clear — no restart). P2.4 IN PROGRESS: full
native GLM-5.3 + DSpark K8 on 8x (operator's "whole model" pivot).
1. [done] **8x full GLM-5.3 MXFP4 @512K DCP1** — up 20:12 UTC (boot
   19:19 → 20:12, ~50 min NFS load 376 GB/rank at ~130 MB/s + capture):
   KV pool **972,032 tokens @512K (1.85x)** — 512K works on the full model
   **without DCP** (sidesteps the DCP>1 sparse-decode bug). Quality gate
   PASSED (RU text clean, identifiers 6/6). Bench `full8x-512k` ran to
   ~74/88 (69 pass / 5 fail, incl. the TC-51 family quirk + TC-80) and was
   **killed by the operator mid-TC-82** ("do not restart — results already
   clear"): no clean run, no report package; the partial score
   (≥69/74 ≈ 93+ with 14 remaining) sits between the 4x anchors.
   Note: `ip route get 192.168.88.16` → dev **enP7s7** (a dedicated mgmt NIC,
   not the 200G rails): the NFS model-load (~1 GB/s aggregate) rides the
   mgmt fabric; the RoCE rails stay idle until NCCL collectives start
   (operator saw the mgmt-switch load in the Mikrotik UI with idle
   uplinks — explained by this).
2. **8x lab GLM-5.3-NVFP4 + MTP k3** (manifest policy): 433 GB → 54.1 GiB/
   node; the lab's own qualified full-model config; A/B vs (1).
3. [running] **8x DS4.1-Flash — in-memory engram** (operator decision, code-
   verified). Recipe `dsv41-flash-8x`: drops `DSV41_ENGRAM_DISK`/`LOCAL`/
   `DIR` + the `/engram-local` mount (the baked `engram.py` patch defaults to
   memory mode). Boot log (first attempt 15:57, second 16:10 after fixes):
   - **GMU 0.91 → 0.85**: the lab policy's 0.91 fails the GB10 admission
     (free 108.78 < 110.7 required; ~12.8 GiB baseline reservation); 0.85
     is the GB10-qualified value.
   - **dgx6-8 cache symlink**: `/home/kilork/.cache/huggingface` was a local
     dir there (docker -v auto-created it empty as root) — ranks 5-7 died
     with LocalEntryNotFoundError; fixed with `rmdir + ln -s /mnt/nas/ai/
     huggingface` (dgx1-5 have the symlink natively).
   - **Load = 122 min** (r0, `Model loading took 49.55 GiB and 7338 s`):
     memory mode does a full sequential scan of the two 101.5 GB sparse
     shards per rank (~203 GB reads/rank) at ~20-40 MB/s under 8-way NFS
     contention. One-off cost; the launcher's 1800 s readiness wait is
     irrelevant for cold 8x loads (probe the API manually).
   - Up 18:28 UTC: KV pool **2,584,421 tokens @512K (4.93x)**, engram
     2×11.80 GiB/rank pinned, DSpark K5 graphs captured, ~24.5 GiB KV
     (matches the 103.4 − 38.4 − 25.4 − 12 math).
   - Quality gate PASSED: RU 300-word text clean (no garbling — the prod
     image + in-memory engram is clean, cf. the lab-tree garbling verdict
     in P1), identifiers 6/6. This also settles the "Boot D" question
     from the r37 bisect: engram without the disk-read path is clean on
     this image.
   - Bench `dev68-dsv41-8x` (18:31 → 19:07 UTC): **161/176 (91/100,
     ★★★★★)** vs 4x prod anchor 167/176 (95) — within run variance (same
     eugr-161-vs-166 pattern); fails: TC-51 (the send_email-before-confirm
     family quirk, present in eugr 2x + full-GLM-dspark8 too), TC-68 invalid
     JSON, TC-74 stateful corrections. Perf vs 4x prod (1,642 pp / 47.0 tg /
     1,738 ms @d0 c1): **2,866 pp / 50.2 tg / 986 ms — prefill +75%, TTFT
     −43%, decode 1.07x** (bandwidth-bound, as expected); d8192 c1: 2,870 / 58.4
     / 3,834 (decode rises with depth). Verdict: 8x = the capacity + TTFT
     lane (4.93x concurrency @512K, KV 2.58M tokens); 4x prod stays the
     default for interactive traffic, 8x for high-concurrency/long-context.
Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is
qualified).

4. [running] **8x full GLM-5.3 NATIVE (zai-org) + DSpark K8 @64K** — the
   operator's "whole model" pivot (21:25 UTC): NOT the RedHatAI MXFP4
   quant — the original `zai-org/GLM-5.3` checkpoint (753.3 GB FP8 e4m3
   block-128, 141 shards, on the NAS: blobs 704 GB, refs/main aca966e4 —
   already downloaded, my "not on NAS" read was wrong: `du` on the
   snapshot counted symlinks, not `blobs/`). The architecture is unchanged
   5.2 → 5.3 (operator), so the lab r34 doc (Gilded Gnosis, GLM-5.2 R7:
   65K model limit, seqs 8, batched 2048, GMU 0.97/0.98 on RTX 6000)
   transfers except GMU (the GB10 admission check caps at ~0.85-0.89;
   measured free 108.78 GiB < 0.91 requirement).
   - **Memory**: 753.3/8 = 94.2 GB/node weights; GMU 0.85 → ~6.8 GB KV →
     64K request = 3.45 GB/rank (1.9x), 96K = 5.1 GB (1.3x); **512K does
     not fit** (26.3 GB) — physical limit, hence max-model-len 65536.
   - **Image**: `aidendle94/sparkrun-vllm-glm53-gb10:production-1.2`
     (31.2 GB, Aiden's GLM-5.2/5.3 production stack — the operator's
     "aiden 5.2 image"; verified in-image: DSparkDraftModel speculators
     loader + glm_moe_dsa arch both present; their DFlash2 drafter runs
     without the r37 workarounds → skip-layers flag + draft page patch
     NOT ported). `production-3.75` (the ds4-gb10 DS4 image) is a dead
     end: no DSpark, no glm_moe_dsa (vLLM 0.11.2 base). Aiden's lil
     launcher repo (`local-inference-lab/lil`, Go) + `configs/models/
     _bases.yaml` read: glm family = reasoning glm45 / tool glm47 /
     B12X attention / VLLM_B12X_MOE_FP4_FORCE_A16=1.
   - **Loader (operator call)**: plain safetensors, not InstantTensor.
     `--load-format auto` on the Aiden image = InstantTensor (verified in
     the first boot log: "Loading safetensors using InstantTensor loader",
     704 GB/rank stream at ~130 MB/s). First Aiden boot died 137 at 21:32
     (the launcher's own stale-container cleanup + a hung r37 first boot
     on the same recipe name; no host OOM, journal clean). With explicit
     `--load-format safetensors`: the standard per-shard loader runs
     (141 shards, ~40 s/shard, ~145 MB/s, started 21:54, ETA ~95 min —
     same per-node NFS volume as InstantTensor with one rank/node, but
     stable and no staging hang).
   - Recipe `glm53-full-native-8x-dspark8`: TP8, DCP1, GMU 0.85, 64K,
     seqs 8, batched 2048, kv fp8_e4m3, moe-backend marlin (the Aiden
     baseline spellings), block 256, speculator K8 dspark, no workarounds.
   - After boot: quality gate (RU + identifiers) + bench vs the 4x dspark8
     anchor (153/176, 892 pp / 33.2 tg / 2,193 ms @131K).

### P3 — optional, time permitting
1. **Qwen3.8-Flash-Next-NVFP4** quick trial (1–2 nodes): new family,
   catalog-updated 09-15 (image limit dropped — vision serving);
   MTP k3, GMU 0.94, gdn_decode b12x, block 16, kv bf16, PLE env vars.
2. **tonyd2wild GLM-Flash v11-dflash2** on 2x: their 262K DFlash2 stack vs
   our eugr 2x (1M) — only if the 2x lane becomes free (it must NOT be
   stopped for this).
3. **DS4.1-Flash lab-policy A/B** (MTP k2, GMU 0.91, seqs 64): requires
   downloading `deepseek-ai/DeepSeek-V4-Flash` @60d8d70 (~510 GB) — a
   multi-hour NAS download; start early if the operator wants it.
4. **SGLang DS4.1 4x** (operator-added 09-18 evening, aidenle.com recipe
   `deepseek-v4-1-flash-sglang-4x-dgx-spark`): the SGLang team's own
   DeepSeek-V4.1 image `lmsysorg/sglang:dev-dsv41` + the `aidendle94/
   sparkrun-sglang-dsv41-gb10:production-1.0` overlay (MIT, ~1.5k lines of
   import hooks: Engram rows from node-local NVMe INSIDE the CUDA graph,
   b12x SM12x kernels for the FP8 dense layers, two SM120 fixes, allocator
   flush, one-shot RDMA collectives). TP4 + EP4, dsv4 attention backend,
   FlashInfer MXFP4 MoE runner, DSpark k5 (block size 5, checkpoint's own
   draft layers). Aiden's fleet has run it as production since 09-17
   (replaced their vLLM stack): prefill +30-45% vs their vLLM lane, decode
   level, +23% KV pool. Prereqs we already have: the /var/tmp/engram-local
   sparse copies on dgx1-4 (reused as-is), weights on the NAS. New: pull
   the 34 GB image to the 4 nodes, recipe with the SGLang engine args
   (`sglang serve` entrypoint — a new entrypoint for us: the launcher's
   SGLang flag set is for the stock engine; this lane needs their overlay's
   dsv4 backend + EP4), earlyoom tuning, root on the nodes (we have it).
   The bench A/B target: our vLLM prod (167/176, 1,642 pp / 47.0 tg /
   1,738 ms @d0 c1).

### P4 — bookkeeping
- File the lab issues (garbling bisect + 3 engine bugs from 09-13).
- End-state (operator, 22:15 UTC): **power the cluster OFF when the tests
  are done** — energy saving, no 8x lane left serving.

### P5 — r37 rebuild under the latest pins (operator, 21:55 UTC)
- "r37 пора бы уже пересобрать под последние реалии": re-cut the jovian DS4.1
  image against the current upstream (eugr b12x master past f1c4e9dd,
  vLLM dev past c687594b, flashinfer, LMCache) — a new rev (r38?) from a
  fresh source.lock. Not blocking tonight (the full-native lane runs on
  the Aiden image; P1's garbling verdict is already in); do it after the
  night's results, then re-baseline the anchor lane before any adoption.

## Guardrails

- **Nothing is serving right now** (the 2x eugr lane was stopped 09-18);
  all 8 nodes + `sw6_mt` are ON as prep. First serve of the night is a
  clean slate.
- Typo probe gates any DS4.1/r37 promotion (the 09-14 lesson).
- `sm121 serve --keep` for first/unknown boots; short-timeout polling.
- Image distribution is the launcher's job at serve time (`--build-node`
  + checksum-label sync) — no manual `docker save|ssh|docker load`.
- Kernel stays 6.17.0-1031 (7.0.0-1019 breaks NCCL/RoCE — never upgrade).

## Open questions for the operator

1. Which blocks matter tonight: P0 (DSpark K8 on the full model, 4x) is
   the stated wish; 8x (P2) is fully prepped and ready to go on your word.
2. After the night: power everything down, or leave an 8x lane serving?