# Night shift 2026-09-27 (UTC)

Plan (user, carried from 09-26): close the DeepSeek perf gap by making weight/
engram I/O **node-local NVMe** (the way the published references do), EXL3
first, then Aiden SGLang engram staging.

## EXL3 3bpw 2x — weight localization (dgx1+dgx2)

Baseline: 13.3 t/s decode — weights streamed from NAS on every forward.
Reference (coolbho3k, same kit rev 878e0ee): weights+engrams on each host's
local NVMe, ~30.8 t/s.

Staging of `models--coolbho3k--DeepSeek-V4.1-Flash-EXL3-3bpw`
(+ DSpark drafter) to `/home/kilork/hf-local/hub`, page15 pack intentionally
**not** staged (optional +1.4% decode per the release notes; 195 G/node).

The staging itself turned into a six-generation saga — every generation of my
mirror script taught a different lesson about the hub layout:

1. **hub-level vs repo-level blob stores** — snapshot entries link
   `../../blobs/<sha>`; on the NAS that link may itself be a symlink into the
   2-char hub store (`hub/blobs/4d/<sha>`). A mirror that materializes only
   the link string's own path leaves 51 weight shards resolving through NAS
   (and later hardlink/fetch passes re-fetching 130 G+ of data that was
   already local).
2. **page15's content lives as real files** inside `snapshots/<rev>/` on the
   NAS — `shutil.copytree` copies them silently (195 G/node wasted, ENOSPC
   on both nodes); `copytree(dirs_exist_ok=True)` is additionally **not
   idempotent on symlinks** (EEXIST aborts the pass).
3. **NAS `st_mtime` staleness over NFS** made a size-only "already copied"
   check report a complete mirror while link targets were absent — size+
   sampled-content hashing (head+tail reads) is the only check that survives
   NFS.
4. Size-based hardlink dedup between stores can bind the *wrong* file when
   two weights share a size; must verify content first (caught: 4 weight
   blobs had been relinked to the wrong shard — engine then failed with
   `Duplicate mapped checkpoint key: …experts.300.w1.trellis`).
5. Final clean pass: wipe local repos, mirror trees + materialize every link
   target, `realpath`-map through the hub store, then per-blob sample-hash
   verify against NAS. Both nodes: 141/141 links local, 0 mismatches
   (`/tmp/stage-exl3-v9.py`, `links ok: 141, dangling: 0`).

## The three crash layers after staging

- `EngineCore init` failure #1: **51 weight blobs silently wrong** (see 4).
  Fixed by the verify pass.
- `FileExistsError: /cache/ds41-backend-attestation` — the kit's receipt
  writer is strict-on-exists (`ROOT.mkdir(exist_ok=False)` + `open('xb')`):
  receipts are **per-run** and the stale root/`worker.json` from yesterday's
  bench run was root-owned, so a plain `rm -rf` as the launcher user failed
  and poisoned every restart. The recipe's `before_serve` now clears all
  three receipt roots (`ds41-backend-attestation`, `ds41-combined-ready`,
  `ds41-display-kv`) with `sudo rm -rf` — correct fresh-run semantics.
- Same error persisted: the **page15 binds** were gone with my no-page15
  tree (the pack is optional, but the kit's loader still references it when
  mounted — the 171-mount manifest of the healthy baseline included 30 NAS
  fallback binds). The hook now falls back to the NAS tree for any snapshot
  directory not staged locally, so the manifest is back to 171 mounts
  (141 local + 30 NAS page15, read-only).

## Result (measured, 27 Sep ~04:20 UTC)

- API up 9 min after serve (streaming weights from local NVMe, full
  attestation passes on both ranks).
- Smoke: 127*89 → 11303, 1.5 s.
- Decode probe (487-token prompt, 400 tok, temp 0): **35.0 t/s**
  (34–41 under DSpark), previous **13.3 t/s** — 2.6×, above the 30.78 t/s
  reference. Prefill unchanged in profile; a bench is queued to verify the
  164/176 quality baseline stands with local weights.

Recipe changes committed: page15 NAS-fallback binds + receipt-clean hook.

## Mia AI-Lab TP4 — forum thread discovery (end of shift)

The thread /t/382725 continuation (posts #136–#156) points at
**MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks** — an SGLang TP4 "production
line" (knapcio's fork + RoCEnante + b12x + prefill SP + gated decode
adapters). Claims that matter: prose decode c1 **87.7 tok/s** (ours,
aiden16 lane: ~20), prefill 4.0–5.9K tok/s cold, 8M-token KV pin with a
1,011,084-token needle **verified**, qeval 72/75; stu.miller's independent
A/B (post #142, Sept 15 → Sept 25 builds, same 4 Sparks): prefill 300K
223.4s → 83.8s, decode c1 43.5 → 67.3, tool-eval 89/100, median turn
2.74 s. Repo is real, dense (29 adapters, row_store.cpp, packed engram
shards, sources.manifest), pinned `lmsysorg/sglang:dev-dsv41@3dbc3130…` +
checkpoint `fb2764a5`. Claims unverified until benched here.
Their `pack_engram.py` (≈48 GiB/rank) is exactly our staging sparse-rows
spec — the localization story and this lane converge.

## Open

- Bench re-run passed on the localized lane: **169/184 (92/100, ★★★★★)**
  — same standing as the 164/176 (93/100) baseline across the slightly
  different scenario pools (harness moved v2.7.0 → v2.7.1.dev7 by the
  operator; TC-74 injection failure is identical to the previous run — not a
  localization regression). Perf table: **pp 690–795 t/s** (was 112),
  **tg 17–33 t/s** (was ~13.3). Report
  `recipes/dsv41-flash-exl3-2x/reports/2026-09-27_053009_exl3-2x-local.*`.
- Aiden SGLang 4x engram localization — ranges derived (tables
  384,006,168 / 384,016,682 rows, contiguous quarter-partitions per rank,
  ~47.2 GiB/rank per Aiden docs); needs dgx3/4 powered and dgx1/2 freed.
- Registry go-live + r39 prod cut — operator decisions.