|
|
@@ -0,0 +1,78 @@
|
|
|
+# Night shift 2026-09-27 (UTC)
|
|
|
+
|
|
|
+Plan (user, carried from 09-26): close the DeepSeek perf gap by making weight/
|
|
|
+engram I/O **node-local NVMe** (the way the published references do), EXL3
|
|
|
+first, then Aiden SGLang engram staging.
|
|
|
+
|
|
|
+## EXL3 3bpw 2x — weight localization (dgx1+dgx2)
|
|
|
+
|
|
|
+Baseline: 13.3 t/s decode — weights streamed from NAS on every forward.
|
|
|
+Reference (coolbho3k, same kit rev 878e0ee): weights+engrams on each host's
|
|
|
+local NVMe, ~30.8 t/s.
|
|
|
+
|
|
|
+Staging of `models--coolbho3k--DeepSeek-V4.1-Flash-EXL3-3bpw`
|
|
|
+(+ DSpark drafter) to `/home/kilork/hf-local/hub`, page15 pack intentionally
|
|
|
+**not** staged (optional +1.4% decode per the release notes; 195 G/node).
|
|
|
+
|
|
|
+The staging itself turned into a six-generation saga — every generation of my
|
|
|
+mirror script taught a different lesson about the hub layout:
|
|
|
+
|
|
|
+1. **hub-level vs repo-level blob stores** — snapshot entries link
|
|
|
+ `../../blobs/<sha>`; on the NAS that link may itself be a symlink into the
|
|
|
+ 2-char hub store (`hub/blobs/4d/<sha>`). A mirror that materializes only
|
|
|
+ the link string's own path leaves 51 weight shards resolving through NAS
|
|
|
+ (and later hardlink/fetch passes re-fetching 130 G+ of data that was
|
|
|
+ already local).
|
|
|
+2. **page15's content lives as real files** inside `snapshots/<rev>/` on the
|
|
|
+ NAS — `shutil.copytree` copies them silently (195 G/node wasted, ENOSPC
|
|
|
+ on both nodes); `copytree(dirs_exist_ok=True)` is additionally **not
|
|
|
+ idempotent on symlinks** (EEXIST aborts the pass).
|
|
|
+3. **NAS `st_mtime` staleness over NFS** made a size-only "already copied"
|
|
|
+ check report a complete mirror while link targets were absent — size+
|
|
|
+ sampled-content hashing (head+tail reads) is the only check that survives
|
|
|
+ NFS.
|
|
|
+4. Size-based hardlink dedup between stores can bind the *wrong* file when
|
|
|
+ two weights share a size; must verify content first (caught: 4 weight
|
|
|
+ blobs had been relinked to the wrong shard — engine then failed with
|
|
|
+ `Duplicate mapped checkpoint key: …experts.300.w1.trellis`).
|
|
|
+5. Final clean pass: wipe local repos, mirror trees + materialize every link
|
|
|
+ target, `realpath`-map through the hub store, then per-blob sample-hash
|
|
|
+ verify against NAS. Both nodes: 141/141 links local, 0 mismatches
|
|
|
+ (`/tmp/stage-exl3-v9.py`, `links ok: 141, dangling: 0`).
|
|
|
+
|
|
|
+## The three crash layers after staging
|
|
|
+
|
|
|
+- `EngineCore init` failure #1: **51 weight blobs silently wrong** (see 4).
|
|
|
+ Fixed by the verify pass.
|
|
|
+- `FileExistsError: /cache/ds41-backend-attestation` — the kit's receipt
|
|
|
+ writer is strict-on-exists (`ROOT.mkdir(exist_ok=False)` + `open('xb')`):
|
|
|
+ receipts are **per-run** and the stale root/`worker.json` from yesterday's
|
|
|
+ bench run was root-owned, so a plain `rm -rf` as the launcher user failed
|
|
|
+ and poisoned every restart. The recipe's `before_serve` now clears all
|
|
|
+ three receipt roots (`ds41-backend-attestation`, `ds41-combined-ready`,
|
|
|
+ `ds41-display-kv`) with `sudo rm -rf` — correct fresh-run semantics.
|
|
|
+- Same error persisted: the **page15 binds** were gone with my no-page15
|
|
|
+ tree (the pack is optional, but the kit's loader still references it when
|
|
|
+ mounted — the 171-mount manifest of the healthy baseline included 30 NAS
|
|
|
+ fallback binds). The hook now falls back to the NAS tree for any snapshot
|
|
|
+ directory not staged locally, so the manifest is back to 171 mounts
|
|
|
+ (141 local + 30 NAS page15, read-only).
|
|
|
+
|
|
|
+## Result (measured, 27 Sep ~04:20 UTC)
|
|
|
+
|
|
|
+- API up 9 min after serve (streaming weights from local NVMe, full
|
|
|
+ attestation passes on both ranks).
|
|
|
+- Smoke: 127*89 → 11303, 1.5 s.
|
|
|
+- Decode probe (487-token prompt, 400 tok, temp 0): **35.0 t/s**
|
|
|
+ (34–41 under DSpark), previous **13.3 t/s** — 2.6×, above the 30.78 t/s
|
|
|
+ reference. Prefill unchanged in profile; a bench is queued to verify the
|
|
|
+ 164/176 quality baseline stands with local weights.
|
|
|
+
|
|
|
+Recipe changes committed: page15 NAS-fallback binds + receipt-clean hook.
|
|
|
+
|
|
|
+## Open
|
|
|
+
|
|
|
+- Full bench re-run on the localized lane (expect ≥164/176 unchanged).
|
|
|
+- Aiden SGLang 4x engram localization (`engram_local.py`, ~47 GiB/rank sparse
|
|
|
+ copies on 4 nodes) — needs dgx3/4 powered; deferred pending user go-ahead.
|
|
|
+- Registry go-live + r39 prod cut — operator decisions.
|
