launcher/history/night-shifts/2026-09-27.md

Night shift 2026-09-27 (UTC)

Plan (user, carried from 09-26): close the DeepSeek perf gap by making weight/ engram I/O node-local NVMe (the way the published references do), EXL3 first, then Aiden SGLang engram staging.

EXL3 3bpw 2x — weight localization (dgx1+dgx2)

Baseline: 13.3 t/s decode — weights streamed from NAS on every forward. Reference (coolbho3k, same kit rev 878e0ee): weights+engrams on each host's local NVMe, ~30.8 t/s.

Staging of models--coolbho3k--DeepSeek-V4.1-Flash-EXL3-3bpw (+ DSpark drafter) to /home/kilork/hf-local/hub, page15 pack intentionally not staged (optional +1.4% decode per the release notes; 195 G/node).

The staging itself turned into a six-generation saga — every generation of my mirror script taught a different lesson about the hub layout:

1. hub-level vs repo-level blob stores — snapshot entries link ../../blobs/<sha>; on the NAS that link may itself be a symlink into the 2-char hub store (hub/blobs/4d/<sha>). A mirror that materializes only the link string's own path leaves 51 weight shards resolving through NAS (and later hardlink/fetch passes re-fetching 130 G+ of data that was already local).
2. page15's content lives as real files inside snapshots/<rev>/ on the NAS — shutil.copytree copies them silently (195 G/node wasted, ENOSPC on both nodes); copytree(dirs_exist_ok=True) is additionally not idempotent on symlinks (EEXIST aborts the pass).
3. NAS st_mtime staleness over NFS made a size-only "already copied" check report a complete mirror while link targets were absent — size+ sampled-content hashing (head+tail reads) is the only check that survives NFS.
4. Size-based hardlink dedup between stores can bind the wrong file when two weights share a size; must verify content first (caught: 4 weight blobs had been relinked to the wrong shard — engine then failed with Duplicate mapped checkpoint key: …experts.300.w1.trellis).
5. Final clean pass: wipe local repos, mirror trees + materialize every link target, realpath-map through the hub store, then per-blob sample-hash verify against NAS. Both nodes: 141/141 links local, 0 mismatches (/tmp/stage-exl3-v9.py, links ok: 141, dangling: 0).

The three crash layers after staging

• EngineCore init failure #1: 51 weight blobs silently wrong (see 4). Fixed by the verify pass.
• FileExistsError: /cache/ds41-backend-attestation — the kit's receipt writer is strict-on-exists (ROOT.mkdir(exist_ok=False) + open('xb')): receipts are per-run and the stale root/worker.json from yesterday's bench run was root-owned, so a plain rm -rf as the launcher user failed and poisoned every restart. The recipe's before_serve now clears all three receipt roots (ds41-backend-attestation, ds41-combined-ready, ds41-display-kv) with sudo rm -rf — correct fresh-run semantics.
• Same error persisted: the page15 binds were gone with my no-page15 tree (the pack is optional, but the kit's loader still references it when mounted — the 171-mount manifest of the healthy baseline included 30 NAS fallback binds). The hook now falls back to the NAS tree for any snapshot directory not staged locally, so the manifest is back to 171 mounts (141 local + 30 NAS page15, read-only).

Result (measured, 27 Sep ~04:20 UTC)

• API up 9 min after serve (streaming weights from local NVMe, full attestation passes on both ranks).
• Smoke: 127*89 → 11303, 1.5 s.
• Decode probe (487-token prompt, 400 tok, temp 0): 35.0 t/s (34–41 under DSpark), previous 13.3 t/s — 2.6×, above the 30.78 t/s reference. Prefill unchanged in profile; a bench is queued to verify the 164/176 quality baseline stands with local weights.

Recipe changes committed: page15 NAS-fallback binds + receipt-clean hook.

Open

• Full bench re-run on the localized lane (expect ≥164/176 unchanged).
• Aiden SGLang 4x engram localization (engram_local.py, ~47 GiB/rank sparse copies on 4 nodes) — needs dgx3/4 powered; deferred pending user go-ahead.
• Registry go-live + r39 prod cut — operator decisions.