launcher/history/night-shifts/2026-09-27.md

Night shift 2026-09-27 (UTC)

Plan (user, carried from 09-26): close the DeepSeek perf gap by making weight/ engram I/O node-local NVMe (the way the published references do), EXL3 first, then Aiden SGLang engram staging.

EXL3 3bpw 2x — weight localization (dgx1+dgx2)

Baseline: 13.3 t/s decode — weights streamed from NAS on every forward. Reference (coolbho3k, same kit rev 878e0ee): weights+engrams on each host's local NVMe, ~30.8 t/s.

Staging of models--coolbho3k--DeepSeek-V4.1-Flash-EXL3-3bpw (+ DSpark drafter) to /home/kilork/hf-local/hub, page15 pack intentionally not staged (optional +1.4% decode per the release notes; 195 G/node).

The staging itself turned into a six-generation saga — every generation of my mirror script taught a different lesson about the hub layout:

1. hub-level vs repo-level blob stores — snapshot entries link ../../blobs/<sha>; on the NAS that link may itself be a symlink into the 2-char hub store (hub/blobs/4d/<sha>). A mirror that materializes only the link string's own path leaves 51 weight shards resolving through NAS (and later hardlink/fetch passes re-fetching 130 G+ of data that was already local).
2. page15's content lives as real files inside snapshots/<rev>/ on the NAS — shutil.copytree copies them silently (195 G/node wasted, ENOSPC on both nodes); copytree(dirs_exist_ok=True) is additionally not idempotent on symlinks (EEXIST aborts the pass).
3. NAS st_mtime staleness over NFS made a size-only "already copied" check report a complete mirror while link targets were absent — size+ sampled-content hashing (head+tail reads) is the only check that survives NFS.
4. Size-based hardlink dedup between stores can bind the wrong file when two weights share a size; must verify content first (caught: 4 weight blobs had been relinked to the wrong shard — engine then failed with Duplicate mapped checkpoint key: …experts.300.w1.trellis).
5. Final clean pass: wipe local repos, mirror trees + materialize every link target, realpath-map through the hub store, then per-blob sample-hash verify against NAS. Both nodes: 141/141 links local, 0 mismatches (/tmp/stage-exl3-v9.py, links ok: 141, dangling: 0).

The three crash layers after staging

• EngineCore init failure #1: 51 weight blobs silently wrong (see 4). Fixed by the verify pass.
• FileExistsError: /cache/ds41-backend-attestation — the kit's receipt writer is strict-on-exists (ROOT.mkdir(exist_ok=False) + open('xb')): receipts are per-run and the stale root/worker.json from yesterday's bench run was root-owned, so a plain rm -rf as the launcher user failed and poisoned every restart. The recipe's before_serve now clears all three receipt roots (ds41-backend-attestation, ds41-combined-ready, ds41-display-kv) with sudo rm -rf — correct fresh-run semantics.
• Same error persisted: the page15 binds were gone with my no-page15 tree (the pack is optional, but the kit's loader still references it when mounted — the 171-mount manifest of the healthy baseline included 30 NAS fallback binds). The hook now falls back to the NAS tree for any snapshot directory not staged locally, so the manifest is back to 171 mounts (141 local + 30 NAS page15, read-only).

Result (measured, 27 Sep ~04:20 UTC)

• API up 9 min after serve (streaming weights from local NVMe, full attestation passes on both ranks).
• Smoke: 127*89 → 11303, 1.5 s.
• Decode probe (487-token prompt, 400 tok, temp 0): 35.0 t/s (34–41 under DSpark), previous 13.3 t/s — 2.6×, above the 30.78 t/s reference. Prefill unchanged in profile; a bench is queued to verify the 164/176 quality baseline stands with local weights.

Recipe changes committed: page15 NAS-fallback binds + receipt-clean hook.

Mia AI-Lab TP4 — forum thread discovery (end of shift)

The thread /t/382725 continuation (posts #136–#156) points at MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks — an SGLang TP4 "production line" (knapcio's fork + RoCEnante + b12x + prefill SP + gated decode adapters). Claims that matter: prose decode c1 87.7 tok/s (ours, aiden16 lane: ~20), prefill 4.0–5.9K tok/s cold, 8M-token KV pin with a 1,011,084-token needle verified, qeval 72/75; stu.miller's independent A/B (post #142, Sept 15 → Sept 25 builds, same 4 Sparks): prefill 300K 223.4s → 83.8s, decode c1 43.5 → 67.3, tool-eval 89/100, median turn 2.74 s. Repo is real, dense (29 adapters, row_store.cpp, packed engram shards, sources.manifest), pinned lmsysorg/sglang:dev-dsv41@3dbc3130… + checkpoint fb2764a5. Claims unverified until benched here. Their pack_engram.py (≈48 GiB/rank) is exactly our staging sparse-rows spec — the localization story and this lane converge.

Mia AI-Lab TP4 — adopted as dsv41-flash-4x-mia (live tonight)

• Vendored MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks@cad252b7d3 into docker-images/mia-tp4 (build.sh + versions.env; the Dockerfile.canary-roce fetch stage pulls its pinned trees per runtime/sources.manifest). Image built on dgx1 in ~15 min — all 17 in-image adapter tests green.
• Two integration bugs found and fixed (each a class of its own):
  1. DSV41_LAUNCHER=tp4 — every TP4 adapter is gated on sitecustomize's _tp4_launcher(), which boot.py/start-tp4.sh set via env. Unset = the gated world silently off and the stock FlashInfer MoE rejects the 576-wide EP1 experts (ValueError … multiples of 128).
  2. TP_SIZE/EP_SIZE/DSV41_TP_PAD=0 — the adapters read the world shape from env, not sglang's flags; tp3_pad.py defaults TP_SIZE to 3, padded heads 64→96 under our TP4 (24 heads/rank) and the sparse-MLA kernel aborted warmup. Our launcher injects NODE_RANK only; boot.py exported these globals and we don't run boot.py.
• Boot: all production-line log signatures present — indexer chunked v3 ARMED (SG18 TP split True), RoCEnante ready world=4, moe_b12x_next armed (EP_SIZE=1), shared-expert K padding, verify_cap conf:0.1, DSpark gamma=5 folded, The server is fired up and ready to roll!. Load weight from NAS took 2060 s (their 170 s is local-NVMe — this is staging P1/P2's case).
• Smoke: 19+23=42, API at http://192.168.88.14:30000 (deepseek-v4.1-flash, 1M context, KV pool 5.79M after autotune clamp from our 8M pin). Decode c1 probe ≈ 29 t/s warm vs 87.7 claimed — explained by unpacked engram rows read over NFS (their measured worker penalty +11 ms/step) and no local pack; expect a jump when staging P2 lands.

Open

• Bench re-run passed on the localized lane: 169/184 (92/100, ★★★★★) — same standing as the 164/176 (93/100) baseline across the slightly different scenario pools (harness moved v2.7.0 → v2.7.1.dev7 by the operator; TC-74 injection failure is identical to the previous run — not a localization regression). Perf table: pp 690–795 t/s (was 112), tg 17–33 t/s (was ~13.3). Report recipes/dsv41-flash-exl3-2x/reports/2026-09-27_053009_exl3-2x-local.*.
• Aiden SGLang 4x engram localization — ranges derived (tables 384,006,168 / 384,016,682 rows, contiguous quarter-partitions per rank, ~47.2 GiB/rank per Aiden docs); needs dgx3/4 powered and dgx1/2 freed.
• Registry go-live + r39 prod cut — operator decisions.

Staging P1 — hub-mirror in the launcher

Implemented per specs/tech-architecture/staging-feature.md (operator directive: recipes declare staging, plain serve does everything):

• [[staging]] recipe block (src/config/staging.rs): kind hub-mirror (P1) / sparse-rows (P2 placeholder, serve aborts with the hint), id/src/dst/repos/skip/verify/min_free.
• Engine src/staging/scripts/hub_mirror.py: walks the repo tree, copies real files atomically, materializes out-of-source link targets as regular files, rewrites snapshot symlinks to the mirror, verifies by sampled content hashes in a ledger inside the mirror (dst/hub/.sm121-staging-ledger.json) — never mtime/size (the stale-NFS class that drove the EXL3 six-generation saga).
• src/staging/{rs,mirror.rs}: per-node parallel upload+run, markers, min-free preflight, dry-run plan; serve integration mounts the mirror's dst instead of the shared cache (spec swap) and rejects --mirror combination; --stage=auto|skip|force.
• E2E on dgx1 (read-only fixture, Mia untouched): first pass copies, second verifies only (copied=0), same-size mutation of a staged blob was detected and re-copied, --skip honored, dangling source link aborted. 345 tests.

Next: migrate EXL3's recipe hook to the declarative block and re-verify.

Staging P2 — sparse-rows (packed Engram)

• src/staging/rows.rs + scripts/sparse_rows.py (clean-room, format = runtime read contract): per-rank packed shards engram-l<L>-r<r>of<tp>.bin (4096 B header + 264 B records), rank from cluster launch rank, tp from recipe, ranges derived from the checkpoint header. Ledger of sampled row hashes per block in dst; corrupt-record detection and repack verified.
• First adoption: dsv41-flash-4x-mia — [[staging]] block (dst under the mia-state bind, min_free 55G) + DSV41_PACKED_DIR=/state/engram. Dry-run prints the per-node rank plan correctly (rank 0..3 / tp 4).
• E2E on dgx1 (synthetic safetensors fixture): pack pass, verify-only second pass, corruption of a packed record detected + repacked; row content round-trip matches the source interleave.

Gotchas noted: first staging serve adds ~15 min packing (NAS source) — beyond the 30-min readiness budget if the weight load is also cold, so the first live serve should use --no-wait; the aiden16 manifest flavor is a pending format; packed shards cost ~48 GiB/node (fits, but P3's LRU needs to know about them).

P2 live cutover — Mia packed Engram, decode 29 → 72-74 t/s

Cutover executed per operator decision: stop dsv41-flash-4x-mia → serve with the [[staging]] block (--no-wait for the first pass). One bug hit — the deployed release binary predated the out_dir.mkdir fix, the pack aborted on missing dst (fixed by rebuild, serve relaunched; staging resumed cleanly, nothing partial written). Pack took ~8 min/node for both layers (~50 GiB/node from the NAS snapshot, 4 nodes in parallel); boot log confirmed Exact nvme Engram ... packed=True for both layers and the derived r0 range [0,96001542) matched the row-split math exactly. Weight load dropped to 1320 s (vs 2060). Decode c1 probes: 71.9 / 73.8 / 74.0 tok/s — 2.5× the unpacked-NAS lane (29 t/s), closing most of the gap to Mia's claimed 87.7 (rest is their 87.7-prose probe shape). This is the measured proof of the staging thesis: a launcher feature, not a manual script, moved a production metric.

Lane state after cutover: dsv41-flash-4x-mia live on 4nodes-switch, packed engram at /home/kilork/mia-state/engram (~50 GiB/node, ledger <dir>/.sm121-staging-engram.json), DSV41_PACKED_DIR=/state/engram. Second serves verify-only (sampled row hashes) in ~2 s. Full bench of the packed lane is the next measurement (note the harness-bias caveat when comparing to the 169/184 EXL3 report).

EXL3 migrated to declarative staging (+ export materialization)

The 171-line inline manifest generator is gone: dsv41-flash-exl3-2x now declares one [[staging]] hub-mirror block (dst = the mirror the manual 2026-09-26 staging created — ledger adoption accepts untracked-but-matching files, so the migration is verify-only) with two exports materializing regular-file /model and /draft-exl3 trees for the O_NOFOLLOW runtime. The page15 engram table — the reason per-file binds existed — keeps its NAS binds, now emitted by the staging engine as the export manifest (/tmp/sm121-staging-exl3-manifests.txt), which extra_mounts_manifest reads back unchanged.

Two script fixes came out of the real-data E2E (the fixture couldn't catch them): the flat blobs/<sha> layout escaped the blob-link filter, and the reference-aware rule itself is new — repo blob-links are staged only when a non-skipped snapshot entry resolves to them (page15-class data never touches local disk again; first run pulled 86 G of page15 blobs twice and filled the node, both passes cleaned up, disk back to 795G/75G). Staging verified per node (dgx1/dgx2): 0 copies of page15, 133+10 export hardlinks, 28 manifest lines. Serve verification is for the lane cutover (Mia holds all four nodes during the bench).

Bench packed-Mia — 161/184 (88/100, ★★★★)

Full tool-eval-bench (seed 42, hardmode, perf conc 1,2, harness v2.7.1.dev7) against the packed-Engram lane: recipes/dsv41-flash-4x-mia/reports/2026-09-27_114508_packed-engram.* (md + json + recipe copy + manifest). Score 161/184 = 88/100 vs the EXL3 localized 169/184 (92/100) — different harness era, so the delta is not pure-Engram (the quality-gate pool changed between dev65-era and v2.7.x; re-baselining the EXL3 anchor under v2.7.1 would be the clean read). Perf confirms the staging thesis: tg c1 59-88 t/s (unpacked lane floored at 29; the packed decode probe read 71.9-74.0), pp 3.2-4.8K t/s. Responsiveness median 1.4 s, Deployability 84, two safety-critical failures to eyeball on the next review pass.

Staging P2.5 — sparse-copy format (aiden16 DSV41_ENGRAM_DIR)

[[staging]] kind="sparse-rows", format="sparse-copy" — same ledger/rank machinery, different output: sparse node-local copies of the original safetensors shards containing exactly the rank's row ranges (holes free, size must match source — the loader mmaps and preads at original offsets)

• engram-local.json row-range manifest + copied index. Sample-hash verification adopts existing trees (manual tools/engram_local.py staging) without re-copying, and rebuilds ranges when they don't verify. E2E on dgx1 (synthetic fixture, rank1/tp4): pack, verify-only, adoption, corrupt-one-row rebuild — all green; sparse-ness confirmed (528 K logical, 147 K allocated).

EXL3 lane cutover — declarative staging live, 46.8 tok/s decode c1

Mia stopped (bench served), EXL3 brought up on 2nodes-dac via the migrated [[staging]] recipe. Three real bugs surfaced on the live cutover and were fixed in hub_mirror.py / config / recipe:

1. hf_cache_dir misplacement in config.toml — it sat after the [[clusters]] headers and was parsed into the cluster table, silently ignored: staging fell back to ~/.cache/huggingface. On dgx1 that is a symlink to the NAS (looked fine), on dgx2 to the node mirror itself (src==dst: page15 vanished, export went full-local, manifest empty). Key moved root-level; dgx2's symlink repointed to the NAS.
2. realpath() on source links broke the composed-model shared-root rule: with a symlinked src the blob targets landed "outside" and were duplicated into the snapshot tree as real files (147–175 GiB/node of dead weight, re-copied every serve). Fixed with syntactic link resolution (normpath(dirname+readlink)), same technique as composed_model mounts. The 175+147 GiB of duplicates removed.
3. Export materialization got two fixes: (a) the mirror's older repo-blob layout behind symlinks is now resolved to the real local blob (weights hardlink locally instead of falling to NAS binds); (b) skip-listed subtrees (page15) get mountpoint placeholders + per-file binds, which is what the runtime's clean_pages_advised guard requires: weight shards on a read-only filesystem (statvfs ST_RDONLY), so the /model and /draft-exl3 binds went back to read_only = true.

Live verification (dgx1+dgx2): staging verify-only (131 links, 30 page15 binds), guard passed (files=56 advised), page15 visible, /model ro, /v1/completions clean, decode 46.8 tok/s c1 (2x the recorded 17–33 tok/s band — the localization goal achieved). Disk: dgx1 865→690 GB used after cleanup (duplicates + /var/tmp/engram-local 49 G stale ×2 + Mia packed shards on dgx1/2, regenerable via [[staging]]).

Side effect worth remembering: the packed-Mia lanes now repack declaratively on their next serve (~8 min/node) instead of carrying the state on disk.