# Night shift 2026-09-27 (UTC)

Plan (user, carried from 09-26): close the DeepSeek perf gap by making weight/
engram I/O **node-local NVMe** (the way the published references do), EXL3
first, then Aiden SGLang engram staging.

## EXL3 3bpw 2x — weight localization (dgx1+dgx2)

Baseline: 13.3 t/s decode — weights streamed from NAS on every forward.
Reference (coolbho3k, same kit rev 878e0ee): weights+engrams on each host's
local NVMe, ~30.8 t/s.

Staging of `models--coolbho3k--DeepSeek-V4.1-Flash-EXL3-3bpw`
(+ DSpark drafter) to `/home/kilork/hf-local/hub`, page15 pack intentionally
**not** staged (optional +1.4% decode per the release notes; 195 G/node).

The staging itself turned into a six-generation saga — every generation of my
mirror script taught a different lesson about the hub layout:

1. **hub-level vs repo-level blob stores** — snapshot entries link
   `../../blobs/<sha>`; on the NAS that link may itself be a symlink into the
   2-char hub store (`hub/blobs/4d/<sha>`). A mirror that materializes only
   the link string's own path leaves 51 weight shards resolving through NAS
   (and later hardlink/fetch passes re-fetching 130 G+ of data that was
   already local).
2. **page15's content lives as real files** inside `snapshots/<rev>/` on the
   NAS — `shutil.copytree` copies them silently (195 G/node wasted, ENOSPC
   on both nodes); `copytree(dirs_exist_ok=True)` is additionally **not
   idempotent on symlinks** (EEXIST aborts the pass).
3. **NAS `st_mtime` staleness over NFS** made a size-only "already copied"
   check report a complete mirror while link targets were absent — size+
   sampled-content hashing (head+tail reads) is the only check that survives
   NFS.
4. Size-based hardlink dedup between stores can bind the *wrong* file when
   two weights share a size; must verify content first (caught: 4 weight
   blobs had been relinked to the wrong shard — engine then failed with
   `Duplicate mapped checkpoint key: …experts.300.w1.trellis`).
5. Final clean pass: wipe local repos, mirror trees + materialize every link
   target, `realpath`-map through the hub store, then per-blob sample-hash
   verify against NAS. Both nodes: 141/141 links local, 0 mismatches
   (`/tmp/stage-exl3-v9.py`, `links ok: 141, dangling: 0`).

## The three crash layers after staging

- `EngineCore init` failure #1: **51 weight blobs silently wrong** (see 4).
  Fixed by the verify pass.
- `FileExistsError: /cache/ds41-backend-attestation` — the kit's receipt
  writer is strict-on-exists (`ROOT.mkdir(exist_ok=False)` + `open('xb')`):
  receipts are **per-run** and the stale root/`worker.json` from yesterday's
  bench run was root-owned, so a plain `rm -rf` as the launcher user failed
  and poisoned every restart. The recipe's `before_serve` now clears all
  three receipt roots (`ds41-backend-attestation`, `ds41-combined-ready`,
  `ds41-display-kv`) with `sudo rm -rf` — correct fresh-run semantics.
- Same error persisted: the **page15 binds** were gone with my no-page15
  tree (the pack is optional, but the kit's loader still references it when
  mounted — the 171-mount manifest of the healthy baseline included 30 NAS
  fallback binds). The hook now falls back to the NAS tree for any snapshot
  directory not staged locally, so the manifest is back to 171 mounts
  (141 local + 30 NAS page15, read-only).

## Result (measured, 27 Sep ~04:20 UTC)

- API up 9 min after serve (streaming weights from local NVMe, full
  attestation passes on both ranks).
- Smoke: 127*89 → 11303, 1.5 s.
- Decode probe (487-token prompt, 400 tok, temp 0): **35.0 t/s**
  (34–41 under DSpark), previous **13.3 t/s** — 2.6×, above the 30.78 t/s
  reference. Prefill unchanged in profile; a bench is queued to verify the
  164/176 quality baseline stands with local weights.

Recipe changes committed: page15 NAS-fallback binds + receipt-clean hook.

## Mia AI-Lab TP4 — forum thread discovery (end of shift)

The thread /t/382725 continuation (posts #136–#156) points at
**MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks** — an SGLang TP4 "production
line" (knapcio's fork + RoCEnante + b12x + prefill SP + gated decode
adapters). Claims that matter: prose decode c1 **87.7 tok/s** (ours,
aiden16 lane: ~20), prefill 4.0–5.9K tok/s cold, 8M-token KV pin with a
1,011,084-token needle **verified**, qeval 72/75; stu.miller's independent
A/B (post #142, Sept 15 → Sept 25 builds, same 4 Sparks): prefill 300K
223.4s → 83.8s, decode c1 43.5 → 67.3, tool-eval 89/100, median turn
2.74 s. Repo is real, dense (29 adapters, row_store.cpp, packed engram
shards, sources.manifest), pinned `lmsysorg/sglang:dev-dsv41@3dbc3130…` +
checkpoint `fb2764a5`. Claims unverified until benched here.
Their `pack_engram.py` (≈48 GiB/rank) is exactly our staging sparse-rows
spec — the localization story and this lane converge.

## Mia AI-Lab TP4 — adopted as `dsv41-flash-4x-mia` (live tonight)

- Vendored `MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks@cad252b7d3` into
  `docker-images/mia-tp4` (build.sh + versions.env; the Dockerfile.canary-roce
  fetch stage pulls its pinned trees per `runtime/sources.manifest`). Image
  built on dgx1 in ~15 min — all 17 in-image adapter tests green.
- Two integration bugs found and fixed (each a class of its own):
  1. **`DSV41_LAUNCHER=tp4`** — every TP4 adapter is gated on sitecustomize's
     `_tp4_launcher()`, which boot.py/start-tp4.sh set via env. Unset = the
     gated world silently off and the stock FlashInfer MoE rejects the
     576-wide EP1 experts (`ValueError … multiples of 128`).
  2. **`TP_SIZE`/`EP_SIZE`/`DSV41_TP_PAD=0`** — the adapters read the world
     shape from env, not sglang's flags; `tp3_pad.py` defaults `TP_SIZE` to 3,
     padded heads 64→96 under our TP4 (24 heads/rank) and the sparse-MLA
     kernel aborted warmup. Our launcher injects NODE_RANK only; boot.py
     exported these globals and we don't run boot.py.
- Boot: all production-line log signatures present — indexer chunked v3
  ARMED (SG18 TP split True), RoCEnante ready world=4, moe_b12x_next armed
  (EP_SIZE=1), shared-expert K padding, verify_cap conf:0.1, DSpark gamma=5
  folded, `The server is fired up and ready to roll!`. Load weight from NAS
  took 2060 s (their 170 s is local-NVMe — this is staging P1/P2's case).
- Smoke: 19+23=42, API at `http://192.168.88.14:30000` (`deepseek-v4.1-flash`,
  1M context, KV pool 5.79M after autotune clamp from our 8M pin).
  Decode c1 probe ≈ 29 t/s warm vs 87.7 claimed — explained by unpacked
  engram rows read over NFS (their measured worker penalty +11 ms/step) and
  no local pack; expect a jump when staging P2 lands.

## Open

- Bench re-run passed on the localized lane: **169/184 (92/100, ★★★★★)**
  — same standing as the 164/176 (93/100) baseline across the slightly
  different scenario pools (harness moved v2.7.0 → v2.7.1.dev7 by the
  operator; TC-74 injection failure is identical to the previous run — not a
  localization regression). Perf table: **pp 690–795 t/s** (was 112),
  **tg 17–33 t/s** (was ~13.3). Report
  `recipes/dsv41-flash-exl3-2x/reports/2026-09-27_053009_exl3-2x-local.*`.
- Aiden SGLang 4x engram localization — ranges derived (tables
  384,006,168 / 384,016,682 rows, contiguous quarter-partitions per rank,
  ~47.2 GiB/rank per Aiden docs); needs dgx3/4 powered and dgx1/2 freed.
- Registry go-live + r39 prod cut — operator decisions.

## Staging P1 — hub-mirror in the launcher

Implemented per `specs/tech-architecture/staging-feature.md` (operator
directive: recipes declare staging, plain `serve` does everything):

- `[[staging]]` recipe block (`src/config/staging.rs`): kind
  `hub-mirror` (P1) / `sparse-rows` (P2 placeholder, serve aborts with the
  hint), `id/src/dst/repos/skip/verify/min_free`.
- Engine `src/staging/scripts/hub_mirror.py`: walks the repo tree, copies
  real files atomically, materializes out-of-source link targets as regular
  files, rewrites snapshot symlinks to the mirror, verifies by **sampled
  content hashes** in a ledger inside the mirror
  (`dst/hub/.sm121-staging-ledger.json`) — never mtime/size (the stale-NFS
  class that drove the EXL3 six-generation saga).
- `src/staging/{rs,mirror.rs}`: per-node parallel upload+run, markers,
  min-free preflight, dry-run plan; serve integration mounts the mirror's
  `dst` instead of the shared cache (spec swap) and rejects `--mirror`
  combination; `--stage=auto|skip|force`.
- E2E on dgx1 (read-only fixture, Mia untouched): first pass copies, second
  verifies only (copied=0), same-size mutation of a staged blob was detected
  and re-copied, `--skip` honored, dangling source link aborted. 345 tests.

Next: migrate EXL3's recipe hook to the declarative block and re-verify.

## Staging P2 — sparse-rows (packed Engram)

- `src/staging/rows.rs` + `scripts/sparse_rows.py` (clean-room, format =
  runtime read contract): per-rank packed shards `engram-l<L>-r<r>of<tp>.bin`
  (4096 B header + 264 B records), rank from cluster launch rank, tp from
  recipe, ranges derived from the checkpoint header. Ledger of sampled row
  hashes per block in dst; corrupt-record detection and repack verified.
- First adoption: `dsv41-flash-4x-mia` — `[[staging]]` block (dst under the
  mia-state bind, `min_free 55G`) + `DSV41_PACKED_DIR=/state/engram`.
  Dry-run prints the per-node rank plan correctly (rank 0..3 / tp 4).
- E2E on dgx1 (synthetic safetensors fixture): pack pass, verify-only
  second pass, corruption of a packed record detected + repacked; row
  content round-trip matches the source interleave.

Gotchas noted: first staging serve adds ~15 min packing (NAS source) —
beyond the 30-min readiness budget if the weight load is also cold, so the
first live serve should use `--no-wait`; the aiden16 manifest flavor is a
pending format; packed shards cost ~48 GiB/node (fits, but P3's LRU needs
to know about them).

## P2 live cutover — Mia packed Engram, decode 29 → 72-74 t/s

Cutover executed per operator decision: stop `dsv41-flash-4x-mia` → serve
with the [[staging]] block (`--no-wait` for the first pass). One bug hit —
the deployed release binary predated the `out_dir.mkdir` fix, the pack
aborted on missing dst (fixed by rebuild, serve relaunched; staging
resumed cleanly, nothing partial written). Pack took ~8 min/node for both
layers (~50 GiB/node from the NAS snapshot, 4 nodes in parallel); boot log
confirmed `Exact nvme Engram ... packed=True` for both layers and the
derived r0 range `[0,96001542)` matched the row-split math exactly.
Weight load dropped to 1320 s (vs 2060). **Decode c1 probes: 71.9 / 73.8 /
74.0 tok/s** — 2.5× the unpacked-NAS lane (29 t/s), closing most of the gap
to Mia's claimed 87.7 (rest is their 87.7-prose probe shape). This is the
measured proof of the staging thesis: a launcher feature, not a manual
script, moved a production metric.

Lane state after cutover: `dsv41-flash-4x-mia` live on 4nodes-switch,
packed engram at `/home/kilork/mia-state/engram` (~50 GiB/node, ledger
`<dir>/.sm121-staging-engram.json`), `DSV41_PACKED_DIR=/state/engram`.
Second serves verify-only (sampled row hashes) in ~2 s. Full bench of the
packed lane is the next measurement (note the harness-bias caveat when
comparing to the 169/184 EXL3 report).

## EXL3 migrated to declarative staging (+ export materialization)

The 171-line inline manifest generator is gone: `dsv41-flash-exl3-2x` now
declares one `[[staging]]` hub-mirror block (dst = the mirror the manual
2026-09-26 staging created — ledger adoption accepts untracked-but-matching
files, so the migration is verify-only) with two exports materializing
regular-file `/model` and `/draft-exl3` trees for the O_NOFOLLOW runtime.
The page15 engram table — the reason per-file binds existed — keeps its
NAS binds, now emitted by the staging engine as the export manifest
(/tmp/sm121-staging-exl3-manifests.txt), which `extra_mounts_manifest`
reads back unchanged.

Two script fixes came out of the real-data E2E (the fixture couldn't catch
them): the flat `blobs/<sha>` layout escaped the blob-link filter, and the
reference-aware rule itself is new — repo blob-links are staged only when
a non-skipped snapshot entry resolves to them (page15-class data never
touches local disk again; first run pulled 86 G of page15 blobs twice and
filled the node, both passes cleaned up, disk back to 795G/75G).
Staging verified per node (dgx1/dgx2): 0 copies of page15, 133+10 export
hardlinks, 28 manifest lines. Serve verification is for the lane cutover
(Mia holds all four nodes during the bench).

## Bench packed-Mia — 161/184 (88/100, ★★★★)

Full `tool-eval-bench` (seed 42, hardmode, perf conc 1,2, harness
v2.7.1.dev7) against the packed-Engram lane:
`recipes/dsv41-flash-4x-mia/reports/2026-09-27_114508_packed-engram.*`
(md + json + recipe copy + manifest). Score 161/184 = 88/100 vs the EXL3
localized 169/184 (92/100) — different harness era, so the delta is not
pure-Engram (the quality-gate pool changed between dev65-era and v2.7.x;
re-baselining the EXL3 anchor under v2.7.1 would be the clean read).
Perf confirms the staging thesis: **tg c1 59-88 t/s** (unpacked lane
floored at 29; the packed decode probe read 71.9-74.0), pp 3.2-4.8K t/s.
Responsiveness median 1.4 s, Deployability 84, two safety-critical failures
to eyeball on the next review pass.

## Staging P2.5 — sparse-copy format (aiden16 `DSV41_ENGRAM_DIR`)

`[[staging]] kind="sparse-rows", format="sparse-copy"` — same ledger/rank
machinery, different output: sparse node-local copies of the original
safetensors shards containing exactly the rank's row ranges (holes free,
size must match source — the loader mmaps and preads at original offsets)
+ `engram-local.json` row-range manifest + copied index. Sample-hash
verification adopts existing trees (manual `tools/engram_local.py` staging)
without re-copying, and rebuilds ranges when they don't verify. E2E on
dgx1 (synthetic fixture, rank1/tp4): pack, verify-only, adoption,
corrupt-one-row rebuild — all green; sparse-ness confirmed (528 K logical,
147 K allocated).

## EXL3 lane cutover — declarative staging live, 46.8 tok/s decode c1

Mia stopped (bench served), EXL3 brought up on `2nodes-dac` via the migrated
`[[staging]]` recipe. Three real bugs surfaced on the live cutover and were
fixed in `hub_mirror.py` / config / recipe:

1. **`hf_cache_dir` misplacement in config.toml** — it sat *after* the
   `[[clusters]]` headers and was parsed into the cluster table, silently
   ignored: staging fell back to `~/.cache/huggingface`. On dgx1 that is a
   symlink to the NAS (looked fine), on dgx2 to the node mirror itself
   (src==dst: page15 vanished, export went full-local, manifest empty).
   Key moved root-level; dgx2's symlink repointed to the NAS.
2. **`realpath()` on source links** broke the composed-model shared-root
   rule: with a symlinked src the blob targets landed "outside" and were
   duplicated into the snapshot tree as real files (147–175 GiB/node of
   dead weight, re-copied every serve). Fixed with syntactic link
   resolution (`normpath(dirname+readlink)`), same technique as
   composed_model mounts. The 175+147 GiB of duplicates removed.
3. **Export materialization** got two fixes: (a) the mirror's older
   repo-blob layout behind symlinks is now resolved to the real local blob
   (weights hardlink locally instead of falling to NAS binds); (b)
   skip-listed subtrees (page15) get mountpoint placeholders + per-file
   binds, which is what the runtime's `clean_pages_advised` guard requires:
   weight shards on a **read-only** filesystem (`statvfs ST_RDONLY`), so
   the /model and /draft-exl3 binds went back to `read_only = true`.

Live verification (dgx1+dgx2): staging verify-only (131 links, 30 page15
binds), guard passed (`files=56` advised), page15 visible, /model ro,
`/v1/completions` clean, **decode 46.8 tok/s c1** (2x the recorded
17–33 tok/s band — the localization goal achieved). Disk: dgx1 865→690 GB
used after cleanup (duplicates + `/var/tmp/engram-local` 49 G stale ×2 +
Mia packed shards on dgx1/2, regenerable via `[[staging]]`).

Side effect worth remembering: **the packed-Mia lanes now repack
declaratively on their next serve** (~8 min/node) instead of carrying the
state on disk.

## Bench exl3-staged — 169/184 (92/100, ★★★★★), quality held on declarative staging

Full run on the freshly migrated lane (`2026-09-27_172837_exl3-staged.*`,
harness v2.7.1.dev7, seed 42, hardmode, perf c1/2): points identical to the
pre-migration manual-staging reference modulo pool (169/184 = 92/100 vs
164/176 = 93/100 on v2.7.0) — **the declarative staging is quality-neutral**.
Responsiveness 19→34/100 (median turn 7.9→4.7 s): local weights pay off on
tool-loop latency. Perf row: pp 714–744, tg 13.6–27.2 (c1/c2). 2
safety-critical failures (TC-74 batched-send order, TC-92 tenant leak) —
model-side, not lane-side, same families as other DS4.1 lanes.

Tooling: `bench --tokenizer` added (EXL3's served id isn't resolvable
offline in the local HF cache); tokenizer pulled from the lane mirror.

## aiden16 cutover — localized Engram + the liveness bug the operator smelled

EXL3 benched (92/100, quality-neutral vs manual staging) and stopped;
aiden16 serves `4nodes-switch` with the Engram rows off the NAS for the
first time. Recipe migration: `[[staging]]` sparse-copy → `/home/kilork/
aiden-engram`, ro-mounted at `/engram-local`, `SPARK_ENGRAM_DIR` repointed.

Two real bugs found on live cutover, both fixed:
1. **sparse-copy had no safetensors header** — stage_sparse_layer copied
   only the engram row ranges; the aiden overlay parses the header first
   and died with "safetensors header is not JSON". Header is now copied
   once per shard (repair fast path: header-only fix on existing trees,
   no range recopy — dgx1/2 healed in seconds).
2. **row-tiling mismatch** — our sparse-copy used `rows*rank//tp`, the
   aiden wrapper tiles `base+rem` (they differ by one row when
   rows%tp!=0, layer14 here), and the overlay cross-checks
   manifest==SPARK_ENGRAM_ROWS. Formula aligned (fixture 1006 rows
   verified == wrapper).

And the liveness bug the operator suspected: `watch()` in readiness.rs
counted the API-not-yet-open window (weight load, 17+ min) as death and
bailed with "containers died mid-follow" while all four containers were
Up. Watch now ignores connection-refused until the head API has answered
(or listens) once. Fix `6ae3c51` — the same class as the earlier
"serve exited but Docker is fine" observation.

Live state: sglang healthy, model `deepseek-v4.1-flash`, spot: ' Paris.'
clean, 300-token generation ≈ 75 t/s decode c1 (Aiden's band is
107/87/36; the bench will tell). Re-bench of this lane on localized
engram rows pending.

## Bench engram-local (aiden16) — 167/184 (91/100, ★★★★★), fresh-context penalty eliminated

Full hardmode run on the localized-Engram config (`2026-09-27_193307_
engram-local.*`, harness v2.7.1.dev7, seed 42): quality is within noise of
the NAS-engram baseline (162/176 ≈ 92/100), Deployability 85, and the perf
row shows what the staging was for: **pp 154-359 → 2,370-3,665 t/s** (10×+),
**tg 8.6-21.8 → 34-52 t/s**, median turn 4.7→1.7 s (Responsiveness 70/100).
The 2-10 s fresh-context prefill penalty of the NAS row-read regime is
gone. Safety fails TC-74/92 — model-side, identical families as the other
DS4.1 lanes (TC-68 passed this time).

## MiMo-V2.6-Flash-MOPD lane — the Flash benchmark winner, 499K KV pool
- config A (GMU 0.85, pool 306,752): 170/184 = 92/100 ★★★★★ (mopd-base)
- config B (GMU 0.88, pool 498,752 = context 327,680 + 52% headroom): 168/184 = 91/100 — noise; perf/responsiveness unchanged. 0.88 is free, keep.
- MOPD > Flash-RL by +9 pts (161→170) at equal pool; Pro-MOPD (573.5 GB) is
  COMPLETE on dgx1 — the "535/573 GB, ~99%" report was a GB↔GiB misread
  (`du` GiB 534 ≈ 573.5 GB decimal). Verified: snapshot `adea8e2c`, 130/130
  index shards, dflash draft + audio tokenizer, 0 `.incomplete`. The 8x Pro
  lane is blocked on fleet capacity, not weights.

## Mikrotik power-off relay fix (`5c82201`)

`power off` for a `[mikrotik]` device only sent API `/system/shutdown` — the
device stayed powered and could boot back (and `power on` became a no-op).
`shutdown_named` now also flips the `on_entity` outlet via HA right after the
API shutdown (warn-only). Verified: SW2-MUC (192.168.88.39) finally silent.
The real off-switch for router SW2 is the `switch.sw6_mt` relay.

## MOPD production serve + config C — 512K context verified (28 Sep)

Cutover: aiden16 stopped, 4nodes-switch powered off (router incl. relay),
2nodes-dac up for MOPD. dgx5 untouched (qwen3.8, user agents).
- Config C (GMU 0.90, `--context-length 524288`): full pool **570,752** +
  SWA 57,024 — the 512K target met with ~109K headroom.
- 515,001-token completion: prefill + decode clean in 625 s (~825 t/s);
  inputs >524,288 rejected with the clean context-length error. Host free
  after the run 7.0/8.1 GB per node, ~8 GB GPU headroom — 0.90 is the safe
  ceiling, no OOM pressure. Smoke OK, decode ~100 t/s c1.
- Today's prod lane: `mimo26-flash-mopd-2x` on 2nodes-dac (dgx1+dgx2).

## serve: one re-probe before "too few free nodes" (`848f902`)

Operator-flagged: serve kept failing "requires 2 nodes, but cluster X has
only 1" — twice today. Root cause is NOT stop-wait (post-stop `docker ps`
was clean each time): the fleet probe is single-shot, and one ssh blip /
a node still warming up after stop or power-on drops that node from the
free set. `prepare_cluster` now sleeps `BootTiming::retry_grace_secs`
(5 s default) and re-probes + re-selects once before bailing, and the
error lists per-node state (free/occupied/unreachable) instead of a bare
count. Tests: `flaky_probe_recovers_on_retry`,
`insufficient_nodes_error_names_the_reason` (→ `orchestrate/retry_tests.rs`).

## Bench config C — 166/184 (90/100, ★★★★½), --parallel 2 (28 Sep)

First two-stream run (`sm121 bench --parallel 2`, flag added today): 92
scenario streams over 2 connections, whole bench ~23 min (vs ~45 serial).
- Score 90 vs A 92 / B 91 — the -1 is within the `--parallel` band (the tool
  itself warns timeouts under server saturation can log as FAIL; TC-68 was
  exactly that shape). TC-74/92 look like real behavioral misses.
- Responsiveness 45 (median 3.4 s) vs 62/63 — stream contention, NOT GMU.
- Perf identical to 0.88: pp ~2,000, tg 37–41 c1 / 56–58 c2 — 0.90 costs
  nothing at short context.
- Config C verdict: keep. 512K is free at the bench level; only the quality
  reproducibility caveat says: quote 92 (config B) for the quality row, C is
  the production shape (512K, pool 570,752).

## mimo26-flash-mopd-4x — production now, bench 168/184 (91/100)

Contingency lane became the production lane (user's call): stopped 2x,
booted 4x on dgx1-4. GMU 0.94 stuck (0.92 booted identical clean — the
trial diff is page-cache margin, not a blocker). Pool 4,137,472; 515K probe
340 s (~1.5K t/s prefill).
- Bench mopd-4x (serial, seed 42): 168/184, Responsiveness 76 (median 1.4s
  — 2x was 62/2.1s), tg 56-90 t/s (2x: 37-58), pp 2.6-3.2K, TTFT halved.
  Fails: TC-43/68/90/92 — recurring set, same quality band as 2x.
- MiMo JIT cache mounts added (all 3 mimo recipes, live-container verified:
  everything lives under /root/.cache/sglang — 35 triton dirs, inductor,
  flashinfer; b12x-roce unused on TP<8): bind to ~/.cache/mimo26-jit/sglang
  on local NVMe. Takes effect next serve (production NOT restarted).

## RoCEnante on TP4 — a15 (28 Sep, ~15:00 UTC)

The vLLM lanes had RoCEnante live on 4 nodes since Sep-07 (b12x core takes
world 2..16); the sglang port's SG17 guard had it pinned to world==8. a15
relaxes the guard to the core's range (2..16, Gloo-exchange-group check and
2-HCA check intact) and `mimo26-flash-mopd-4x` turns it on with
SGLANG8_ROCE_ALLREDUCE=1 + GMU 0.88.
- Boot: `RoCEnante ready: world=4 hcas=rocep1s0f0,roceP2p1s0f0`, ROUTE fires
  on first decode all-reduce. Pool 3,677,056 @0.88.
- Perf vs the 0.94 NCCL run: tg d0 c1 75.9 (+20%), d4096 c1 76.3 (+36%),
  c2 +15%, d8192 flat, TTFT ~10% better, pp unchanged.

## Full bench on RoCEnante — 168/184 (91/100), quality-neutral (28 Sep)

roce-quality (serial, seed 42): score holds the band (168/184, fails only
TC-68/90/92 — TC-43 recovered vs the 12:18 run), Responsiveness 76→79
(median 1.2 s). tg: d0 74.3/c2 97.7, d4096 c2 96.7, d8192 c2 91.6 — decode
+10-15% over the NCCL baseline with pp/TTFT equal-or-better. a15+0.88+RoCE
is the production shape.