recipes/docker-images/mia-tp4/docs/fable-new.md

Decode tok/s on TP3: what to borrow from knapcio's TP4 profile, and what else to do

Date: 2026-09-24. Audited: knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4 at 4d8f4c0 (README, CHANGELOG, docs/session-20260919.md, docs/upstream-watch.md, every adapter, the vendored runtime/b12x), this repository at 25379c2 plus the uncommitted working tree, the live TP3 fleet (boot of 2026-09-24 02:12 local), logs/profile-2026-09-10/REPORT.md and the journals of spark1/spark3. Nothing was restarted or benchmarked for this document: the fleet was serving a dashboard's prefill sweep the whole time, and every number below is either theirs (labelled "TP4, knapcio"), ours from REPORT.md, or read from the current boot log.

1. Ranked list

#	Action	Expected on TP3	Effort	Risk
1	Re-pack the Engram shards on all three nodes (./start.sh pack). They are missing: the head's ~/dsv41-engram/ is empty, spark2 holds two TP2 shards (r1of2), spark3 has no directory. Every rank logs packed=False, 2 reads per lookup, and the workers read the checkpoint over NFS from the head.	REPORT §6/§8 measured this state at ~11 ms/step of rank-0 all-reduce wait; boot 2 → 3 (packing + b12x) went 120 → 98 ms/step. ≈ +10 % decode, and the head stops serving NFS during decode.	30 min, no code, no boot beyond the one you need anyway	none (spark2 needs its 101 GB of TP2 shards deleted first: 61 GB free)
2	Quiet spark3. It runs a desktop (graphical.target, gdm active), 13 other containers, sparkDash at 23 % CPU, and has 4.0–4.4 GB MemAvailable against spark2's 7.7 GB. It is the rank that ran out of memory tonight (journal 01:33:54, NVRM ... NV_ERR_NO_MEMORY) and, in REPORT §1, the rank whose spikes set the step.	Removes the straggler and the wedge risk; 2–5 % by REPORT's rank skew; the dashboard's 2-second GPU polling alone cost knapcio 0.6 ms/step.	30 min	low
3	Turn on the adapters already copied from knapcio, one switch per boot, at MEM_FRACTION_STATIC=0.95 (order in §4.1).	Greedy step ≈ −9 to −10 ms of 82 (≈ +12 %): wo_a fp8 twin ≈ −6.5 ms, draft LM head fp8 ≈ −0.9 ms, Engram prefetch ≈ −1.5 to −2.5 ms; sampled chat another +2–4 % accepted tokens (block verification, draft temperature).	1 boot each, ≈ 13 min per boot	tonight's boot enabled all of them at once at 0.98 and spark3 ran out of memory; the switches themselves are lossless by construction
4	DSPARK_BLOCK_SIZE=5 with DSV41_VERIFY_CAP=conf:0.1 (after 3).	Code +10–25 % (knapcio: k=3 costs −16 to −26 % on code on two fleets), prose 0 to +5 % (their cap: prose c1 66.0 → 69.7, c4 +6 %). The cap is what makes k=5 safe on prose, which is why k=3 was chosen here.	2 boots (k=5 alone, then with the cap)	none: greedy output identical, sampled goes through block verification
5	Port adapter/autotune_keep.py (88 lines, one hook). Our boot log shows the same per-rank caches disagree, discarding them twice per boot.	Ends the boot-to-boot tactic lottery FlashInfer plays under EP (knapcio: several % between boots, 26 re-tunes → 0) and removes two autotune passes per boot.	20 min	none
6	Unlock the GPU clocks. All three nodes run nvidia-clock-lock.service = nvidia-smi -lgc 0,2200; live SM clock 2177–2190 MHz, default applications clock 2418.	knapcio measured the 2200 cap at −1.5 % prose, −0.7 % code, −1.5 to −5 % prefill.	one command per node, no boot	thermal/power: whoever installed the lock had a reason; check nvidia-smi -q -d TEMPERATURE,PERFORMANCE under load
7	A/B EP_SIZE=1 (tensor-sharded experts instead of expert-parallel).	knapcio: EP4 → EP2 halved the per-layer straggler wait (NCCL 16.5 → 10.2 ms/step). At TP3 the measurable skew is ~2 ms (rank 0 NCCL 15.6 vs rank 2 13.4 ms), so ≤ 3 %. Bytes per rank are identical either way.	1 boot	the MXFP4 runner at intermediate 768 is untested here
8	Engram row cache in the GB10 display reservation (DSV41_ENGRAM_DRM_NODE, needs row_store.cpp + engram_backend.py from knapcio, nvidia-drm modeset=1, headless, reboot).	~1.8 GiB per node outside MemAvailable for one layer's row cache; on their traffic hit rate 67–99 %. Speed-neutral on TP4 once the prefetch is on; on TP3 it is the only way to have a row cache at all.	host change on 3 nodes + reboot	driver 580.x only (ours is 580.159.03, theirs 580.173.02)
9	RoCEnante (b12x one-shot RDMA all-reduce) for the TP all-reduces.	~88 SUM all-reduces per step at 70–100 µs median (REPORT §1); b12x measured 65 → 24 µs per 48 KB all-reduce on 4 Sparks. Ceiling ≈ −4 to −6 ms/step (5–7 %), larger than their TP4 gain because NCCL is 16 % of our step versus 1 % of theirs.	high: the published overlay stripes every peer across both HCAs with same-index HCAs, which cannot connect on our pairwise triangle; §6.6 has the patch shape	hang risk (b12x #313), a new transport inside the graphs
10	Canary image (upstream dsv4.1 branch at f80c91a4b, Dockerfile.canary).	TP4, knapcio: prose c1 51.6 → 55.4 (+7 %), c8 +11 %, code c16 +41 %; the branch's mHC/metadata kernels target the "~2000 tiny kernels, 10 ms" line of our budget.	high: new image, every TP3 hook (tp3_pad, mxfp8_b12x scale repair, flash_mla_sm120) re-validated	padded vocab under the branch's sharded argmax; sampled decode −5 % vs greedy on that image

Items 1, 2, 5 and 6 need no engine boot beyond the next one. Items 3 and 4 are eight boots at ~13 min each. Items 7–10 are separate projects.

2. Where a TP3 decode step goes

From REPORT §12 (boot 6, 2026-09-10; k=5, packed shards, b12x, NCCL trimmed; 82 ms mean, rank 0):

Component	ms	Notes
b12x MXFP8 dense GEMMs	~17	at M=6, ~79 µs per call
wo_a bf16 einsum (cutlass_80_wmma)	~11	4 groups × 1024 × 4096 bf16 = 32 MiB per layer, 43 layers = 1.34 GiB per step at ~135 GB/s
LM head bf16	~2	43,136 padded vocab rows per rank
MoE grouped MXFP4 GEMM	~27	bandwidth-bound, scales with the verify rows (6 at k=5)
NCCL, true cost	13–16	104 collectives across 3 nodes, 70–100 µs median per LL all-reduce
~2000 small kernels	~10	attention 2, hyper-connection mixers 2, quant/norm/MoE aux
Engram host callbacks	~1	with packed shards; 2.7 ms unpacked
CPU gaps	~4

Today's boot is not that state: k=3 (4 verify rows), shards unpacked, KV pin clamped (available_bytes=1.04 GB, max_total_num_tokens=491520 against the 750,000 pin), head MemAvailable 2.9 GB. The last two hours of Decode batch lines at one running request give a median gen throughput of 38 tok/s (p90 46.5, peak 50.0) at a mean accept length of 2.94 out of a maximum of 4 (k=3). The README's 37.9 tok/s prose c1 figure matches.

For scale, knapcio's TP4 production step is 41–47 ms on prose at k=5 with conf:0.1, prose c1 69.1, code c1 113.8. Their base image at the same k=5 was 51.6 / 96.7, so their adapters and profile are worth +34 % / +18 % on TP4. Not all of that transfers: two of their biggest levers are memory (16 request slots, 4 GiB row cache) and one is a TP4-only shape (§5).

3. What knapcio's repository is, and what is already here

It is a downstream fork of this repository (credits Mia for the launcher, Engram store, b12x routing, memory model), tuned for four Sparks with a switched RoCE fabric, and it keeps every change as an env-gated import hook in adapter/sitecustomize.py, the same pattern as ours. Production stack as of 2026-09-23 night, with their measured deltas:

Their lever	Measured by them (TP4)	On TP3
EP_SIZE=2 instead of 4	NCCL 16.5 → 10.2 ms/step	EP=1 is our analogue, §6.5
DSV41_CACHE_GIB=4, 16 ways	67–99 % Engram hit rate	no memory for it; §4.2 item 8
MAX_RUNNING_REQUESTS=16	adds a c16 tier, +64 % aggregate over c8	16 graphs cost 5.6 GB; we have 4
DSPARK_BLOCK_SIZE=5	k=3: prose 33.8 vs 44.6, code 80.4 vs 102.7	with the verify cap, yes
DSV41_SHARED_PAD_K=1	−0.9 ms/step	not needed: shared expert K is 768 at TP3 (2304/3), already a multiple of 128; the boot log has no b12x rejection
DSV41_INDEXER_CHUNKED=1 + chunk 4096 (+ prefill TP split)	prefill 128k 3499 → 4364; decoders during a long prefill 5.7 → 7.8 tok/s	prefill; secondary
DSV41_ENGRAM_PREFETCH=1	step 51.7 → 49.5 ms, prose c1 57–59 → 60–61	copied, off
DSV41_WO_A_W8=1 (+ MID, DROP)	43 layers 3.43 → 2.20 ms; step 52.9 → 51.7; MID c4 82.1 → 78.3 ms verify step; DROP frees 722 MB/rank	copied and adapted for TP3, off
DSV41_DRAFT_HEAD_FP8=1	1405 → 721 µs; step 51.7 → 50.9	copied, off
DSV41_DRAFT_TAU=0.8	+1.2 % accepted tokens (sampled)	copied, off
DSV41_BLOCK_VERIFY=1	+1.8 % prose / +2.5 % coding-with-thinking accepted (sampled)	copied, off
DSV41_FOLDED_FENCE=1	closes the sglang#40919 race	copied, off
DSV41_VERIFY_CAP=conf:0.1	prose c1 66.0 → 69.7, c4 +6 %, sampled thinking +5 %; fixed caps lose 8–40 %	copied, off
DSV41_AUTOTUNE_KEEP=1	26 re-tunes per boot → 0	not copied
RoCEnante (Dockerfile.canary-roce)	+3 % prose c1, +8 % structured, c16 +4.5 %	not copied, needs a triangle patch
canary image (dsv4.1 at f80c91a4b)	prose c1 +7 %, code c16 +41 % over base	not copied
DSV41_FAST_LOAD=1	boot 343 → 124 s, KV pool −3 to −13 %	not wanted here (§5)
display-reservation row cache	+1.8 GiB headroom per node	not copied
--sleep-on-idle, --enable-cache-report, --min-free-slots-delay 1	idle CPU 47 → 14 %; no decode change	ours already / n.a. below 8 slots

Already in the working tree (uncommitted, 2026-09-24 00:52): block_verify.py, draft_head_fp8.py, draft_tau.py, engram_prefetch.py, folded_result_fence.py, verify_cap.py byte-identical to knapcio's; wo_a_w8.py adapted for TP3 (the twin is [G, 1024, 4096] with G = 4 local o_groups after the 8 → 12 pad, and an "einsum bridge" because the dev-dsv41 image has no wo_a_bf16_small_batch kernel and dispatches wo_a from _apply_wo_a_bf16_matmul); the matching tests; sitecustomize.py, start.sh and the Dockerfile hooks. start.sh forces every switch off on TP3 with the comment that turning them on "made spark3 drop its weights at FlashInfer autotune". The journal agrees on the what but not on the why: the head container started at 01:32:16, spark3's kernel logged NV_ERR_NO_MEMORY from kgrctxAllocCtxBuffers at 01:33:54 (a GPU context allocation failing for lack of memory, 98 s into the weight load), spark3 was rebooted at 01:49, and the .env note says the fraction was 0.98 for that boot. That is memory exhaustion on the node with the least headroom, with every switch on and 2 % of RAM left for everything outside the static pool. It does not implicate any single adapter. §4.1 gives the order that separates them.

4. Borrow list, item by item

4.1 Already copied: what each does on TP3, and the order to enable them

Each is a DSV41_* variable forwarded by start.sh to all three containers; each refuses to arm if the engine symbol it patches has drifted, and ./start.sh doctor prints the switch line. Keep MEM_FRACTION_STATIC=0.95 (the working value; 0.98 is what tonight's boot ran). One switch per boot, smoke + a 4-prompt greedy batch + one sampled prompt after each, and Decode batch lines from an idle fleet (#running-req must equal the concurrency being measured).

1. DSV41_FOLDED_FENCE=1. Correctness only: clones the six AcceptOuts tensors on the folded (all-greedy) verify path before the overlapped D2H copy. Free. Start here because it touches no weights.
2. DSV41_DRAFT_TAU=0.8. Scales the draft proposal temperature for sampled requests; the same tensor drives proposal and acceptance, so the output distribution is unchanged. +1.2 % accepted tokens at T=1 / top_p 0.95 on their traffic. Invisible on greedy benches.
3. DSV41_BLOCK_VERIFY=1. Block verification (Sun et al., ICLR 2025) for sampled rows: exact output distribution, +1.8 % (prose) / +2.5 % (coding with thinking) accepted tokens per step. Invisible on greedy benches. Together with 2 this is the +2–4 % real-chat gain.
4. DSV41_DRAFT_HEAD_FP8=1. The draft's block logits come from an fp8 copy of the shared LM head (target keeps bf16). At TP3 the head is 43,136 × 5120 bf16 = 442 MB per rank read once per step (≈ 1.9 ms), so the fp8 copy saves ≈ 0.9 ms/step. Memory: +221 MB per rank for the copy.
5. DSV41_WO_A_W8=1 alone (MID and DROP off). After load it builds an fp8 twin (e4m3 + one exponent per 32×32 block) of each bf16 wo_a from the checkpoint's own fp8 bytes and keeps it only when the dequantized twin equals the bf16 weight bit for bit; the 2–8-row verify/draft path then runs the Triton kernel on the twin. This is the biggest single item on TP3 because our wo_a is twice theirs: 4 groups per rank instead of 2 (the o_groups 8 → 12 pad), so 1.34 GiB per step at ~135 GB/s ≈ 11 ms (REPORT §12). Half the bytes at their measured 164 GB/s ≈ 4.3 ms: ≈ −6.5 ms/step. Rank 2's four groups are all zero, so its twins are exact trivially. Memory during the check: +16 MiB per layer of twin next to the bf16 copy (+0.7 GB per rank until DROP).
6. DSV41_WO_A_W8_MID=1. The 9–192-row kernel for verify/draft rows at c2–c4 (their verify step c4 82.1 → 78.3 ms, c8 120.0 → 118.1 ms). Our bridge only takes it during decode/verify/draft-extend forwards, so short prefills stay on einsum. Worth +3–5 % per stream at c4.
7. DSV41_ENGRAM_PREFETCH=1. Forks a side stream right after EngramHasher.forward, runs the id copy, the host row lookup and the row copies for both Engram layers there, and joins at each layer's gather; inside the CUDA graph that is a fork/join with a host node. Their step 51.7 → 49.5 ms. Ours: the two callbacks are ~1 ms/step packed (2.7 ms unpacked); layer 1's lookup has ~1 ms of GPU work to hide behind and layer 14's ~15 ms, so expect −1.5 to −2.5 ms/step. Pinned staging buffers are allocated at the first forward, before capture. DSV41_ENGRAM_PREFETCH_CHECK=1 re-runs the old path and counts differing gathers (they saw 0).
8. DSV41_WO_A_W8_DROP=1 last. After a second bit-identical check, the bf16 wo_a becomes a zero-stride view of a one-element tag and every call is served from the twin; prefill dequantizes into a fresh 32 MiB buffer per call. On TP3 this releases 43 × 32 MiB = 1.34 GiB per rank, which on the head is more than the entire current KV budget (1.04 GB). Enable it alone on its own boot and read the DSV4 memory calculation ... available_bytes= line: that is the one adapter that moves the KV pool, and the one the start.sh comment blames. If the boot is clean, the freed memory can go to the pin (it is clamped today) or stay as head headroom, which REPORT §6 and the 2026-09-10 session showed is worth more than any kernel when it drops below ~1 GB.

Then, with all of the above on:

1. DSPARK_BLOCK_SIZE=5 on its own boot (baseline the prose/code c1 and c4 numbers against k=3: the README's "22.3 → 25.4 prose at k=3" predates the packed shards, b12x and the fp8 kernels, and knapcio's same measurement went the other way). The boot log's DSpark gamma mismatch ... draft config block_size=5 line says the draft was trained for 5.
2. DSV41_VERIFY_CAP=conf:0.1 with k=5. Per request and step, only the leading drafts whose running product of the draft confidence head's survival stays ≥ 0.1 are verified; the dead rows' router ids are overwritten with the anchor row's (one in-graph Triton launch per layer, 38 µs/step for 43 layers), so they add no expert reads, and acceptance is capped through the engine's own cutoff. Exact. Their gain is 5–6 % on prose at c1/c4 and +5 % on sampled thinking traffic; ours should be at least that, because the MoE is 33 % of our step versus ~25 % of theirs and a dead row on TP3 costs more expert bytes. DSV41_VERIFY_CAP_LOG=<file> records confidences per step on rank 0 (costs a sync) if the threshold needs re-fitting for TP3; they found per-position thresholds worth only +0.4 % over the single 0.1. Note DSV41_VERIFY_CAP_STRIDE defaults to DSPARK_BLOCK_SIZE + 1, so it follows k.

Expected sum, greedy prose c1: 82 → ≈ 72 ms/step from items 4–7 (+12–14 %), plus whatever k=5 + cap gives on top; code c1 +10–25 % from k=5 alone. Sampled chat: +2–4 % more from 2–3.

4.2 Not yet copied, worth taking

• adapter/autotune_keep.py (DSV41_AUTOTUNE_KEEP=1). Hooks sglang.srt.model_executor.runner.flashinfer_autotune; our image has the same _autotune_cache_digest / flashinfer_autotune_context / flashinfer_autotune_cache_path symbols (checked in the running container). Replaces the per-rank byte digest of the cache (which can never agree under EP, so the cache is deleted every boot and the MoE tactics re-drawn from timing noise, sglang#40320) with the load decision plus a launch fingerprint sidecar. Copy the file, add the hook and the module name to the finder list in sitecustomize.py, forward the variable in start.sh. Note their _VOLATILE list already excludes the verify-cap/draft switches from the fingerprint so A/B boots share tactics.
• DSPARK_BLOCK_SIZE=5 + conf:0.1: above.
• row_store.cpp / engram_backend.py display-reservation backing (item 8 in §1): 46 lines of C++ (row_store_next_cache_on_drm, a DRM dumb buffer on /dev/dri/card0) and 11 lines of Python; the diff against ours is otherwise empty. Only useful once the host is headless with nvidia-drm modeset=1 fbdev=0; going headless alone raised their KV pool by ~1 M tokens.
• Measurement practice (docs/session-20260919.md, README "Measurement notes"): bench from a worker, discard the first two points after a boot (cold row cache, FlashInfer tuning during the first run), check #running-req for foreign traffic, quote usage.completion_tokens / elapsed or the gen throughput field, never SSE chunk counts, and set the dashboard's GPU poll to 10 s (POLL_INTERVAL_GPU=10000, POLL_INTERVAL_BANDWIDTH=10000). Their scripts/qeval.py (75 auto-scored tasks, McNemar pairing) is the quality gate to run after the kernel switches; it executes model-generated Python, so run it from a worker.
• runtime/roce_tp4_adapt.py + runtime/sglang-rocenante.patch are the starting point for §6.6, not usable as they are.

4.3 Their own dead ends, so we do not repeat them

Draft fine-tune on own traffic (+3.4 % held-out, 0 % on new prompts), multi-pass drafting (+2–4 % accepted for +9 % step per pass), expert-sharing routing (changes routing, step +7 %), n-gram lookup (+1 % even as an oracle), relaxed acceptance (not exact, +20 % accepted only), split-K MXFP8 for the small-M projections (1.5–3× slower than b12x), hyper-connection kernel variants (23–28 µs whatever the layout), Markov W2 unsharded (+0.6 ms/step), parallel Engram misses in row_store.cpp (nothing; the side-stream prefetch is what removed the gap), RoCE all-gather port (all-gathers are 0.56 % of their step), fixed verify caps (−8 to −40 %), per-request adaptive block size (+4 % code at best), chunk 1024 on TP4 (prefill −34 to −38 % and decoders behind a prefill got worse), sglang#39704 (±2 % below c32), newer dsv4.1 heads (SM121 loses the candidate indexer after #39671), NVFP4 experts (no bandwidth saved on GB10, +16 GiB), LuZ-0.1.7 fused kernels (few %, not bit-exact, version-bound), DSpark SPS table (crashes the Engram path on this model, which closes the "profile the SPS table" item in our own REPORT §2).

5. Not worth borrowing on TP3, and why

Their item	Why not here
MAX_RUNNING_REQUESTS=16	16 decode graphs cost them ~5.6 GB; the head has ~3 GB free while serving
DSV41_CACHE_GIB=4	the 2026-09-10 session measured the last GiB of head RAM at 2.4× throughput (12.4 → 30.1 tok/s) when the row cache was turned off; only the display-reservation variant fits
DSV41_SHARED_PAD_K	TP4-specific: K = 2304/4 = 576 is not a multiple of 128; at TP3 K = 768 is, and the boot log shows no b12x rejection
DSV41_FAST_LOAD	reads the checkpoint eagerly into pinned host RAM; costs 3–13 % of the KV pool, and our pool is already clamped by the head's 1.04 GB budget
EP_SIZE=2 literally	3 is not divisible by 2; the analogue is EP_SIZE=1 (§6.5)
CHUNKED_PREFILL_SIZE=4096	our long-prompt peak scales with chunk × prefix (docs/chunked-prefill-memory.md); 1024 is what fits 208k on the head
--min-free-slots-delay 1	the admission delayer only engages at ≥ 8 slots
switchless ring, NCCL_SWITCHLESS_RING_ONLY	four-node ring only; our three nodes are a full triangle and boot on the defaults

6. Beyond borrowing: what this audit found on the TP3 fleet

6.1 The Engram shards are gone (the largest item on the list)

Current boot, rank 0, one minute of stats: Engram layer=1 lookups=43128 hit_rate=0.0% reads=86256 cache=0.0GiB ... packed=False; the same on TP1 and TP2. Two reads per lookup means weight and scale are read from the checkpoint tensors, and on the workers the checkpoint is the NFS export from spark1. On disk: ~/dsv41-engram/ on the head is empty (mtime 2026-09-17 22:09:49), ~/dsv41-3x-spark/engram/ on spark2 and spark3 are empty with the same mtime, and spark2's ~/dsv41-engram/ holds engram-l1-r1of2.bin / engram-l14-r1of2.bin from a TP2 experiment on 2026-09-12 (101 GB). The loader looks for engram-l{layer}-r{rank}of{tp}.bin under DSV41_PACKED_DIR, so nothing matches.

REPORT §6 traced exactly this state to ~11 ms/step of rank-0 waiting at the all-reduce that follows each Engram gather (35–43 % of its NCCL time), and §8 recorded the fix: step 120 → 98 ms together with b12x, Engram wait 11 → 0.5 ms. The 2026-09-10 session also noted that ./start.sh build's rsync --delete used to sweep the workers' shards; the README now says engram/ is excluded, but something on 2026-09-17 at 22:09 (the night of the memory-guard trip in logs/oom-guard.log) emptied all three directories at once. Whatever it was, ./start.sh pack rewrites them (~63 GiB per node, ~10 min per node, .partial then rename, safe while serving; picked up at the next boot). spark2 has 61 GB free against 63 GiB needed: delete its two TP2 shards first. Confirm with packed=True in every rank's Exact nvme Engram line.

6.2 GPU clocks are locked at 2200 MHz on every node

nvidia-clock-lock.service (nvidia-smi -lgc 0,2200) is active on spark1/2/3; the SM clock reads 2177–2190 MHz under load against a default applications clock of 2418 and a maximum of 3003. knapcio measured the same 2200 cap on a bandwidth-bound step at −1.5 % prose decode, −0.7 % code, −1.5 to −5 % prefill. nvidia-smi -rgc on each node restores the default without a boot; keep it only if temperatures or power under a sustained bench say otherwise (they reported 57–59 °C uncapped at 2509–2554 MHz).

6.3 spark3 is the weak rank

graphical.target with gdm active, 13 non-serving containers (sparkDash at 23 % CPU, autoclip ×5, redis, postgres, local-llm, model-trader ×2, slate, nexusui), load average 12 in the five minutes after its reboot, 4.0–4.4 GB MemAvailable against spark2's 7.7 GB. REPORT §1 already named spark3's spikes as what sets the step (the slowest rank does), and tonight it was the rank that ran out of memory. Moving sparkDash and the rest to spark2 (idle apart from the worker) or to a fourth machine, and systemctl set-default multi-user.target on spark3, is the cheapest straggler fix on the list. If the dashboard stays, raise its GPU/bandwidth poll interval to 10 s; it also polls /server_info, /v1/loads, /model_info and /v1/models on the head every 2–3 s.

6.4 What tonight's failure was, and how to run the retry

Timeline from the journals: adapters copied 00:52; head container started 01:32:16 with every switch on and (per the .env note) MEM_FRACTION_STATIC=0.98; spark3 kernel NV_ERR_NO_MEMORY from kgrctxAllocCtxBuffers at 01:33:54; spark3 clean-rebooted 01:49–01:50; head restarted 01:52 and again 02:12 with the switches off. 0.98 leaves 2.4 GB per node for the CUDA context, the tokenizer/detokenizer on the head, NCCL's 139 MB, the row-store threads and the desktop on spark3; the adapters add ~0.9 GB per rank transiently (wo_a twins next to the bf16 copies until DROP, the fp8 draft head, prefetch staging). The start.sh comment's mechanism (DROP's one-element stand-in while the autotune target-verify forward runs the prefetch and the remap) is possible but unproven; the memory arithmetic is sufficient on its own. Retry at 0.95, one switch per boot in the §4.1 order, DROP last, spark3 quiet first.

6.5 EP_SIZE=1: the TP3 analogue of their EP2

At EP=3 each rank holds 128 of the 384 routed experts at full width and runs only the tokens routed to them; the per-layer straggler is the rank with the most routed work. At EP=1 with TP=3 each rank holds all 384 experts at one third of the intermediate width (768 of 2304): the same bytes per rank, every rank does identical work, no straggler, and still one post-MoE all-reduce (REPORT §4 established that EP=3 without an all-to-all backend already does "input replicated, local experts, one all-reduce"). knapcio's EP4 → EP2 took 6 ms/step off their NCCL time for that reason. Our measurable skew is smaller (rank 0 waits ~2 ms more than rank 2 spends), so the upside is ≤ 3 %; the cost is more, smaller grouped GEMM problems per layer (up to 36 distinct experts at 768 wide instead of ~11 at 2304). One boot answers it. tp3_pad pads the draft's 128 experts to 129 keyed on TP, not EP, so it behaves as today.

6.6 RoCEnante on a three-node triangle: what would have to change

The b12x runtime (runtime/b12x/b12x/comm/roce/) opens every HCA listed in B12X_ROCE_HCA, creates one QP per (HCA, peer), and in connect_qp connects HCA h of rank i to HCA h of rank j using the peer's GID at the same index; post_op then stripes every peer's payload across all HCAs and writes one completion flag per (peer, HCA), and the GPU kernel waits for every flag. That assumes a switch where each rail reaches every peer. Our fabric is rocep1s0f1 10.0.22.1 → spark2, rocep1s0f0 10.0.23.1 → spark3, spark2 ↔ spark3 on 10.0.33.0/24: each HCA reaches exactly one peer, and the same-index pairing is not even guaranteed to be the cabled pair. The 2026-09-14 session's note that "each peer gets its own dedicated rail" is the right topology but not what the code does.

The patch is contained: (1) at roce_create/connect time, choose for every peer the local HCA whose GID shares the peer's subnet (/24 on the IPv4-mapped GID) and the peer's HCA on that link, instead of index-equal pairing; (2) in post_op, send one stripe per peer on that HCA and write both flag words for that peer over the same QP, so the kernel's wait loop is untouched; (3) drop the len(hca_names) != 2 warning path in the overlay. Then rhys101's SG17 overlay (214-line sglang-rocenante.patch, plus knapcio's roce_tp4_adapt.py relaxing world size to 4/8) needs world size 3 allowed and SGLANG_ROCE_MAX_SIZE sized for the 4-slot step (4 × 6 × 5120 × 2 B = 240 KB, inside the 512 KiB default). Ceiling on our profile: ~88 SUM all-reduces per step (40 wo_b, 40 post-MoE, 2 Engram, 6 in the draft) at 70–100 µs → ~25–30 µs each, i.e. −4 to −6 ms of an 82 ms step; the 6 all-gathers and 8 broadcasts stay on NCCL. That is a 5–7 % lever that costs a new transport inside the CUDA graphs and the open b12x #313 wedge report; the result-boundary health check in the overlay is the mitigation and SGLANG_ROCE_ALLREDUCE=0 the escape hatch.

6.7 Head memory and the KV pin

available_bytes=1.04 GB this boot, so the 750,000-token pin was silently clamped to 491,520 (the same silent clamp REPORT §3 describes). Two things on this list give the head memory back without touching kernels: DROP (1.34 GiB per rank of bf16 wo_a, §4.1 item 8) and the display-reservation row cache (1.8 GiB per node outside MemAvailable). Spend it as headroom first: the 2026-09-10 session measured the head's last GiB at 2.4× throughput, and REPORT §14 put the decode-side cost of a bigger pool at zero. A bigger pin only matters for long contexts.

6.8 Things that look like levers and are not

• Uneven head split (32/16/16 local heads, no padding). The SM120 sparse-MLA kernel only exists for 8/16/32/64/128 local heads, and the step is set by the rank with the most heads, which would still be
  1. The padding costs ranks 0 and 1 nothing extra on the critical path; the true cost of TP3 is 32 heads per rank instead of 21.3, and only a 24-head kernel would recover it.
• Skipping rank 2's all-zero projections. Rank 2 would finish earlier and wait; the step is ranks 0/1. Giving rank 2 more experts to use that slack is out: the experts are ~90 GB of each rank's ~104 GB.
• Markov W2 at TP3. 43,136 × 256 bf16 = 22 MB per rank against the 25.2 MB L2; their pricing shows the read collapsing to DRAM speed once it no longer fits. Worth ~0.3–0.5 ms/step if it is falling out here; not worth a change until the bigger items are in.
• SGLANG_SPEC_TP_SYNC=all. 13 tiny syncs, ~1 ms/step (REPORT §1). Not worth the correctness risk.

7. Boot plan and measurement protocol

Before any boot (no engine restart needed): delete spark2's TP2 shards, ./start.sh pack on all nodes, quiet spark3 (or at least the dashboard's poll interval), decide on the clock lock, port autotune_keep.

Boot	Change	Record
A	packed shards, k=3, everything else as today	packed=True ×3 ranks; available_bytes; prose/code c1 and c4 via benchmarks/decode_window.py or sparkDash from spark2; 40-step profile (scripts/profile/profile_decode.sh) for the new budget
B	+ DSV41_FOLDED_FENCE=1 DSV41_DRAFT_TAU=0.8 DSV41_BLOCK_VERIFY=1 DSV41_AUTOTUNE_KEEP=1 (no weight-touching switches)	boot clean; greedy text identical to A; sampled c1 tok/s at T=1 / top_p 0.95 vs A
C	+ DSV41_DRAFT_HEAD_FP8=1	step probe; acceptance unchanged
D	+ DSV41_WO_A_W8=1	[wo_a_w8] lines: 43 of 43 exact twins on every rank; step −6 ms expected
E	+ DSV41_WO_A_W8_MID=1	c2/c4 per-stream
F	+ DSV41_ENGRAM_PREFETCH=1 (_CHECK=1 for this boot only)	differing gathers: 0; step −2 ms expected
G	+ DSV41_WO_A_W8_DROP=1	available_bytes up by ~1.3 GB; greedy text unchanged after the first request
H	DSPARK_BLOCK_SIZE=5	prose and code c1/c4 vs G
I	+ DSV41_VERIFY_CAP=conf:0.1	prose c1/c4, sampled thinking, spec_verify_ct; qeval from a worker
J	EP_SIZE=1 (against I)	step, NCCL share, #running-req clean

Rules that their and our measurements both insist on: idle fleet and #running-req checked in the log; first two points after a boot discarded; the same sparkDash/decode_window.py prompts every time; greedy for kernel A/Bs (sampling costs ~5 % and adds noise), sampled T=1 / top_p 0.95 for the draft-side switches; usage.completion_tokens / elapsed or gen throughput, never SSE chunks; the memory guard armed for anything that changes memory; and no long prompts in the boot tests.