overnight-progress.md 22 KB
Overnight TP3 campaign — progress log
Started 2026-09-24 03:11 IDT (00:11 UTC). Window ends ~11:11 IDT; validation/restore reserve from ~10:25.
Raw evidence: results/overnight-20260924-0311/.
Benchmark: RESULTS_DIR=… scripts/overnight/bench.sh LABEL 5 512 — runs
benchmarks/overnight_bench.py from spark2 against the head,
5 reps per workload after a discarded warm-up, 512 max tokens, thinking off. Tokens from
usage.completion_tokens; decode tok/s = (tokens-1)/(last delta - first delta). Acceptance and
step time joined from the head's Decode batch lines (step = accept_len x bs / gen throughput).
Constant background during every measurement: sparkDash on spark3 polls the head API (5 GETs every 2 s, 0.06 % CPU). Pausing it was not permitted by the session's safety policy, so it stays on.
0. Starting state (03:11-03:25)
- Configs, container env/mounts for all three ranks, adapter hashes, git diff: saved in the results dir.
- Running: TP3/EP3, DSpark k=3, 4 slots, chunk 1024, KV pin 750k (effective pool 491,520), fraction 0.95, b12x MXFP8, cache 0 GiB, adapters off, packed=False on all ranks (engram dirs empty).
- Mixed SGLang builds across ranks. head and spark2 run base
lmsysorg/sglang:dev-dsv41=37939c26(2026-09-10; public on Docker Hub aslmsysorg/sglang@sha256:3dbc3130…, whose amd64 member is the 0xSero pin; its internal labels saylocal/sglang:dev, version 0.0.0.dev0); spark3's tag of the same name points at381b27ff(2026-09-11,da64c5cbb). 40 SGLang source files differ, among themmodels/deepseek_v4.py,deepseek_v4_dspark.py,dspark_worker_v2.py, MoE layer/topk, fp8 quantization and the memory pool; spark3 also has 9 extra modules (e.g.wo_a_bf16_small_batch.py,all_reduce_fusion.py). Rank 2 therefore runs a different model implementation than ranks 0/1. Likely contributors: per-rank autotune-cache disagreement, thewo_a_w8adapter selecting different paths per rank (it keys on the presence of the newer kernel module), and greedy nondeterminism. - Disk: head 378 GB free, spark2 61 GB free (needs 67.6 GB for its TP3 shards), spark3 392 GB free.
spark2 holds 95 GB of old TP2 shards at
/home/zurih/dsv41-engram(outside the worker mount).
b0 — original configuration (mixed builds, unpacked)
| workload | decode tok/s median (min-max) | TTFT | accept len | step ms |
|---|---|---|---|---|
| C1 prose | 29.28 (27.68-29.51) | 0.30 s | 2.17 | 73.2 |
| C1 code | 44.65 (42.39-44.79) | 0.30 s | 3.27 | 74.0 |
| C1 sampled chat | 28.87 (26.65-29.72) | 0.28 s | 2.23 | 77.8 |
| C4 aggregate (2 prose + 2 code) | 63.78 (60.18-67.14), per-stream 19.46 | 2.50 | 133.8 (bs 4) |
Greedy prose produced 4 distinct outputs out of 5, greedy code 3 of 5: not deterministic at T=0.
1. Packed Engram restored, rank images converged (03:25-03:48)
- Fleet drained at 03:24 (last foreign chat request 02:29).
- spark2 disk: old TP2 shards
engram-l1-r1of2.bin+engram-l14-r1of2.bin(101,379,729,008 bytes) copied to spark1:/home/mia/archive/dsv41-engram-tp2-spark2/, SHA-256 identical on both sides (SHA256SUMSstored there), then removed from spark2 (/home/zurih/dsv41-engram/MOVED.txtpoints to the new location). spark2 free space 61 -> 156 GB. - Packing:
pack_engram.pyfor ranks 0/1/2 into the real mounts (head~/dsv41-engram, workers/home/zurih/dsv41-3x-spark/engram); 92-111 s per 31.5 GiB layer. Free afterwards: head ~310 GB, spark2 93 GB, spark3 329 GB. - Image convergence: spark3's images were preserved as
lmsysorg/sglang:dev-dsv41-da64c5cbb-20260911anddsv41-3x-spark:mixed-da64c5cbb-20260924; the head'slmsysorg/sglang:dev-dsv41(37939c26) wasdocker save | docker loaded onto spark3 and the overlay rebuilt on all three nodes (identicaldeepseek_v4.pyhash everywhere). - Verified after restart:
packed=Truefor layers 1 and 14 on ranks 0, 1 and 2 (six checks). - The autotune "per-rank caches disagree" message still appears twice per boot with identical images: it is the EP-structural problem, not the version mismatch.
- KV pool on this boot: 226,816 tokens (was 491,520). The head had 1.35 GB less free after the draft load than the previous boot (the boot started with ~86 GB of page cache from archiving/packing). Needles at 96k still pass; tracked per boot below.
B1 — converged + packed (new baseline)
| workload | decode tok/s median (min-max) | TTFT | accept | step ms |
|---|---|---|---|---|
| C1 prose | 29.40 (29.19-29.48) | 0.29 s | 2.12 | 72.1 |
| C1 code | 45.68 (44.12-46.01) | 0.29 s | 3.45 | 72.9 |
| C1 sampled chat | 28.73 (28.19-30.99) | 0.26 s | 2.16 | 73.1 |
| C4 aggregate | 66.88 (66.23-70.15), per-stream 20.69 | 2.46 | 131.5 |
Quality gate: 8/8 tasks, 8/8 under 4-way concurrency (all identical to C1 outputs), greedy repeat (256 tok) 1 distinct of 3, needle at 28.7k and 95.5k prompt tokens found (17 s / 68 s). Greedy code 1 distinct of 5 (was 3 in b0); greedy 512-token prose still 5 distinct of 5.
Packing gave +2 % code and +5 % C4, prose unchanged: on this boot Engram is no longer on the critical path (profile below), but the step was already ~72 ms before.
2. Profile of B1 (rank 0, 44 decode steps, C1 prose)
Step wall 71.3 ms median, GPU busy 67.6 ms. Per step:
| kernel | ms/step | calls/step |
|---|---|---|
| MoE grouped GEMM (cutlass GroupProblemShape, MXFP4) | 23.8 | 88 |
| b12x dense block-scaled MXFP8 GEMM | 17.0 | 234 |
| NCCL all-reduce (LL ring) | 12.9 | 92 |
bf16 wmma GEMM = wo_a einsum (1/layer, 43+3) |
11.5 | 46 |
| mHC, sparse MLA, router, quant, norms, rest | ~6 | |
| Engram-following NCCL wait | 0.7 (5 % of NCCL) |
wo_a is the largest single-op target, as the plan predicted; Engram waiting is gone.
3. Autotune-cache retention ported (adapter/autotune_keep.py)
Copied from knapcio 4d8f4c0 with its test, hooked in sitecustomize.py, DSV41_AUTOTUNE_KEEP
forwarded on head and worker launches (also DSV41_ENGRAM_PREFETCH_CHECK, and
SGLANG_DSPARK_FOLDED_SAMPLING only when set, so unset stays the engine's AUTO).
Fix added: the engine's cache key omits block size, graph sizes and adapter switches, and when no
rank has a matching sidecar every digest is "" -> the stock gate sees agreement and FlashInfer loads
each rank's stale file. The port now deletes a rank's cache when its sidecar does not match the
launch, so a boot either reuses matched caches on all ranks or retunes from scratch. Test extended.
Enabled from E1 on (its no-sidecar path is equivalent to the stock discard); reuse is checked on a
matched repeat boot.
4. wo_a FP8 twin (adapter/wo_a_w8.py, einsum bridge on dev-dsv41)
E1 — W8=1, MID=0, DROP=0: failed at boot (04:13). The 40 target twins (+3 draft) add ~688 MiB
and the head was already at the margin: ValueError: Loaded weights leave no GPU memory for the KV
cache under --mem-fraction-static=0.95 ... minimum viable = 0.9503. Twin-only cannot run at 0.95;
raising the fraction is what took spark3 down at 01:32, so not retried. Log: boot-e1-fail-head.log.
E1b — W8=1, DROP=1 (04:27): booted, but acceptance collapsed to 1.0. C1 prose/code/chat 14.3/14.1/14.3
tok/s, C4 26.6. The bf16 copies were released (40 target = 1280 MB, 3 draft = 96 MB per rank) and
the KV pool rose to the 750k pin (749,824; memory calculation 841,472 before the pin). Target output
stayed coherent. Cause (source): dev-dsv41's draft attention (deepseek_v4_dspark.py:354) runs its
own torch.einsum on self.wo_a.weight and never reaches _apply_wo_a_bf16_matmul, so DROP handed
it the 1-element zero tag: every draft was garbage. The draft twins are also unused on this image.
Fix: in bridge mode the adapter now leaves the draft's wo_a alone (no twin, no drop).
A profile of E1b validated the target dispatch: _wo_a_partial_w8 runs 40x/step for 3.47 ms total,
replacing ~10 ms of bf16 wmma einsum on those layers (about -6.5 ms/step), and the draft's three
einsums on zero-stride tags took 3.78 ms.
E1c — W8=1, DROP=1 on target layers only, MID=0 (05:04)
KV pool 749,824 (pin; calculation 792,576). Head memory-stall time during the whole benchmark 2.8 ms.
| workload | decode tok/s (min-max) | vs B1 | accept | step ms |
|---|---|---|---|---|
| C1 prose | 28.97 (28.82-29.02) | -1.5 % | 1.98 | 67.9 |
| C1 code | 49.58 (49.39-49.85) | +8.5 % | 3.45 | 69.3 |
| C1 sampled chat | 31.34 (30.13-32.21) | +9.1 % | 2.20 | 68.9 |
| C4 aggregate | 61.34 (59.96-65.40), per-stream 18.85 | -8.3 % | 2.40 | 146.7 |
The bs=1 step fell ~4 ms (72 -> 68-69 ms). Prose tok/s did not follow because its greedy text changed (the twin's fp32 accumulation order differs from einsum), and this prompt's acceptance moved 2.12 -> 1.98; prose outputs were already non-repeatable in B1. C4 regressed as predicted: at k=3 a 4-request verify is 16 rows, which without MID falls to the per-call dequantize + einsum path.
E2 — W8 + DROP (target) + MID (05:24) — promoted
KV pool 749,824 (pin). Autotune: "tuned and saved" on each boot (no sidecar yet, as designed).
| workload | decode tok/s (min-max) | vs B1 | accept | step ms |
|---|---|---|---|---|
| C1 prose | 28.86 (28.40-29.05) | -1.8 % | 1.98 | 68.3 |
| C1 prose2 (added here) | 29.56 (29.50-29.94) | 2.00 | 66.5 | |
| C1 code | 48.04 (47.94-48.11) | +5.2 % | 3.29 | 68.9 |
| C1 sampled chat | 31.96 (30.76-32.53) | +11.2 % | 2.20 | 68.8 |
| C4 aggregate | 69.64 (68.88-72.72), per-stream 21.72 | +4.1 % | 2.44 | 125.6 |
Quality gate: 8/8, 8/8 concurrent (7 of 8 byte-identical to C1; one differs under batching), greedy repeat 1 distinct of 3, 28.7k needle found. MID restored C4 (146.7 -> 125.6 ms step; below B1's 131.5).
Measurement note. Greedy text is repeatable within a boot for code but differs between boots
(B1, E1c and E2 each produced a different code text, see text_sha): FlashInfer re-draws its tactics
every boot, so the numerics and hence the text move, and acceptance moves with the text
(code 3.45 in E1c vs 3.29 in E2 with identical kernels at bs=1). Cross-boot C1 tok/s therefore carries
a few % of text noise; bs=1 step time (72.1 -> 68.3-68.9 ms) is the cleaner signal. Autotune
retention should reduce this on repeated boots of one configuration.
5. DSpark block size
E3 — E2 + DSPARK_BLOCK_SIZE=5 (05:47)
| workload | decode tok/s (min-max) | vs E2 (k=3) | accept | step ms |
|---|---|---|---|---|
| C1 prose | 26.65 (26.58-26.87) | -7.7 % | 2.12 | 78.6 |
| C1 prose2 | 24.97 (24.64-25.07) | -15.5 % | 1.92 | 77.4 |
| C1 code | 53.40 (53.28-53.47) | +11.2 % | 4.19 | 78.8 |
| C1 sampled chat | 29.22 (28.11-30.97) | -8.6 % | 2.35 | 78.9 |
| C4 aggregate | 62.45 (60.92-62.89), per-stream 21.28 | -10.3 % | 2.61 | 123.1 |
k=5 adds ~10 ms to the bs=1 step (68 -> 78.6 ms: 6 verify rows instead of 4, 5 draft rows instead of 3) and prose acceptance barely moves (1.98-2.00 -> 1.92-2.12), so only code gains. Not promoted alone; the confidence cap (E4) is the mechanism meant to recover prose at k=5.
6. Confidence cap
E4 — E3 + DSV41_VERIFY_CAP=conf:0.1 + DSV41_BLOCK_VERIFY=1 (06:09) — promoted
Block verification is enabled as the cap's dependency for sampled rows (it honours the live length;
without it the chain sampler would ignore the cut). Log: cap captured into the verify graphs for
M=6/12/18/24, confidence present.
| workload | decode tok/s (min-max) | vs E2 (k=3) | vs E3 (k=5) | accept | step ms |
|---|---|---|---|---|---|
| C1 prose | 30.20 (30.20-30.30) | +4.6 % | +13.3 % | 2.00 | 68.4 |
| C1 prose2 | 28.10 (27.98-28.46) | -4.9 % | +12.5 % | 1.85 | 66.0 |
| C1 code | 52.67 (52.27-52.82) | +9.6 % | -1.4 % | 4.15 | 78.8 |
| C1 sampled chat | 31.77 (29.70-32.89) | -0.6 % | +8.7 % | 2.21 | 69.8 |
| C4 aggregate | 68.90 (67.26-70.39), per-stream 22.82 | -1.1 % | +10.3 % | 2.31 | 112.1 |
Quality gate: 8/8, 8/8 concurrent (all 8 byte-identical to C1), greedy repeat 1 distinct of 3, 28.7k needle found. The cap brings prose steps back to the k=3 cost (66-68 ms) while code keeps the k=5 acceptance (4.15). Prose mean over both prompts 29.15 vs 29.21 (E2): neutral; code +9.6 %; sampled and C4 neutral. Greedy 512-token code varied within the boot (4 distinct of 5; E3 without the cap: 2 of 5), prose already varied at k=3.
Autotune: DSV41_DRAFT_HEAD_FP8, DSV41_ENGRAM_PREFETCH(_CHECK) added to the adapter's volatile list
(they touch no FlashInfer-tuned op), so from E5 on those A/B arms can share tuned tactics.
7. Draft LM head FP8
E5 — E4 + DSV41_DRAFT_HEAD_FP8=1 (06:33) — promoted
Twin of the draft LM head (43136 x 5120 per rank) built at attach; the KV calculation fell 1,072,384 -> 956,928 tokens (~192 MB, matching the ~210 MiB estimate) and the pool stays at the 750k pin.
| workload | decode tok/s (min-max) | vs E4 | accept | step ms (E4) |
|---|---|---|---|---|
| C1 prose | 30.50 (30.24-30.79) | +1.0 % | 2.10 | 68.3 (68.4) |
| C1 prose2 | 27.93 (27.91-28.33) | -0.6 % | 1.82 | 65.0 (66.0) |
| C1 code | 54.33 (54.16-54.62) | +3.2 % | 4.30 | 77.5 (78.8) |
| C1 sampled chat | 32.98 (32.26-34.61) | +3.8 % | 2.34 | 69.1 (69.8) |
| C4 aggregate | 69.47 (68.33-69.57), per-stream 23.29 | +0.8 % | 2.37 | 113.3 (112.1) |
Step time fell on every C1 workload (0.1-1.3 ms), the size expected; target tokens are unaffected by construction (only the draft's logits use the twin).
8. Engram prefetch and an autotune-retention bug
E6a — E5 + DSV41_ENGRAM_PREFETCH=1 + DSV41_ENGRAM_PREFETCH_CHECK=1 (06:54): correctness
After the quality-gate traffic (8/8, 8/8 concurrent and byte-identical, needle 28.7k found) the check
counters read [0, 0] differing gathers on ranks 0, 1 and 2 over ~91k lookups per layer
(prefill incl. the needle, decode, 4-way concurrency). No fallback/error messages. Not timed
(check mode repeats the lookup).
Autotune retention never reused a cache on this image. E6a differed from E5 only in volatile
switches, yet every rank logged "tuned and saved". The live scheduler's environment carries
SGLANG_RUN_ID=sglang-run-<timestamp>-<rand>, set fresh by the launcher on every boot and matched by
the fingerprint's SGLANG_ prefix, so no sidecar could ever match. Fixed (excluded, test added);
before the fix the adapter behaved exactly like the stock discard path, so E1-E6a are unaffected.
E6b — E5 + DSV41_ENGRAM_PREFETCH=1, check off (07:09) — promoted
| workload | decode tok/s (min-max) | vs E5 | accept | step ms (E5) |
|---|---|---|---|---|
| C1 prose | 31.73 (31.51-31.76) | +4.0 % | 2.10 | 65.5 (68.3) |
| C1 prose2 | 29.11 (28.88-29.66) | +4.2 % | 1.82 | 62.7 (65.0) |
| C1 code | 56.33 (56.06-56.48) | +3.7 % | 4.32 | 75.1 (77.5) |
| C1 sampled chat | 32.50 (31.21-35.01); 10 reps: 32.94 (31.81-34.34) | -1.5 % (noise, sd 1.3) | 2.12-2.17 | 66.2 (69.1) |
| C4 aggregate | 69.27 (68.03-70.08), per-stream 23.46 | -0.3 % | 2.31 |
The bs=1 step fell 2.3-2.9 ms on every workload at unchanged acceptance: more than the 0.7 ms rank-0 profile estimate, because the other ranks' Engram waits surface inside rank 0's NCCL time. The folded draft sampler is active (AUTO chose it: "draft proposal (greedy + sampling) folded into the draft cuda graph"), which the draft-temperature experiment needs.
9. Sampled decoding: draft temperature
E7 — E6b + DSV41_DRAFT_TAU=0.8 (07:33) — neutral, not promoted
First boot to reuse the autotune caches: all three ranks logged reused for both caches and no
"per-rank caches disagree" message (E6b's sidecars, same launch fingerprint; DRAFT_TAU is volatile).
With the same tactics, the greedy code text was byte-identical to E6b's (787b5ff22342656b) and two of
four prose texts matched: cross-boot greedy output is now reproducible for the first time tonight.
KV pool 697,600 on this boot (calculation below the 750k pin; boot-to-boot head free-memory variance).
| workload | E7 tok/s | E6b tok/s | accept E7 / E6b |
|---|---|---|---|
| C1 sampled chat, 10 reps | 33.02 (31.59-33.83) | 32.94 (31.81-34.34) | 2.15 / 2.17 |
| C1 sampled chat, 5 reps | 33.61 (32.85-36.49) | 32.50 (31.21-35.01) | 2.25 / 2.12 |
| C1 prose / prose2 / code | 31.63 / 29.11 / 56.41 | 31.73 / 29.11 / 56.33 | unchanged |
| C4 aggregate | 70.11 | 69.27 | 2.34 / 2.31 |
Pooled over 15 sampled runs: +1.2 % tok/s, +1.4 % acceptance, the size the offline estimate predicts but inside run-to-run noise (sd ~0.9-1.3 tok/s). Exact by construction and greedy is untouched, but it is not a demonstrated win, so it is left out of the final configuration.
10. TP3/EP1
E8 — E6b + EP_SIZE=1 (07:53) — promoted (final candidate)
Boots cleanly (MXFP4 experts tensor-sharded to 768-wide intermediates; new autotune cache keys). KV calculation 929,024 -> pool at the 750k pin.
| workload | decode tok/s (min-max) | vs E6b (EP3) | accept | step ms (E6b) |
|---|---|---|---|---|
| C1 prose | 34.20 (33.73-34.43) | +7.8 % | 2.04 | 58.8 (65.5) |
| C1 prose2 | 32.59 (31.92-32.69) | +12.0 % | 1.83 | 57.0 (62.7) |
| C1 code | 62.53 (61.68-62.83) | +11.0 % | 3.98 | 68.8 (75.1) |
| C1 sampled chat | 35.82 (35.70-38.41) | +10.2 % | 2.21 | 60.3 (66.2) |
| C4 aggregate | 75.96 (74.59-76.86), per-stream 24.44 | +9.7 % | 2.36 |
Quality gate: 8/8, 8/8 concurrent (7 byte-identical), greedy repeat 1 distinct of 3, needle 28.7k. Long context: needles at 95,541 and 190,886 prompt tokens found (58.5 s / 136.9 s); prose, code and sampled requests run concurrently alongside completed normally. No NaN/inf/OOM/traceback lines. Head PSI during the 190k prefill: some avg10 1.33.
Profile (rank 0, 42 steps, prose): step 56.1 ms median. NCCL all-reduce 12.9 -> 5.6 ms/step (the
per-layer straggler wait collapsed: every rank now streams a third of every active expert instead of
whole experts on whichever rank owns them); MoE grouped GEMM 23.8 -> 21.1 ms; b12x dense MXFP8 17.4 ms
(now the largest fixed cost with MoE); wo_a twin 3.5 ms; draft wo_a einsum 2.4 ms; draft head fp8 1.0 ms.
E9 — E8 with DSV41_VERIFY_CAP=conf:0.2 (08:21) — neutral, not promoted
Reused E8's EP1 caches on all ranks (threshold is volatile), so only the cap differs.
| workload | E9 conf:0.2 | E8 conf:0.1 | accept E9 / E8 |
|---|---|---|---|
| C1 prose | 33.93 | 34.20 (-0.8 %) | 1.94 / 2.04 |
| C1 prose2 | 32.66 | 32.59 (+0.2 %) | 1.80 / 1.83 |
| C1 code | 63.30 | 62.53 (+1.2 %) | 4.12 / 3.98 |
| C1 sampled | 37.31 | 35.82 (+4.2 %, sd ~1) | 2.17 / 2.21 |
| C4 aggregate | 73.83 | 75.96 (-2.8 %) | 2.33 / 2.36 |
Mixed, within noise except C4; conf:0.1 kept.
E10 — E8 with k=3 and no cap (08:37) — rejected
| workload | E10 (EP1 k=3) | E8 (EP1 k=5 conf:0.1) |
|---|---|---|
| C1 prose | 34.23 | 34.20 |
| C1 prose2 | 31.52 (-3.3 %) | 32.59 |
| C1 code | 55.99 (-10.5 %) | 62.53 |
| C1 sampled | 36.98 (+3.2 %, noise) | 35.82 |
| C4 aggregate | 76.20 | 75.96 |
k=5 + cap keeps its code advantage on EP1 and is otherwise equal. Final configuration = E8. Note: E10 retuned into the same EP1 cache directory (the engine's cache key omits block size), so E8's tactics were overwritten; the final boot retunes once, and a repeat boot checks reuse.
11. Final configuration installed as .env (08:50)
Previous .env saved to state/env.pre-overnight-20260924. Effective settings identical to
envs/e8-ep1.env: EP_SIZE=1, DSV41_WO_A_W8/MID/DROP=1, DSV41_DRAFT_HEAD_FP8=1,
DSV41_VERIFY_CAP=conf:0.1, DSV41_BLOCK_VERIFY=1, DSV41_ENGRAM_PREFETCH=1, DSPARK_BLOCK_SIZE=5,
DSV41_AUTOTUNE_KEEP=1; everything else as before (fraction 0.95, pin 750k, 4 slots, chunk 1024).
F1 — final configuration from .env, fresh tuning (08:54)
KV pool 658,176 (calculation below the pin this boot). Autotune "tuned and saved" (E10 had overwritten the EP1 caches).
| workload | F1 | E8 (same config, other tactics) | accept | step ms |
|---|---|---|---|---|
| C1 prose | 34.29 (33.73-34.52) | 34.20 | 2.05 | 58.7 |
| C1 prose2 | 32.27 (31.85-32.46) | 32.59 | 1.88 | 58.1 |
| C1 code | 62.69 (62.00-62.76) | 62.53 | 4.00 | 68.5 |
| C1 sampled chat | 36.57 (35.78-39.02) | 35.82 | 2.25 | 60.7 |
| C4 aggregate | 76.03 (74.99-76.22), per-stream 24.52 | 75.96 | 2.40 |
Reproduces E8 within 1-2 % on an independent tuning pass. Quality gate: 8/8, 8/8 concurrent (7 byte-identical), greedy repeat 1 distinct of 3, needles at 28.7k and 95.5k found (15 s / 56 s). Host memory while serving: spark1 2 GB available (PSI some avg10 0.26), spark2 7 GB, spark3 3 GB.
F2 — identical repeat boot (09:12) — left serving
All three ranks reused both autotune caches (no "disagree"); packed=True for both layers on all
three ranks; KV pool 749,824 (pin).
| workload | F2 | F1 | accept | step ms |
|---|---|---|---|---|
| C1 prose | 34.15 (33.86-34.52) | 34.29 | 2.04 | 59.0 |
| C1 prose2 | 32.56 (32.00-32.66) | 32.27 | 1.85 | 57.3 |
| C1 code | 62.24 (61.47-62.85) | 62.69 | 3.98 | 69.7 |
| C1 sampled chat | 37.20 (36.25-37.76) | 36.57 | 2.20 | 60.2 |
| C4 aggregate | 75.79 (73.36-76.60), per-stream 24.31 | 76.03 | 2.34 |
Quality gate 8/8, 8/8 concurrent, greedy repeat 1 distinct of 3, needle 28.7k found. Greedy code text identical to F1's. Final report: overnight-results.md.
12. Extra experiment in the remaining time: draft wo_a on the fp8 twin
E11 — final + DSV41_WO_A_W8_DRAFT=1 (09:32) — neutral, rejected
New default-off switch in adapter/wo_a_w8.py: in bridge mode the draft's 3 wo_a get fp8 twins
(bf16 kept, +48 MB) and the draft module's global torch is replaced by a proxy whose einsum
routes "bgd,grd->bgr" with a twinned weight to the twin kernels (2-8 rows small, 9-192 MID); all
other calls pass through. Volatile for autotune: E11 reused F2's tactics on all ranks.
| workload | E11 | F2 | accept E11 / F2 | step ms E11 / F2 |
|---|---|---|---|---|
| C1 prose | 33.86 | 34.15 (-0.8 %) | 1.93 / 2.04 | 58.4 / 59.0 |
| C1 prose2 | 32.25 | 32.56 (-1.0 %) | 1.73 / 1.85 | 57.5 / 57.3 |
| C1 code | 61.67 | 62.24 (-0.9 %) | 4.12 / 3.98 | 68.2 / 69.7 |
| C1 sampled | 36.73 | 37.20 (-1.3 %) | 2.23 / 2.20 | 60.3 / 60.2 |
| C4 aggregate | 76.80 | 75.79 (+1.3 %) | 2.49 / 2.34 |
Quality 8/8, 8/8 concurrent, needle found. The step moved -1.5 to +0.2 ms (less than the 2.4 ms the
draft einsum costs) and prose acceptance dipped; net neutral-to-negative on the priority workloads.
Switch left off (default); code kept for a later profile-guided look.
Fleet restored to the final .env configuration (F3) at ~09:47.