Home Verkennen Help
Inloggen
sm121
/
recipes
2
0
Vork 0
Bestanden Issues 0 Pull-aanvragen 0 Wiki
Boom: f1c95cde2d
Aftakkingen Labels
recipes
/
docker-images
/
mia-tp4
/
docs
/
overnight-results.md

overnight-results.md 15 KB
Geschiedenis Ruwe

Overnight TP3 optimization — results

2026-09-24, 03:11-11:11 IDT, three DGX Sparks (GB10), TP_SIZE=3 throughout. Plan: merged-new.md. Running log with every experiment: overnight-progress.md. Raw evidence (per-request JSONL, summaries, acceptance joins, boot logs, env files, profiles): results/overnight-20260924-0311/.

Result: the fleet serves a configuration that is 17-40 % faster single-stream and 19 % faster at C4 than the one it started with, validated on two boots with no measured quality regression. Biggest levers: TP3/EP1 (+8-12 %), the wo_a FP8 twin with DROP/MID, k=5 with the confidence cap (code), and Engram prefetch. Two operational defects were found and fixed on the way: rank 2 was running a different SGLang build from ranks 0/1, and the packed Engram shards were missing on every rank.

workload (5 reps, 512 tokens, from spark2) original (b0) final F1 (fresh tune) final F2 (cache reuse) change b0 -> F2
C1 prose, greedy (tok/s) 29.28 34.29 34.15 +16.6 %
C1 prose2, greedy (added at E2; E2 value in first column) (29.56) 32.27 32.56 (+10.1 % vs E2)
C1 code, greedy 44.65 62.69 62.24 +39.4 %
C1 sampled chat (T=0.7) 28.87 36.57 37.20 +28.9 %
C4 aggregate, 2 prose + 2 code 63.78 76.03 75.79 +18.8 %
C4 per-stream decode 19.46 24.52 24.31 +24.9 %
TTFT, C1 (52-token prompt) 0.28-0.30 s 0.25-0.29 s 0.25-0.29 s unchanged
bs=1 step, prose / code 73.2 / 74.0 ms 58.7 / 68.5 ms 59.0 / 69.7 ms -14 / -4 ms (k 3 -> 5)
acceptance, prose / code 2.17 / 3.27 2.05 / 4.00 2.04 / 3.98 code +22 % (k=5)
effective KV pool (tokens) 491,520 658,176 749,824 (pin) see memory section

Quality on the final configuration (F1 and F2, plus E8 with identical settings): 8/8 checkable tasks, 8/8 under 4-way concurrency, greedy 256-token repeat 1 distinct of 3, needles found at 28.7k and 95.5k (F1) and at 190.9k prompt tokens (E8) with prose/code/sampled requests running alongside. No NaN/inf/OOM/traceback lines in any final-configuration log. With autotune reuse, the greedy code text was byte-identical between F1 and F2.

The fleet is left serving the final configuration from the default .env via ./start.sh (F3, rebooted ~09:47 after the E11 extra experiment; same settings and tactics as F2).

How it was measured

  • scripts/overnight/bench.sh LABEL 5 512 runs benchmarks/overnight_bench.py from spark2 against the head: one discarded warm-up per workload and one C4 warm-up wave, then 5 repetitions per workload, 512 max tokens, thinking off. Workloads: greedy prose (lighthouse essay), greedy prose2 (fishing-village story, added at E2), greedy code (LRU cache + tests), sampled chat (T=0.7, top_p=0.95); C4 = four concurrent requests (2 prose + 2 code) x 5 waves.
  • Tokens are usage.completion_tokens; never SSE event counts. C1 decode tok/s = (tokens - 1) / (last content delta - first content delta); C4 aggregate = total completion tokens / wave wall time. Acceptance and step time come from the head's Decode batch lines inside each phase window (scripts/overnight/accept.py: step = accept_len x bs / gen throughput).
  • scripts/overnight/quality.sh LABEL [needle_k,...] runs benchmarks/overnight_quality.py: 8 checkable tasks (arithmetic, JSON schema, generated is_prime executed against 10 cases, sorting, counting, facts), the same 8 fired 4 at a time, a greedy repeat check, and needle retrieval.
  • Every experiment is its own env file (.env + the listed changes) in results/.../envs/, booted with ENV_FILE=<file> ./start.sh serve. start.sh sources the env file with set -a, so command-line overrides of variables that .env defines are silently ignored; use an env file.
  • Foreign traffic: none during measurements (last foreign chat request 02:29). sparkDash on spark3 polls the API every 2 s (5 GETs, 0.06 % CPU); pausing it was refused by the session's safety policy, so it is a constant background in every number here.
  • Cross-boot noise. Until autotune retention worked (E7), FlashInfer re-drew its tactics every boot, which changes the numerics, hence the greedy text, hence acceptance: a few % of C1 tok/s between boots of one configuration. bs=1 step time is the steadier signal and is reported alongside. From E7 on, arms that differ only in volatile switches reuse one tactic set, and greedy code output was byte-identical across boots.

Starting state and what was repaired

issue found action
Packed Engram shards absent on all ranks (packed=False) spark2's old TP2 shards (101.4 GB) moved to spark1:/home/mia/archive/dsv41-engram-tp2-spark2/ (SHA-256 verified both sides, SHA256SUMS there; pointer at spark2:/home/zurih/dsv41-engram/MOVED.txt); TP3 shards packed on all three ranks; packed=True for layers 1 and 14 on ranks 0/1/2 after restart
Rank 2 ran a different SGLang build (spark3's lmsysorg/sglang:dev-dsv41 = 381b27ff / da64c5cbb, 40 source files different from ranks 0/1 incl. deepseek_v4.py and the DSpark worker) spark3's images kept as lmsysorg/sglang:dev-dsv41-da64c5cbb-20260911 and dsv41-3x-spark:mixed-da64c5cbb-20260924; the head's base (37939c26) loaded onto spark3; overlay rebuilt everywhere
Autotune caches discarded twice per boot adapter/autotune_keep.py ported, plus two fixes (below)

Accepted changes (in order of application)

step change evidence
B1 packed Engram + identical images on all ranks C4 +5 %, code +2 %; Engram waiting gone from the profile
E2 DSV41_WO_A_W8=1, DSV41_WO_A_W8_DROP=1, DSV41_WO_A_W8_MID=1 (target layers only) bs=1 step 72 -> 68-69 ms, bs=4 131.5 -> 125.6 ms; KV pool back to the 750k pin (DROP frees 1280 MB/rank)
E4 DSPARK_BLOCK_SIZE=5 + DSV41_VERIFY_CAP=conf:0.1 + DSV41_BLOCK_VERIFY=1 code +9.6 %, prose/sampled/C4 neutral vs k=3
E5 DSV41_DRAFT_HEAD_FP8=1 step -0.1 to -1.3 ms on all C1 workloads, +210 MiB
E6b DSV41_ENGRAM_PREFETCH=1 (check-mode pass first: 0 differing gathers on all ranks) step -2.3 to -2.9 ms everywhere, C1 +3.7 to +4.2 %
E8 EP_SIZE=1 (TP3/EP1) C1 +7.8 to +12 %, C4 +9.7 %; NCCL 12.9 -> 5.6 ms/step
infra DSV41_AUTOTUNE_KEEP=1 (fixed port) all three ranks reused both caches on E7, E9 and the final repeat boot; greedy output reproducible across boots

Rejected or neutral

experiment result
E1: wo_a W8 twin without DROP does not boot at fraction 0.95: +688 MiB leaves "no GPU memory for the KV cache ... minimum viable 0.9503"
E1b: W8 + DROP as shipped booted but acceptance 1.0 (C1 ~14 tok/s): dev-dsv41's draft runs its own einsum on self.wo_a.weight, which DROP had replaced by a zero tag. Fixed in the adapter (draft left on bf16 in bridge mode)
E1c: W8 + DROP without MID C4 -8 %: 16-row verifies fall to per-call dequantize; MID fixes it (E2)
E3: k=5 without the cap code +11 %, prose -8 to -16 %, C4 -10 %
E7: DSV41_DRAFT_TAU=0.8 sampled +1.2 % pooled over 15 runs, inside noise; not a demonstrated win, left off
E9: conf:0.2 instead of 0.1 mixed within noise, C4 -2.8 %
E10: k=3 without cap on EP1 code -10.5 %, prose2 -3.3 %, prose/C4 equal: k=5 + conf:0.1 stays
E11: draft wo_a einsum on the fp8 twin (DSV41_WO_A_W8_DRAFT=1, new, default off) C1 -0.8 to -1.3 %, C4 +1.3 % on the same tactics; step -1.5 to +0.2 ms, prose acceptance dipped; left off

Adapter fixes made tonight

  • adapter/wo_a_w8.py: in einsum-bridge mode (dev-dsv41) the draft's wo_a is never dropped (the draft never reaches the shared dispatch, so DROP broke it) and by default not twinned; the new DSV41_WO_A_W8_DRAFT=1 (default off, forwarded by start.sh, volatile for autotune) routes the draft's own einsum to a twin instead (E11, neutral).
  • adapter/autotune_keep.py (ported from knapcio 4d8f4c0):
    1. deletes a rank's cache when its sidecar does not match the launch. Otherwise, when no rank has a matching sidecar, every digest is "", the stock gate sees agreement, and FlashInfer still loads each rank's stale file (the cache key omits block size, graph sizes and adapter switches);
    2. excludes SGLANG_RUN_ID (a per-boot timestamp the launcher sets) from the fingerprint; with it, no sidecar could ever match and nothing was reused;
    3. volatile list extended with DSV41_DRAFT_HEAD_FP8, DSV41_ENGRAM_PREFETCH(_CHECK). Tests extended for (1) and (2).
  • start.sh: forwards DSV41_AUTOTUNE_KEEP and DSV41_ENGRAM_PREFETCH_CHECK on head and workers, and SGLANG_DSPARK_FOLDED_SAMPLING only when set (unset stays the engine's AUTO, which chose the folded sampler on every boot tonight).

Exact final configuration

.env (installed 08:50; the previous file is state/env.pre-overnight-20260924). Changed keys only; everything else is unchanged (TP_SIZE=3, NNODES=3, MEM_FRACTION_STATIC / HEAD_MEM_FRACTION_STATIC 0.95, MAX_TOTAL_TOKENS 750000, MAX_RUNNING_REQUESTS 4, CHUNKED_PREFILL_SIZE 1024, DSV41_CACHE_GIB 0, DSV41_MXFP8_BACKEND b12x, SGLANG_FLASHINFER_MOE_FUSED_FINALIZE 0, PYTORCH_CUDA_ALLOC_CONF expandable_segments:False, CONTEXT_LENGTH 262144, NCCL settings as before):

EP_SIZE=1
DSV41_WO_A_W8=1
DSV41_WO_A_W8_MID=1
DSV41_WO_A_W8_DROP=1
DSV41_DRAFT_HEAD_FP8=1
DSV41_VERIFY_CAP=conf:0.1
DSV41_BLOCK_VERIFY=1
DSV41_ENGRAM_PREFETCH=1
DSPARK_BLOCK_SIZE=5
DSV41_AUTOTUNE_KEEP=1

Images: dsv41-3x-spark:local rebuilt from this working tree on all three nodes on the common base lmsysorg/sglang:dev-dsv41@sha256:3dbc3130… (public on Docker Hub; arm64 image id 37939c26c0…, 2026-09-10, SGLang 0.0.0.dev0, FlashInfer 0.6.18). Packed Engram: head ~/dsv41-engram/engram-l{1,14}-r0of3.bin, workers /home/zurih/dsv41-3x-spark/engram/engram-l{1,14}-r{1,2}of3.bin (31.5 GiB each). Engine-side facts on the final boots: folded DSpark sampling active (AUTO), verify cap captured for M=6/12/18/24, packed=True on all ranks, autotune caches reused on the repeat boot.

Reproducible commands

# build (all three nodes) and serve the final configuration
./start.sh build
./start.sh                                   # = serve, reads .env
# benchmark from spark2 and join acceptance/step from the head log
RESULTS_DIR=docs/results/<dir> scripts/overnight/bench.sh <label> 5 512
# only some C1 workloads / phases
PHASES=c1 C1_WORKLOADS=chat_sampled RESULTS_DIR=… scripts/overnight/bench.sh <label> 10 512
# quality gate incl. needles (k tokens)
RESULTS_DIR=… scripts/overnight/quality.sh <label> 30,100,200
# an experiment arm = .env plus changes
RESULTS_DIR=… BASE_ENV=.env scripts/overnight/mkenv.sh <name> KEY=VAL …
ENV_FILE=$RESULTS_DIR/envs/<name>.env ./start.sh serve
# decode profile of rank 0 (send one request from a worker while it waits), analyse on spark2
scripts/profile/profile_decode.sh 45 /tmp/prof

Memory tradeoffs

item per-rank effect
wo_a DROP (40 target layers) bf16 copies released: -1280 MB; fp8 twins kept: +640 MB + scales. Net about -640 MB vs the original. The draft's 3 wo_a stay bf16 (96 MB). Twins without DROP (+688 MiB) do not fit at 0.95.
draft LM head fp8 twin +210 MiB (43136 x 5120 e4m3 + scales); bf16 head kept (the target uses it)
Engram prefetch side stream only; no measurable change
EP1 vs EP3 same expert bytes per rank; KV calculation 929k (E8) vs 957k (E5, EP3): within boot variance
KV pool (effective) original boot 491,520; B1 226,816; W8+DROP boots at the 750k pin (749,824); final boots 658,176 (F1) and 749,824 (F2). The engine sizes from the minimum free memory across ranks, which varies by ~0.5 GB between boots; below the pin the pool shrinks rather than failing. 190,886-token needle passed on E8.
host headroom while serving head ~2 GB MemAvailable, spark2 ~7 GB, spark3 ~3-4 GB (unchanged from the original); head PSI "some" avg10 peaked at 1.33 during a 190k-token prefill

Rollback

  • Whole campaign: cp state/env.pre-overnight-20260924 .env && ./start.sh. The rebuilt image keeps every adapter off unless its variable is set, so the old .env reproduces the old serving flags on converged images with packed Engram. (Deliberately not restoring the mixed-build fleet: the original spark3 overlay is kept as dsv41-3x-spark:mixed-da64c5cbb-20260924 if it is ever needed.)
  • Single feature: set it back in .env and ./start.sh: EP_SIZE=3; DSPARK_BLOCK_SIZE=3 with DSV41_VERIFY_CAP=0; DSV41_WO_A_W8=0 (turns MID/DROP off too); DSV41_DRAFT_HEAD_FP8=0; DSV41_ENGRAM_PREFETCH=0; DSV41_AUTOTUNE_KEEP=0 (stock discard-and-retune).
  • Stale autotune caches: delete ~/.cache/sglang/flashinfer/autotune/0.6.18/sm121/*/rank_*.{json,launch} on all three nodes; the next boot retunes.
  • Packed Engram needs no rollback (serving falls back to the checkpoint if the files are absent). The old TP2 rank-1 shards are at spark1:/home/mia/archive/dsv41-engram-tp2-spark2/ (SHA256SUMS).

Remaining opportunities (ranked by the final profile)

  1. b12x dense MXFP8 GEMMs, 17.4 ms/step (234 calls): now the largest fixed cost next to MoE. Shape-specific tile tuning at M=6-24, or fusing the MXFP8 activation quantize (0.7 ms) into the producer.
  2. MoE grouped GEMM, 21.1 ms/step under EP1: narrower 768-wide shards; worth a tactic sweep now that caches persist.
  3. Draft wo_a on bf16, 2.4 ms/step: tried as E11 (DSV41_WO_A_W8_DRAFT=1, einsum proxy); neutral. A profile of that switch should check the dispatch and why draft acceptance moved.
  4. NCCL all-reduce 5.6 ms/step: the triangle-aware RoCEnante port remains the transport lever (deferred tonight: needs a protocol proof and standalone three-rank validation).
  5. Draft temperature 0.8: +1.2 % sampled, not significant in 15 runs; a longer sampled run would settle it (it is exact and volatile, so it can be A/B'd on one tactic set).
  6. KV pool determinism: pin at ~650k, or drop page cache before boot, so the pool is the same every boot (it ranged 658k-750k tonight).
  7. Not attempted (scope or time): clock unlock (prohibited tonight), DRM-backed row cache (host change), the newer da64c5cbb/canary image as a whole-fleet upgrade, more request slots / C8, SPS/STS tables, folded-result fence, per-position cap thresholds, k=4 + cap.

After the campaign: long-prompt crash and chunk size (2026-09-24 afternoon)

In normal use of the final configuration, a single new prompt of at least ~200k tokens ran spark1 out of memory during prefill (1024-token chunks; first NV_ERR_NO_MEMORY at 13:23, host hung until a hard reset at 14:05). The KV cache was only ~31 % full: the per-chunk indexer transient was the peak. The campaign's long-context check had stopped at 191k, so the range 191k-262k was never validated on this stack, and no memory guard was running.

Change: CHUNKED_PREFILL_SIZE=768 (default in .env.example). The next boot got the 750k KV pin with a 1.24M budget and the head at ~5.2 GB MemAvailable (was ~2 GB). Not yet measured: the largest prompt that is safe at 768, and whether EP1 or the larger KV cache lowered the ceiling compared with the pre-campaign configuration (208k verified there, 256k failed). Until that is measured, run scripts/verify/memguard.py on the head for long-context work.