|
|
@@ -0,0 +1,121 @@
|
|
|
+# DSV4.1-1017 4x SGLang — performance A/B (2026-10-18)
|
|
|
+
|
|
|
+Lane: `dsv41-flash-1017-4x-sglang` (Aiden `production-1.7.1` + the baked 1017
|
|
|
+encoder patch, image `dsv41-sglang-1017:v1`). Venue: **4nodes-switch (dgx1-4)**,
|
|
|
+200G RoCE switch, weights mmap'd from the NAS HF cache, Engram rows node-local
|
|
|
+(`[[staging]]` sparse-copy). The live production lane on `dgx58-switch` was not
|
|
|
+touched during this campaign.
|
|
|
+
|
|
|
+Question: the lane measures pp ~3.0k t/s / tg ~45 tok/s (c1) while a comparable
|
|
|
+community lane (knapcio v2.4 TP4, SGLang `dev-dsv41`, same 1017 checkpoint on 4
|
|
|
+Sparks) reports pp 5,385 / tg 63.9 (c1) and 94.1 (c4). Which of the visible
|
|
|
+config differences actually move our numbers?
|
|
|
+
|
|
|
+## Method
|
|
|
+
|
|
|
+Two instruments, both run from `pluto.lan` (off the Mac's WiFi path):
|
|
|
+
|
|
|
+1. `sm121 bench --perf-only` — llama-benchy pp2048/tg128, c1/c2, depth 0/4K/8K.
|
|
|
+ Same harness/version as every other lane report (v2.8.0).
|
|
|
+2. `/tmp/f1017/probe.py` (kept with the campaign) — 2 discarded warm-up passes +
|
|
|
+ N=5 measured passes of a **cold ~2048-token prompt** (unique nonce → no radix
|
|
|
+ hit) asking for a 150-word paragraph; decode rate is
|
|
|
+ `(completion_tokens-1)/(last_delta - ttft)` using the streamed `usage`
|
|
|
+ block (speculative decoding emits multi-token deltas, so counting SSE events
|
|
|
+ undercounts by ~2x). Median of the 5. This is the tie-breaker: llama-benchy
|
|
|
+ rows on this stack scatter ±30% run to run, the probe does not.
|
|
|
+
|
|
|
+Boot cost per config on this lane is 26-40 min (476 GB of fp8 weights off the
|
|
|
+NAS + CUDA-graph capture), so variants were kept few and chosen from the
|
|
|
+community configs rather than swept blindly.
|
|
|
+
|
|
|
+## Results
|
|
|
+
|
|
|
+llama-benchy (pp t/s / tg t/s), single pass each:
|
|
|
+
|
|
|
+| config | d0 c1 | d0 c2 | d4K c1 | d8K c1 | d8K c2 |
|
|
|
+|---|---|---|---|---|---|
|
|
|
+| baseline (`ep-size 4`, chunk 2048, ctx 512K) | 3,019 / **44.9** | 2,597 / 52.6 | 3,038 / 46.6 | 2,870 / 39.6 | 2,430 / 53.6 |
|
|
|
+| `--enable-deepseek-v4-fp4-indexer` | 2,942 / 45.1 | 2,128 / 55.5 | 3,019 / 48.5 | 2,829 / 43.6 | 2,132 / 51.3 |
|
|
|
+| `ep-size 2` | 3,054 / 61.0 | 1,089 / 58.8 | 2,372 / 42.3 | 1,965 / 41.1 | 1,753 / 56.9 |
|
|
|
+| `ep-size 2` (repeat) | 2,485 / 43.2 | 2,607 / 60.5 | 3,009 / 44.1 | 2,692 / 48.6 | 1,874 / 65.5 |
|
|
|
+| `ep-size 4` + chunk 8192 + ctx 1M | 3,275 / 42.3 | — (crash, see below) | — | — | — |
|
|
|
+| **final: `ep-size 2`, chunk 2048, ctx 1M** (quality run) | 2,358 / 45.6 | 1,916 / 67.8 | 2,533 / 45.4 | 2,859 / 45.8 | 2,677 / 38.5 |
|
|
|
+
|
|
|
+probe.py (median of 5, cold 2K prompt → decode tok/s and TTFT≈prefill):
|
|
|
+
|
|
|
+| config | TTFT | decode c1 |
|
|
|
+|---|---|---|
|
|
|
+| `ep-size 2`, chunk 2048, ctx 512K | 0.754 s | **41.7** |
|
|
|
+| `ep-size 4` + chunk 8192 + ctx 1M | 0.711 s | 37.8 / 37.5 (two runs) |
|
|
|
+| `ep-size 2` + chunk 8192 + ctx 1M | 0.702 s | **43.0** |
|
|
|
+| `ep-size 2`, chunk 2048, ctx 1M (final) | 0.76 s | 38.8 / 41.8 (two runs) |
|
|
|
+
|
|
|
+Quality (tool-eval-bench v2.8.0, seed 42, hardmode, thinking on):
|
|
|
+
|
|
|
+| config | score | responsiveness | median turn | failures |
|
|
|
+|---|---|---|---|---|
|
|
|
+| baseline EP4 / 512K | 87/100 (168/194) | 71 | 1.7 s | TC-68/74/88/89/91/92 |
|
|
|
+| final EP2 / 1M | 87/100 (168/194) | **75** | **1.4 s** | TC-68/71/74/88/89/92 |
|
|
|
+
|
|
|
+The two runs swap one scenario each way (TC-91 vs TC-71) — single-trial noise
|
|
|
+at the same score. Adopted in `recipes/dsv41-flash-1017-4x-sglang` (commit
|
|
|
+`fe0d386` in the recipe repo): `--ep-size 2`, `--context-length 1048576`,
|
|
|
+`--chunked-prefill-size 2048`, `SPARK_ENGRAM_MAX_IDS=65536`.
|
|
|
+
|
|
|
+## Findings
|
|
|
+
|
|
|
+1. **`--ep-size 2` (TP4/EP2, the knapcio shape) is the one real decode win.**
|
|
|
+ Probe: 41.7-43.0 vs 37.5-37.8 at EP4. It halves the per-step MoE
|
|
|
+ dispatch/combine fan-out (2 nodes instead of 4) at the cost of keeping the
|
|
|
+ full expert set on each node pair — memory is fine (KV pool stays ~9.2M
|
|
|
+ tokens, ~12-13 GB GPU headroom left). Adopted as the lane default.
|
|
|
+2. **`--enable-deepseek-v4-fp4-indexer` is neutral-to-negative here.** Decode
|
|
|
+ unchanged within noise, c2/8K rows dropped ~15-20%. Reverted; the flag is
|
|
|
+ `experimental` in this build and the DSA indexer is not our bottleneck.
|
|
|
+3. **`--chunked-prefill-size 8192` is blocked by an Engram coupling, not by
|
|
|
+ performance.** The per-chunk Engram lookup is `max_prefill_tokens x hash
|
|
|
+ columns` (~2 x chunk x 12): chunk 2048 → 49,152 ids, which is exactly what
|
|
|
+ the recipe's `SPARK_ENGRAM_MAX_IDS=65536` covers; chunk 8192 → 190,608 ids
|
|
|
+ and the engine dies mid-request with
|
|
|
+ `RuntimeError: engram layer 1: a lookup of 190608 ids exceeds
|
|
|
+ SPARK_ENGRAM_MAX_IDS=131072`. Raising the id budget (≥ 196,608, e.g.
|
|
|
+ `262144`) is the prerequisite for any larger chunk. Kept at 2048 by choice:
|
|
|
+ per the agent-serving findings, 2048 gives ~2.9x the decode share of 8192
|
|
|
+ under a concurrent prefill, which is the interactive-agent trade we want.
|
|
|
+ (At a 2048-token prompt the chunk size does not change pp at all — the first
|
|
|
+ cell of the 8192 run was 3,275 vs 3,019 baseline, inside noise.)
|
|
|
+4. **`--context-length 1048576` costs nothing** (the KV pool is sized by
|
|
|
+ `--mem-fraction-static`, not by the context cap: 9.18M tokens with the 1M
|
|
|
+ cap vs 9.57M at 512K) and matches DeepSeek's 1017 recommendation. Adopted.
|
|
|
+5. The remaining gap to the knapcio number is **not** in these flags: their
|
|
|
+ stack (`dev-dsv41` + Mia adapter: fast loader, L2 prefetch, chunked indexer
|
|
|
+ v3, mxfp8 b12x MoE, loop-abort) is a different overlay. Porting that is a
|
|
|
+ separate project, not a flag flip.
|
|
|
+
|
|
|
+## Operational note: slow-NAS boots
|
|
|
+
|
|
|
+One boot attempt died with `TP rank 0 could finish the model loading, but there
|
|
|
+are other ranks that didn't finish loading` while ranks 1/3 still had shards in
|
|
|
+flight — the NAS read rate had collapsed to ~1/6 of the morning's (48 shards took
|
|
|
+~95 min instead of 13), rank 0 finished first and the TCPStore closed under the
|
|
|
+stragglers. Nothing in the config was wrong; a retry on the same config booted
|
|
|
+cleanly with 13.5 GB GPU headroom. Two things follow: (a) a boot during NAS
|
|
|
+contention can fail on rank desync, so retry before suspecting the config;
|
|
|
+(b) the engram/weight pre-staging on node-local storage is what keeps a
|
|
|
+*restart* cheap — the 26-95 min here is all NAS weight read.
|
|
|
+
|
|
|
+## Not yet measured
|
|
|
+
|
|
|
+- `SGLANG_DSV41_REASONING_EFFORT=75` (knapcio default; stu.miller measured 1017
|
|
|
+ at `max` using ~40% of the old model's reasoning tokens). Changes token volume
|
|
|
+ and therefore responsiveness far more than any flag above — needs its own
|
|
|
+ quality + responsiveness A/B.
|
|
|
+- DSPARK `--speculative-dspark-block-size` sweep (3/5/7); accept length on the
|
|
|
+ live lane sits at 3.0-5.2 with block 5.
|
|
|
+- `--depth` sweep (16K..256K) on the final config; the 1M context cap is adopted
|
|
|
+ but the long-context rows are unmeasured on this venue.
|
|
|
+- The 2x alternative: thread 386121 / `x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark`
|
|
|
+ (TensorFold + vLLM on EXL3 2.9bpw, decode c1 59-103 tok/s on two Sparks) is a
|
|
|
+ different checkpoint + engine; a dgx1+dgx2 (`2nodes-dac`) lane is the natural
|
|
|
+ place to try it.
|
