launcher/specs/tech-architecture/dsv41-1017-tuning.md

DSV4.1-1017 4x SGLang — performance A/B (2026-10-18)

Lane: dsv41-flash-1017-4x-sglang (Aiden production-1.7.1 + the baked 1017 encoder patch, image dsv41-sglang-1017:v1). Venue: 4nodes-switch (dgx1-4), 200G RoCE switch, weights mmap'd from the NAS HF cache, Engram rows node-local ([[staging]] sparse-copy). The live production lane on dgx58-switch was not touched during this campaign.

Question: the lane measures pp ~3.0k t/s / tg ~45 tok/s (c1) while a comparable community lane (knapcio v2.4 TP4, SGLang dev-dsv41, same 1017 checkpoint on 4 Sparks) reports pp 5,385 / tg 63.9 (c1) and 94.1 (c4). Which of the visible config differences actually move our numbers?

Method

Two instruments, both run from pluto.lan (off the Mac's WiFi path):

1. sm121 bench --perf-only — llama-benchy pp2048/tg128, c1/c2, depth 0/4K/8K. Same harness/version as every other lane report (v2.8.0).
2. /tmp/f1017/probe.py (kept with the campaign) — 2 discarded warm-up passes + N=5 measured passes of a cold ~2048-token prompt (unique nonce → no radix hit) asking for a 150-word paragraph; decode rate is (completion_tokens-1)/(last_delta - ttft) using the streamed usage block (speculative decoding emits multi-token deltas, so counting SSE events undercounts by ~2x). Median of the 5. This is the tie-breaker: llama-benchy rows on this stack scatter ±30% run to run, the probe does not.

Boot cost per config on this lane is 26-40 min (476 GB of fp8 weights off the NAS + CUDA-graph capture), so variants were kept few and chosen from the community configs rather than swept blindly.

Results

llama-benchy (pp t/s / tg t/s), single pass each:

config	d0 c1	d0 c2	d4K c1	d8K c1	d8K c2
baseline (ep-size 4, chunk 2048, ctx 512K)	3,019 / 44.9	2,597 / 52.6	3,038 / 46.6	2,870 / 39.6	2,430 / 53.6
--enable-deepseek-v4-fp4-indexer	2,942 / 45.1	2,128 / 55.5	3,019 / 48.5	2,829 / 43.6	2,132 / 51.3
ep-size 2	3,054 / 61.0	1,089 / 58.8	2,372 / 42.3	1,965 / 41.1	1,753 / 56.9
ep-size 2 (repeat)	2,485 / 43.2	2,607 / 60.5	3,009 / 44.1	2,692 / 48.6	1,874 / 65.5
ep-size 4 + chunk 8192 + ctx 1M	3,275 / 42.3	— (crash, see below)	—	—	—
final: ep-size 2, chunk 2048, ctx 1M (quality run)	2,358 / 45.6	1,916 / 67.8	2,533 / 45.4	2,859 / 45.8	2,677 / 38.5

probe.py (median of 5, cold 2K prompt → decode tok/s and TTFT≈prefill):

config	TTFT	decode c1
ep-size 2, chunk 2048, ctx 512K	0.754 s	41.7
ep-size 4 + chunk 8192 + ctx 1M	0.711 s	37.8 / 37.5 (two runs)
ep-size 2 + chunk 8192 + ctx 1M	0.702 s	43.0
ep-size 2, chunk 2048, ctx 1M (final)	0.76 s	38.8 / 41.8 (two runs)

Quality (tool-eval-bench v2.8.0, seed 42, hardmode, thinking on):

config	score	responsiveness	median turn	failures
baseline EP4 / 512K	87/100 (168/194)	71	1.7 s	TC-68/74/88/89/91/92
final EP2 / 1M	87/100 (168/194)	75	1.4 s	TC-68/71/74/88/89/92

The two runs swap one scenario each way (TC-91 vs TC-71) — single-trial noise at the same score. Adopted in recipes/dsv41-flash-1017-4x-sglang (commit fe0d386 in the recipe repo): --ep-size 2, --context-length 1048576, --chunked-prefill-size 2048, SPARK_ENGRAM_MAX_IDS=65536.

Findings

1. --ep-size 2 (TP4/EP2, the knapcio shape) is the one real decode win. Probe: 41.7-43.0 vs 37.5-37.8 at EP4. It halves the per-step MoE dispatch/combine fan-out (2 nodes instead of 4) at the cost of keeping the full expert set on each node pair — memory is fine (KV pool stays ~9.2M tokens, ~12-13 GB GPU headroom left). Adopted as the lane default.
2. --enable-deepseek-v4-fp4-indexer is neutral-to-negative here. Decode unchanged within noise, c2/8K rows dropped ~15-20%. Reverted; the flag is experimental in this build and the DSA indexer is not our bottleneck.
3. --chunked-prefill-size 8192 is blocked by an Engram coupling, not by performance. The per-chunk Engram lookup is max_prefill_tokens x hash columns (~2 x chunk x 12): chunk 2048 → 49,152 ids, which is exactly what the recipe's SPARK_ENGRAM_MAX_IDS=65536 covers; chunk 8192 → 190,608 ids and the engine dies mid-request with RuntimeError: engram layer 1: a lookup of 190608 ids exceeds SPARK_ENGRAM_MAX_IDS=131072. Raising the id budget (≥ 196,608, e.g. 262144) is the prerequisite for any larger chunk. Kept at 2048 by choice: per the agent-serving findings, 2048 gives ~2.9x the decode share of 8192 under a concurrent prefill, which is the interactive-agent trade we want. (At a 2048-token prompt the chunk size does not change pp at all — the first cell of the 8192 run was 3,275 vs 3,019 baseline, inside noise.)
4. --context-length 1048576 costs nothing (the KV pool is sized by --mem-fraction-static, not by the context cap: 9.18M tokens with the 1M cap vs 9.57M at 512K) and matches DeepSeek's 1017 recommendation. Adopted.
5. The remaining gap to the knapcio number is not in these flags: their stack (dev-dsv41 + Mia adapter: fast loader, L2 prefetch, chunked indexer v3, mxfp8 b12x MoE, loop-abort) is a different overlay. Porting that is a separate project, not a flag flip.

Operational note: slow-NAS boots

One boot attempt died with TP rank 0 could finish the model loading, but there are other ranks that didn't finish loading while ranks 1/3 still had shards in flight — the NAS read rate had collapsed to ~1/6 of the morning's (48 shards took ~95 min instead of 13), rank 0 finished first and the TCPStore closed under the stragglers. Nothing in the config was wrong; a retry on the same config booted cleanly with 13.5 GB GPU headroom. Two things follow: (a) a boot during NAS contention can fail on rank desync, so retry before suspecting the config; (b) the engram/weight pre-staging on node-local storage is what keeps a restart cheap — the 26-95 min here is all NAS weight read.

Not yet measured

• SGLANG_DSV41_REASONING_EFFORT=75 (knapcio default; stu.miller measured 1017 at max using ~40% of the old model's reasoning tokens). Changes token volume and therefore responsiveness far more than any flag above — needs its own quality + responsiveness A/B.
• DSPARK --speculative-dspark-block-size sweep (3/5/7); accept length on the live lane sits at 3.0-5.2 with block 5.
• --depth sweep (16K..256K) on the final config; the 1M context cap is adopted but the long-context rows are unmeasured on this venue.
• The 2x alternative: thread 386121 / x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark (TensorFold + vLLM on EXL3 2.9bpw, decode c1 59-103 tok/s on two Sparks) is a different checkpoint + engine; a dgx1+dgx2 (2nodes-dac) lane is the natural place to try it.