DSV4.1-1017 4x SGLang — performance A/B (2026-10-18)

Lane: dsv41-flash-1017-4x-sglang (Aiden production-1.7.1 + the baked 1017 encoder patch, image dsv41-sglang-1017:v1). Venue: 4nodes-switch (dgx1-4), 200G RoCE switch, weights mmap'd from the NAS HF cache, Engram rows node-local ([[staging]] sparse-copy). The live production lane on dgx58-switch was not touched during this campaign.

Question: the lane measures pp ~3.0k t/s / tg ~45 tok/s (c1) while a comparable community lane (knapcio v2.4 TP4, SGLang dev-dsv41, same 1017 checkpoint on 4 Sparks) reports pp 5,385 / tg 63.9 (c1) and 94.1 (c4). Which of the visible config differences actually move our numbers?

Method

Two instruments, both run from pluto.lan (off the Mac's WiFi path):

sm121 bench --perf-only — llama-benchy pp2048/tg128, c1/c2, depth 0/4K/8K. Same harness/version as every other lane report (v2.8.0).
/tmp/f1017/probe.py (kept with the campaign) — 2 discarded warm-up passes + N=5 measured passes of a cold ~2048-token prompt (unique nonce → no radix hit) asking for a 150-word paragraph; decode rate is (completion_tokens-1)/(last_delta - ttft) using the streamed usage block (speculative decoding emits multi-token deltas, so counting SSE events undercounts by ~2x). Median of the 5. This is the tie-breaker: llama-benchy rows on this stack scatter ±30% run to run, the probe does not.

Boot cost per config on this lane is 26-40 min (476 GB of fp8 weights off the NAS + CUDA-graph capture), so variants were kept few and chosen from the community configs rather than swept blindly.

Results

llama-benchy (pp t/s / tg t/s), single pass each:

config	d0 c1	d0 c2	d4K c1	d8K c1	d8K c2
baseline (`ep-size 4`, chunk 2048, ctx 512K)	3,019 / 44.9	2,597 / 52.6	3,038 / 46.6	2,870 / 39.6	2,430 / 53.6
`--enable-deepseek-v4-fp4-indexer`	2,942 / 45.1	2,128 / 55.5	3,019 / 48.5	2,829 / 43.6	2,132 / 51.3
`ep-size 2`	3,054 / 61.0	1,089 / 58.8	2,372 / 42.3	1,965 / 41.1	1,753 / 56.9
`ep-size 2` (repeat)	2,485 / 43.2	2,607 / 60.5	3,009 / 44.1	2,692 / 48.6	1,874 / 65.5
`ep-size 4` + chunk 8192 + ctx 1M	3,275 / 42.3	— (crash, see below)	—	—	—
final: `ep-size 2`, chunk 2048, ctx 1M (quality run)	2,358 / 45.6	1,916 / 67.8	2,533 / 45.4	2,859 / 45.8	2,677 / 38.5

probe.py (median of 5, cold 2K prompt → decode tok/s and TTFT≈prefill):

config	TTFT	decode c1
`ep-size 2`, chunk 2048, ctx 512K	0.754 s	41.7
`ep-size 4` + chunk 8192 + ctx 1M	0.711 s	37.8 / 37.5 (two runs)
`ep-size 2` + chunk 8192 + ctx 1M	0.702 s	43.0
`ep-size 2`, chunk 2048, ctx 1M (final)	0.76 s	38.8 / 41.8 (two runs)

Quality (tool-eval-bench v2.8.0, seed 42, hardmode, thinking on):

config	score	responsiveness	median turn	failures
baseline EP4 / 512K	87/100 (168/194)	71	1.7 s	TC-68/74/88/89/91/92
final EP2 / 1M	87/100 (168/194)	75	1.4 s	TC-68/71/74/88/89/92

The two runs swap one scenario each way (TC-91 vs TC-71) — single-trial noise at the same score. Adopted in recipes/dsv41-flash-1017-4x-sglang (commit fe0d386 in the recipe repo): --ep-size 2, --context-length 1048576, --chunked-prefill-size 2048, SPARK_ENGRAM_MAX_IDS=65536.

Findings

--ep-size 2 (TP4/EP2, the knapcio shape) is the one real decode win. Probe: 41.7-43.0 vs 37.5-37.8 at EP4. It halves the per-step MoE dispatch/combine fan-out (2 nodes instead of 4) at the cost of keeping the full expert set on each node pair — memory is fine (KV pool stays ~9.2M tokens, ~12-13 GB GPU headroom left). Adopted as the lane default.
--enable-deepseek-v4-fp4-indexer is neutral-to-negative here. Decode unchanged within noise, c2/8K rows dropped ~15-20%. Reverted; the flag is experimental in this build and the DSA indexer is not our bottleneck.
--chunked-prefill-size 8192 is blocked by an Engram coupling, not by performance. The per-chunk Engram lookup is max_prefill_tokens x hash columns (~2 x chunk x 12): chunk 2048 → 49,152 ids, which is exactly what the recipe's SPARK_ENGRAM_MAX_IDS=65536 covers; chunk 8192 → 190,608 ids and the engine dies mid-request with RuntimeError: engram layer 1: a lookup of 190608 ids exceeds SPARK_ENGRAM_MAX_IDS=131072. Raising the id budget (≥ 196,608, e.g. 262144) is the prerequisite for any larger chunk. Kept at 2048 by choice: per the agent-serving findings, 2048 gives ~2.9x the decode share of 8192 under a concurrent prefill, which is the interactive-agent trade we want. (At a 2048-token prompt the chunk size does not change pp at all — the first cell of the 8192 run was 3,275 vs 3,019 baseline, inside noise.)
--context-length 1048576 costs nothing (the KV pool is sized by --mem-fraction-static, not by the context cap: 9.18M tokens with the 1M cap vs 9.57M at 512K) and matches DeepSeek's 1017 recommendation. Adopted.
The remaining gap to the knapcio number is not in these flags: their stack (dev-dsv41 + Mia adapter: fast loader, L2 prefetch, chunked indexer v3, mxfp8 b12x MoE, loop-abort) is a different overlay. Porting that is a separate project, not a flag flip.

Operational note: slow-NAS boots

One boot attempt died with TP rank 0 could finish the model loading, but there are other ranks that didn't finish loading while ranks 1/3 still had shards in flight — the NAS read rate had collapsed to ~1/6 of the morning's (48 shards took ~95 min instead of 13), rank 0 finished first and the TCPStore closed under the stragglers. Nothing in the config was wrong; a retry on the same config booted cleanly with 13.5 GB GPU headroom. Two things follow: (a) a boot during NAS contention can fail on rank desync, so retry before suspecting the config; (b) the engram/weight pre-staging on node-local storage is what keeps a restart cheap — the 26-95 min here is all NAS weight read.

Depth sweep (final config, llama-benchy pp2048/tg128, 2 runs per point)

ctx rows = prefill the full depth then decode 128; pp2048 rows = a 2048-token suffix at that depth (the agent-shaped case).

depth	ctx pp (c1/c2)	ctx tg (c1/c2)	pp2048 tg (c1/c2)
16,384	3,530 / 2,279	39.0 / 32.7	46.5 / 62.5
65,536	1,532 / 2,151	32.0 / 6.0	47.3 / 62.1
131,072	1,545 / 1,561	36.1 / 2.7	42.1 / 61.2
262,144	1,447 / 1,525	50.5 / 1.4	46.4 / 67.5

Reading: single-stream decode after a 262K prefill is fine (50.5), and the agent-shaped case (short suffix at depth) holds 42-67 tok/s through the whole range. What collapses is two concurrent deep conversations: the c2 ctx rows fall to 6.0 / 2.7 / 1.4 tok/s because both streams' prefills serialize through the 2048-token chunk loop and decode gets the interleaved slots only. This is the documented prefill-vs-decode contention (see dgx-spark-agent-serving-findings.md), not a regression of this config. If long-context multi-session serving becomes the goal, the lever is --chunked-prefill-size (with SPARK_ENGRAM_MAX_IDS raised accordingly) or an admission cap, not more KV.

Reasoning-effort A/B (2026-10-18 evening)

SGLANG_DSV41_REASONING_EFFORT 100 (Aiden's default, our lane) vs 75 (knapcio's default), same EP2/2048/1M config, same harness:

effort	score	Hard Mode	responsiveness	median turn	tg c1 (d0)
100	87/100	42/56 (75%)	75	1.4 s	45.6
75	86/100	41/56 (73%)	75	1.4 s	40.3

The effort lever does not pay on 1017. The "max costs 3.5x the reasoning tokens" observation belongs to the old V4.1-Flash; 1017 at max already spends ~40% of that (stu.miller, forum 382725 #266), so dropping to 75 buys no measurable turn time here and costs a point. Lane stays at 100.

Not yet measured

DSPARK --speculative-dspark-block-size sweep (3/5/7); accept length on the live lane sits at 3.0-5.2 with block 5.
The 2x alternative: thread 386121 / x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark (TensorFold + vLLM on EXL3 2.9bpw, decode c1 59-103 tok/s on two Sparks) is a different checkpoint + engine; a dgx1+dgx2 (2nodes-dac) lane is the natural place to try it.

dsv41-1017-tuning.md 8.2 KB Lịch sử Raw