launcher/specs/tech-architecture/dsv41-1017-tuning.md

DSV4.1-1017 4x SGLang — performance A/B (2026-10-18)

Lane: dsv41-flash-1017-4x-sglang (Aiden production-1.7.1 + the baked 1017 encoder patch, image dsv41-sglang-1017:v1). Venue: 4nodes-switch (dgx1-4), 200G RoCE switch, weights mmap'd from the NAS HF cache, Engram rows node-local ([[staging]] sparse-copy). The live production lane on dgx58-switch was not touched during this campaign.

Question: the lane measures pp ~3.0k t/s / tg ~45 tok/s (c1) while a comparable community lane (knapcio v2.4 TP4, SGLang dev-dsv41, same 1017 checkpoint on 4 Sparks) reports pp 5,385 / tg 63.9 (c1) and 94.1 (c4). Which of the visible config differences actually move our numbers?

Method

Two instruments, both run from pluto.lan (off the Mac's WiFi path):

1. sm121 bench --perf-only — llama-benchy pp2048/tg128, c1/c2, depth 0/4K/8K. Same harness/version as every other lane report (v2.8.0).
2. /tmp/f1017/probe.py (kept with the campaign) — 2 discarded warm-up passes + N=5 measured passes of a cold ~2048-token prompt (unique nonce → no radix hit) asking for a 150-word paragraph; decode rate is (completion_tokens-1)/(last_delta - ttft) using the streamed usage block (speculative decoding emits multi-token deltas, so counting SSE events undercounts by ~2x). Median of the 5. This is the tie-breaker: llama-benchy rows on this stack scatter ±30% run to run, the probe does not.

Boot cost per config on this lane is 26-40 min (476 GB of fp8 weights off the NAS + CUDA-graph capture), so variants were kept few and chosen from the community configs rather than swept blindly.

Results

llama-benchy (pp t/s / tg t/s), single pass each:

config	d0 c1	d0 c2	d4K c1	d8K c1	d8K c2
baseline (ep-size 4, chunk 2048, ctx 512K)	3,019 / 44.9	2,597 / 52.6	3,038 / 46.6	2,870 / 39.6	2,430 / 53.6
--enable-deepseek-v4-fp4-indexer	2,942 / 45.1	2,128 / 55.5	3,019 / 48.5	2,829 / 43.6	2,132 / 51.3
ep-size 2	3,054 / 61.0	1,089 / 58.8	2,372 / 42.3	1,965 / 41.1	1,753 / 56.9
ep-size 2 (repeat)	2,485 / 43.2	2,607 / 60.5	3,009 / 44.1	2,692 / 48.6	1,874 / 65.5
ep-size 4 + chunk 8192 + ctx 1M	3,275 / 42.3	— (crash, see below)	—	—	—
final: ep-size 2, chunk 2048, ctx 1M (quality run)	2,358 / 45.6	1,916 / 67.8	2,533 / 45.4	2,859 / 45.8	2,677 / 38.5

probe.py (median of 5, cold 2K prompt → decode tok/s and TTFT≈prefill):

config	TTFT	decode c1
ep-size 2, chunk 2048, ctx 512K	0.754 s	41.7
ep-size 4 + chunk 8192 + ctx 1M	0.711 s	37.8 / 37.5 (two runs)
ep-size 2 + chunk 8192 + ctx 1M	0.702 s	43.0
ep-size 2, chunk 2048, ctx 1M (final)	0.76 s	38.8 / 41.8 (two runs)

Quality (tool-eval-bench v2.8.0, seed 42, hardmode, thinking on):

config	score	responsiveness	median turn	failures
baseline EP4 / 512K	87/100 (168/194)	71	1.7 s	TC-68/74/88/89/91/92
final EP2 / 1M	87/100 (168/194)	75	1.4 s	TC-68/71/74/88/89/92

The two runs swap one scenario each way (TC-91 vs TC-71) — single-trial noise at the same score. Adopted in recipes/dsv41-flash-1017-4x-sglang (commit fe0d386 in the recipe repo): --ep-size 2, --context-length 1048576, --chunked-prefill-size 2048, SPARK_ENGRAM_MAX_IDS=65536.

Findings

1. --ep-size 2 (TP4/EP2, the knapcio shape) is the one real decode win. Probe: 41.7-43.0 vs 37.5-37.8 at EP4. It halves the per-step MoE dispatch/combine fan-out (2 nodes instead of 4) at the cost of keeping the full expert set on each node pair — memory is fine (KV pool stays ~9.2M tokens, ~12-13 GB GPU headroom left). Adopted as the lane default.
2. --enable-deepseek-v4-fp4-indexer is neutral-to-negative here. Decode unchanged within noise, c2/8K rows dropped ~15-20%. Reverted; the flag is experimental in this build and the DSA indexer is not our bottleneck.
3. --chunked-prefill-size 8192 is blocked by an Engram coupling, not by performance. The per-chunk Engram lookup is max_prefill_tokens x hash columns (~2 x chunk x 12): chunk 2048 → 49,152 ids, which is exactly what the recipe's SPARK_ENGRAM_MAX_IDS=65536 covers; chunk 8192 → 190,608 ids and the engine dies mid-request with RuntimeError: engram layer 1: a lookup of 190608 ids exceeds SPARK_ENGRAM_MAX_IDS=131072. Raising the id budget (≥ 196,608, e.g. 262144) is the prerequisite for any larger chunk. Kept at 2048 by choice: per the agent-serving findings, 2048 gives ~2.9x the decode share of 8192 under a concurrent prefill, which is the interactive-agent trade we want. (At a 2048-token prompt the chunk size does not change pp at all — the first cell of the 8192 run was 3,275 vs 3,019 baseline, inside noise.)
4. --context-length 1048576 costs nothing (the KV pool is sized by --mem-fraction-static, not by the context cap: 9.18M tokens with the 1M cap vs 9.57M at 512K) and matches DeepSeek's 1017 recommendation. Adopted.
5. The remaining gap to the knapcio number is not in these flags: their stack (dev-dsv41 + Mia adapter: fast loader, L2 prefetch, chunked indexer v3, mxfp8 b12x MoE, loop-abort) is a different overlay. Porting that is a separate project, not a flag flip.

Operational note: slow-NAS boots

One boot attempt died with TP rank 0 could finish the model loading, but there are other ranks that didn't finish loading while ranks 1/3 still had shards in flight — the NAS read rate had collapsed to ~1/6 of the morning's (48 shards took ~95 min instead of 13), rank 0 finished first and the TCPStore closed under the stragglers. Nothing in the config was wrong; a retry on the same config booted cleanly with 13.5 GB GPU headroom. Two things follow: (a) a boot during NAS contention can fail on rank desync, so retry before suspecting the config; (b) the engram/weight pre-staging on node-local storage is what keeps a restart cheap — the 26-95 min here is all NAS weight read.

Depth sweep (final config, llama-benchy pp2048/tg128, 2 runs per point)

ctx rows = prefill the full depth then decode 128; pp2048 rows = a 2048-token suffix at that depth (the agent-shaped case).

depth	ctx pp (c1/c2)	ctx tg (c1/c2)	pp2048 tg (c1/c2)
16,384	3,530 / 2,279	39.0 / 32.7	46.5 / 62.5
65,536	1,532 / 2,151	32.0 / 6.0	47.3 / 62.1
131,072	1,545 / 1,561	36.1 / 2.7	42.1 / 61.2
262,144	1,447 / 1,525	50.5 / 1.4	46.4 / 67.5

Reading: single-stream decode after a 262K prefill is fine (50.5), and the agent-shaped case (short suffix at depth) holds 42-67 tok/s through the whole range. What collapses is two concurrent deep conversations: the c2 ctx rows fall to 6.0 / 2.7 / 1.4 tok/s because both streams' prefills serialize through the 2048-token chunk loop and decode gets the interleaved slots only. This is the documented prefill-vs-decode contention (see dgx-spark-agent-serving-findings.md), not a regression of this config. If long-context multi-session serving becomes the goal, the lever is --chunked-prefill-size (with SPARK_ENGRAM_MAX_IDS raised accordingly) or an admission cap, not more KV.

Reasoning-effort A/B (2026-10-18 evening)

SGLANG_DSV41_REASONING_EFFORT 100 (Aiden's default, our lane) vs 75 (knapcio's default), same EP2/2048/1M config, same harness:

effort	score	Hard Mode	responsiveness	median turn	tg c1 (d0)
100	87/100	42/56 (75%)	75	1.4 s	45.6
75	86/100	41/56 (73%)	75	1.4 s	40.3

The effort lever does not pay on 1017. The "max costs 3.5x the reasoning tokens" observation belongs to the old V4.1-Flash; 1017 at max already spends ~40% of that (stu.miller, forum 382725 #266), so dropping to 75 buys no measurable turn time here and costs a point. Lane stays at 100.

Not yet measured

• DSPARK --speculative-dspark-block-size sweep (3/5/7); accept length on the live lane sits at 3.0-5.2 with block 5.

• The 2x alternative: thread 386121 / x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark (TensorFold + vLLM on EXL3 2.9bpw, decode c1 59-103 tok/s on two Sparks) is a different checkpoint + engine; a dgx1+dgx2 (2nodes-dac) lane is the natural place to try it.

  The TensorFold 2x alternative (measured 2026-10-19)

Upstream x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark (forum 386121) on 2nodes-dac (dgx1 head + dgx2 worker), EXL3 2.9bpw weights on node-local disk, TensorFold 0.6.0 + Jay Leaton patches, 262144 context, thinking on / effort 75. Bring-up detail + artifacts: recipes/dsv41-flash-1017-2x-tensorfold/NOTE.md.

metric	TensorFold 2x	4x SGLang (EP2/1M)
quality (v2.8.0, seed 42, hardmode)	89/100	87/100
responsiveness / median turn	23 / 6.7 s	75 / 1.4 s
decode c1 (probe median)	50.4 tok/s	38.8-43.0 tok/s
prefill for the same ~2K prompt	5.4 s (~380 tok/s)	0.75 s (~2,700 tok/s)
nodes	2	4

So the trade is explicit: half the cluster, better decode and no quality loss, at 4-5x the turn latency (prefill is ~7x slower — the harness scores it as responsiveness 23). TensorFold 2x is the long-generation / batch shape; the 4x SGLang lane remains the interactive-agent shape. Not purity-matched: thinking effort 75 vs 100, EXL3 2.9bpw vs fp8.

Status 2026-10-19: the lane is now a real sm121 recipe — recipes/dsv41-flash-1017-2x-tensorfold + docker-images/dsv41-tf-1017 + the tensorfold_1017_prepare built-in host script. sm121 serve --recipe dsv41-flash-1017-2x-tensorfold --cluster 2nodes-dac stages, builds, syncs and launches it end to end (API ready ~4 min after the 24.5 GB image sync), and sm121 stop closes the runlock. The upstream repo's vLLM profile on the same weights is blocked in our environment — the exact failure and the fix options are in the last section of that lane's NOTE.md (the NFS Engram volume carries only NAS symlinks that dangle inside the worker's vLLM container; rsync mode wants 196 GiB free on the worker).

Four-lane comparison (2026-10-20/21 night shift)

lane	quality (v2.8.0, seed 42, hardmode)	responsiveness / median turn	pp c1	tg c1 / c2 / c4	KV pool
1017 SGLang 4x (this lane, EP2/1M)	87/100	75 / 1.4 s	3,019	45.6 / 67.8 / —	9.30M
1017 TensorFold 2x (EXL3 2.9bpw)	89/100	23 / 6.7 s	~380 (2K prompt)	50.4 / — / —	26.6M
1017 FLARE 4x (local-inference-lab gb10 pkg)	unmeasurable — crashes	—	~2.4k	22–27 / — / —	26.6M
Beam-1.0 NVFP4 4x (different model)	90/100	34 / 4.7 s	2,501	26.2 / 41.1 / 62.3	6.81M

FLARE detail: the official 4-Spark gb10 package validates (sm_121, 1M ctx, 16 concurrency, MTP 1-6, 74.7 GiB/rank, 130 graphs) and runs over RoCE with two rails per peer, but decodes at 22-27 tok/s and dies on the first agent-sized prompt — tensor-parallel peer failed: RoCE proxy failed: two-shot op N abandoned: kernel timed out at seq N (short chat prompts are fine). Full bench: 0/100, every scenario 500/connection-refused. The FLARE runtime itself is fine on GB10 (Qwen3.8-Flash-Next tp1: 43.3 tok/s, 1.3 s TTFT, ~1 s load) with one API gap: tool_choice: "required" returns HTTP 500.

Verdict: the EP2/1M SGLang lane stays the interactive default; Beam is the quality leader for hard batch work; TensorFold remains the 2-node batch shape; FLARE is not viable for us at this package revision.