# DSV4.1-1017 4x SGLang — performance A/B (2026-10-18)

Lane: `dsv41-flash-1017-4x-sglang` (Aiden `production-1.7.1` + the baked 1017
encoder patch, image `dsv41-sglang-1017:v1`). Venue: **4nodes-switch (dgx1-4)**,
200G RoCE switch, weights mmap'd from the NAS HF cache, Engram rows node-local
(`[[staging]]` sparse-copy). The live production lane on `dgx58-switch` was not
touched during this campaign.

Question: the lane measures pp ~3.0k t/s / tg ~45 tok/s (c1) while a comparable
community lane (knapcio v2.4 TP4, SGLang `dev-dsv41`, same 1017 checkpoint on 4
Sparks) reports pp 5,385 / tg 63.9 (c1) and 94.1 (c4). Which of the visible
config differences actually move our numbers?

## Method

Two instruments, both run from `pluto.lan` (off the Mac's WiFi path):

1. `sm121 bench --perf-only` — llama-benchy pp2048/tg128, c1/c2, depth 0/4K/8K.
   Same harness/version as every other lane report (v2.8.0).
2. `/tmp/f1017/probe.py` (kept with the campaign) — 2 discarded warm-up passes +
   N=5 measured passes of a **cold ~2048-token prompt** (unique nonce → no radix
   hit) asking for a 150-word paragraph; decode rate is
   `(completion_tokens-1)/(last_delta - ttft)` using the streamed `usage`
   block (speculative decoding emits multi-token deltas, so counting SSE events
   undercounts by ~2x). Median of the 5. This is the tie-breaker: llama-benchy
   rows on this stack scatter ±30% run to run, the probe does not.

Boot cost per config on this lane is 26-40 min (476 GB of fp8 weights off the
NAS + CUDA-graph capture), so variants were kept few and chosen from the
community configs rather than swept blindly.

## Results

llama-benchy (pp t/s / tg t/s), single pass each:

| config | d0 c1 | d0 c2 | d4K c1 | d8K c1 | d8K c2 |
|---|---|---|---|---|---|
| baseline (`ep-size 4`, chunk 2048, ctx 512K) | 3,019 / **44.9** | 2,597 / 52.6 | 3,038 / 46.6 | 2,870 / 39.6 | 2,430 / 53.6 |
| `--enable-deepseek-v4-fp4-indexer` | 2,942 / 45.1 | 2,128 / 55.5 | 3,019 / 48.5 | 2,829 / 43.6 | 2,132 / 51.3 |
| `ep-size 2` | 3,054 / 61.0 | 1,089 / 58.8 | 2,372 / 42.3 | 1,965 / 41.1 | 1,753 / 56.9 |
| `ep-size 2` (repeat) | 2,485 / 43.2 | 2,607 / 60.5 | 3,009 / 44.1 | 2,692 / 48.6 | 1,874 / 65.5 |
| `ep-size 4` + chunk 8192 + ctx 1M | 3,275 / 42.3 | — (crash, see below) | — | — | — |
| **final: `ep-size 2`, chunk 2048, ctx 1M** (quality run) | 2,358 / 45.6 | 1,916 / 67.8 | 2,533 / 45.4 | 2,859 / 45.8 | 2,677 / 38.5 |

probe.py (median of 5, cold 2K prompt → decode tok/s and TTFT≈prefill):

| config | TTFT | decode c1 |
|---|---|---|
| `ep-size 2`, chunk 2048, ctx 512K | 0.754 s | **41.7** |
| `ep-size 4` + chunk 8192 + ctx 1M | 0.711 s | 37.8 / 37.5 (two runs) |
| `ep-size 2` + chunk 8192 + ctx 1M | 0.702 s | **43.0** |
| `ep-size 2`, chunk 2048, ctx 1M (final) | 0.76 s | 38.8 / 41.8 (two runs) |

Quality (tool-eval-bench v2.8.0, seed 42, hardmode, thinking on):

| config | score | responsiveness | median turn | failures |
|---|---|---|---|---|
| baseline EP4 / 512K | 87/100 (168/194) | 71 | 1.7 s | TC-68/74/88/89/91/92 |
| final EP2 / 1M | 87/100 (168/194) | **75** | **1.4 s** | TC-68/71/74/88/89/92 |

The two runs swap one scenario each way (TC-91 vs TC-71) — single-trial noise
at the same score. Adopted in `recipes/dsv41-flash-1017-4x-sglang` (commit
`fe0d386` in the recipe repo): `--ep-size 2`, `--context-length 1048576`,
`--chunked-prefill-size 2048`, `SPARK_ENGRAM_MAX_IDS=65536`.

## Findings

1. **`--ep-size 2` (TP4/EP2, the knapcio shape) is the one real decode win.**
   Probe: 41.7-43.0 vs 37.5-37.8 at EP4. It halves the per-step MoE
   dispatch/combine fan-out (2 nodes instead of 4) at the cost of keeping the
   full expert set on each node pair — memory is fine (KV pool stays ~9.2M
   tokens, ~12-13 GB GPU headroom left). Adopted as the lane default.
2. **`--enable-deepseek-v4-fp4-indexer` is neutral-to-negative here.** Decode
   unchanged within noise, c2/8K rows dropped ~15-20%. Reverted; the flag is
   `experimental` in this build and the DSA indexer is not our bottleneck.
3. **`--chunked-prefill-size 8192` is blocked by an Engram coupling, not by
   performance.** The per-chunk Engram lookup is `max_prefill_tokens x hash
   columns` (~2 x chunk x 12): chunk 2048 → 49,152 ids, which is exactly what
   the recipe's `SPARK_ENGRAM_MAX_IDS=65536` covers; chunk 8192 → 190,608 ids
   and the engine dies mid-request with
   `RuntimeError: engram layer 1: a lookup of 190608 ids exceeds
   SPARK_ENGRAM_MAX_IDS=131072`. Raising the id budget (≥ 196,608, e.g.
   `262144`) is the prerequisite for any larger chunk. Kept at 2048 by choice:
   per the agent-serving findings, 2048 gives ~2.9x the decode share of 8192
   under a concurrent prefill, which is the interactive-agent trade we want.
   (At a 2048-token prompt the chunk size does not change pp at all — the first
   cell of the 8192 run was 3,275 vs 3,019 baseline, inside noise.)
4. **`--context-length 1048576` costs nothing** (the KV pool is sized by
   `--mem-fraction-static`, not by the context cap: 9.18M tokens with the 1M
   cap vs 9.57M at 512K) and matches DeepSeek's 1017 recommendation. Adopted.
5. The remaining gap to the knapcio number is **not** in these flags: their
   stack (`dev-dsv41` + Mia adapter: fast loader, L2 prefetch, chunked indexer
   v3, mxfp8 b12x MoE, loop-abort) is a different overlay. Porting that is a
   separate project, not a flag flip.

## Operational note: slow-NAS boots

One boot attempt died with `TP rank 0 could finish the model loading, but there
are other ranks that didn't finish loading` while ranks 1/3 still had shards in
flight — the NAS read rate had collapsed to ~1/6 of the morning's (48 shards took
~95 min instead of 13), rank 0 finished first and the TCPStore closed under the
stragglers. Nothing in the config was wrong; a retry on the same config booted
cleanly with 13.5 GB GPU headroom. Two things follow: (a) a boot during NAS
contention can fail on rank desync, so retry before suspecting the config;
(b) the engram/weight pre-staging on node-local storage is what keeps a
*restart* cheap — the 26-95 min here is all NAS weight read.

## Depth sweep (final config, llama-benchy pp2048/tg128, 2 runs per point)

`ctx` rows = prefill the full depth then decode 128; `pp2048` rows = a 2048-token
suffix at that depth (the agent-shaped case).

| depth | ctx pp (c1/c2) | ctx tg (c1/c2) | pp2048 tg (c1/c2) |
|---|---|---|---|
| 16,384 | 3,530 / 2,279 | 39.0 / 32.7 | 46.5 / 62.5 |
| 65,536 | 1,532 / 2,151 | 32.0 / **6.0** | 47.3 / 62.1 |
| 131,072 | 1,545 / 1,561 | 36.1 / **2.7** | 42.1 / 61.2 |
| 262,144 | 1,447 / 1,525 | 50.5 / **1.4** | 46.4 / 67.5 |

Reading: single-stream decode after a 262K prefill is fine (50.5), and the
agent-shaped case (short suffix at depth) holds 42-67 tok/s through the whole
range. What collapses is **two concurrent deep conversations**: the c2 `ctx`
rows fall to 6.0 / 2.7 / 1.4 tok/s because both streams' prefills serialize
through the 2048-token chunk loop and decode gets the interleaved slots only.
This is the documented prefill-vs-decode contention (see
`dgx-spark-agent-serving-findings.md`), not a regression of this config. If
long-context multi-session serving becomes the goal, the lever is
`--chunked-prefill-size` (with `SPARK_ENGRAM_MAX_IDS` raised accordingly) or
an admission cap, not more KV.

## Reasoning-effort A/B (2026-10-18 evening)

`SGLANG_DSV41_REASONING_EFFORT` 100 (Aiden's default, our lane) vs 75 (knapcio's
default), same EP2/2048/1M config, same harness:

| effort | score | Hard Mode | responsiveness | median turn | tg c1 (d0) |
|---|---|---|---|---|---|
| 100 | **87/100** | 42/56 (75%) | 75 | 1.4 s | 45.6 |
| 75 | 86/100 | 41/56 (73%) | 75 | 1.4 s | 40.3 |

**The effort lever does not pay on 1017.** The "max costs 3.5x the reasoning
tokens" observation belongs to the *old* V4.1-Flash; 1017 at `max` already spends
~40% of that (stu.miller, forum 382725 #266), so dropping to 75 buys no
measurable turn time here and costs a point. Lane stays at 100.

## Not yet measured

- DSPARK `--speculative-dspark-block-size` sweep (3/5/7); accept length on the
  live lane sits at 3.0-5.2 with block 5.
- The 2x alternative: thread 386121 / `x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark`
  (TensorFold + vLLM on EXL3 2.9bpw, decode c1 59-103 tok/s on two Sparks) is a
  different checkpoint + engine; a dgx1+dgx2 (`2nodes-dac`) lane is the natural
  place to try it.
## The TensorFold 2x alternative (measured 2026-10-19)

Upstream `x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark` (forum 386121) on
`2nodes-dac` (dgx1 head + dgx2 worker), EXL3 2.9bpw weights on node-local disk,
TensorFold 0.6.0 + Jay Leaton patches, 262144 context, thinking on / effort 75.
Bring-up detail + artifacts: `recipes/dsv41-flash-1017-2x-tensorfold/NOTE.md`.

| metric | TensorFold 2x | 4x SGLang (EP2/1M) |
|---|---|---|
| quality (v2.8.0, seed 42, hardmode) | **89/100** | 87/100 |
| responsiveness / median turn | **23 / 6.7 s** | **75 / 1.4 s** |
| decode c1 (probe median) | **50.4 tok/s** | 38.8-43.0 tok/s |
| prefill for the same ~2K prompt | 5.4 s (~380 tok/s) | 0.75 s (~2,700 tok/s) |
| nodes | 2 | 4 |

So the trade is explicit: **half the cluster, better decode and no quality loss,
at 4-5x the turn latency** (prefill is ~7x slower — the harness scores it as
responsiveness 23). TensorFold 2x is the long-generation / batch shape; the 4x
SGLang lane remains the interactive-agent shape. Not purity-matched: thinking
effort 75 vs 100, EXL3 2.9bpw vs fp8.

**Status 2026-10-19:** the lane is now a real sm121 recipe —
`recipes/dsv41-flash-1017-2x-tensorfold` + `docker-images/dsv41-tf-1017` + the
`tensorfold_1017_prepare` built-in host script. `sm121 serve --recipe
dsv41-flash-1017-2x-tensorfold --cluster 2nodes-dac` stages, builds, syncs and
launches it end to end (API ready ~4 min after the 24.5 GB image sync), and
`sm121 stop` closes the runlock. The upstream repo's **vLLM profile on the same
weights is blocked** in our environment — the exact failure and the fix options
are in the last section of that lane's `NOTE.md` (the NFS Engram volume carries
only NAS symlinks that dangle inside the worker's vLLM container; rsync mode
wants 196 GiB free on the worker).

## Four-lane comparison (2026-10-20/21 night shift)

| lane | quality (v2.8.0, seed 42, hardmode) | responsiveness / median turn | pp c1 | tg c1 / c2 / c4 | KV pool |
|---|---|---|---|---|---|
| **1017 SGLang 4x** (this lane, EP2/1M) | 87/100 | **75 / 1.4 s** | 3,019 | **45.6 / 67.8 / —** | 9.30M |
| 1017 TensorFold 2x (EXL3 2.9bpw) | 89/100 | 23 / 6.7 s | ~380 (2K prompt) | 50.4 / — / — | 26.6M |
| 1017 FLARE 4x (`local-inference-lab` gb10 pkg) | **unmeasurable — crashes** | — | ~2.4k | 22–27 / — / — | 26.6M |
| Beam-1.0 NVFP4 4x (different model) | **90/100** | 34 / 4.7 s | 2,501 | 26.2 / 41.1 / 62.3 | 6.81M |

FLARE detail: the official 4-Spark gb10 package validates (`sm_121`, 1M ctx,
16 concurrency, MTP 1-6, 74.7 GiB/rank, 130 graphs) and runs over RoCE with two
rails per peer, but decodes at 22-27 tok/s and **dies on the first agent-sized
prompt** — `tensor-parallel peer failed: RoCE proxy failed: two-shot op N
abandoned: kernel timed out at seq N` (short chat prompts are fine). Full bench:
0/100, every scenario 500/connection-refused. The FLARE *runtime* itself is fine
on GB10 (Qwen3.8-Flash-Next tp1: 43.3 tok/s, 1.3 s TTFT, ~1 s load) with one API
gap: `tool_choice: "required"` returns HTTP 500.

Verdict: the EP2/1M SGLang lane stays the interactive default; Beam is the
quality leader for hard batch work; TensorFold remains the 2-node batch shape;
FLARE is not viable for us at this package revision.

## DeepInfer (deepseek-ai/DeepInfer) — assessed 2026-10-21, not runnable on GB10

DeepSeek open-sourced their production inference framework (14.9k stars, Apache
2.0). It targets exactly our model family — "Supported models:
DeepSeek-V4.1-Flash", default checkpoint `deepseek-ai/DeepSeek-V4.1-Flash-1017`
— with optimised kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect,
TileLang), attention-DP/CP + MoE EP with EPLB, DSpark with confidence-scheduled
verification, SWA bounded replay, async scheduling, batch-invariant inference,
deterministic sampling and structured outputs.

**Three blockers, in order of severity:**

1. **Offline inference only.** The README states the release is offline and that
   a complete service still needs "a KV cache system", "VE/Prefill/Decode
   disaggregation" and "load balancing across data-parallel replicas" — i.e. no
   server. There is no HTTP API to point an agent at; it is a Python library
   (`examples/offline_inference.py`).
2. **SM100-only.** `deep_cluster/hardware/device.py::is_sm_major_10()` gates on
   compute capability **major 10** (SM100/101/103) and
   `get_bf16_tflops_limit()` raises `RuntimeError("unknown GPU type")` otherwise.
   GB10 is **SM121**. The vendored kernels (FlashMLA/DeepGEMM/DeepEP) are
   SM90/SM100-targeted too.
3. **Build cost.** `DEEPINFER_ACCELERATOR=nvidia ./build.sh` needs GCC >= 13,
   CUDA 13, Python 3.12, four vendored CUDA projects built from submodules, plus
   an NCCL 2.32.3 force-reinstall; arm64 is not a documented target.

**Verdict:** not usable this shift, and a port is a project (kernels to sm121 +
write the serving layer), not an evening. What *is* worth stealing:

- **SWA bounded replay** (`DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact`,
  `..._KEEP_TOKENS=140`): replay only a bounded suffix of the decoder prefill,
  keeping at least the sliding-window span. This is a prefill-latency idea we
  could look for in the SGLang/VL stack for our 1017 lane.
- **Confidence-scheduled verification** (`DEEPINFER_ENABLE_MTP_VARLEN=1` +
  `DEEPINFER_USE_NEXTN_DECODE`): schedule verification by draft confidence — the
  same family as the "adaptive verification" knob Beam's DFlash2 wanted.
- **Batch-invariant inference** — determinism across batch composition; relevant
  if we ever need reproducible evals.

Re-check when the repo grows a serving path or an sm120/sm121 target.