Ana Sayfa Keşfet Yardım
Giriş Yap
sm121
/
launcher
2
0
Çatalla 0
Dosyalar Sorunlar 0 Değişiklik İstekleri 0 Wiki
Ağaç: e215b55245
Dallar Biçim İmleri
launcher
/
specs
/
tech-architecture
/
dsv41-1017-tuning.md

dsv41-1017-tuning.md 18 KB
Geçmiş Ham

DSV4.1-1017 4x SGLang — performance A/B (2026-10-18)

Lane: dsv41-flash-1017-4x-sglang (Aiden production-1.7.1 + the baked 1017 encoder patch, image dsv41-sglang-1017:v1). Venue: 4nodes-switch (dgx1-4), 200G RoCE switch, weights mmap'd from the NAS HF cache, Engram rows node-local ([[staging]] sparse-copy). The live production lane on dgx58-switch was not touched during this campaign.

Question: the lane measures pp ~3.0k t/s / tg ~45 tok/s (c1) while a comparable community lane (knapcio v2.4 TP4, SGLang dev-dsv41, same 1017 checkpoint on 4 Sparks) reports pp 5,385 / tg 63.9 (c1) and 94.1 (c4). Which of the visible config differences actually move our numbers?

Method

Two instruments, both run from pluto.lan (off the Mac's WiFi path):

  1. sm121 bench --perf-only — llama-benchy pp2048/tg128, c1/c2, depth 0/4K/8K. Same harness/version as every other lane report (v2.8.0).
  2. /tmp/f1017/probe.py (kept with the campaign) — 2 discarded warm-up passes + N=5 measured passes of a cold ~2048-token prompt (unique nonce → no radix hit) asking for a 150-word paragraph; decode rate is (completion_tokens-1)/(last_delta - ttft) using the streamed usage block (speculative decoding emits multi-token deltas, so counting SSE events undercounts by ~2x). Median of the 5. This is the tie-breaker: llama-benchy rows on this stack scatter ±30% run to run, the probe does not.

Boot cost per config on this lane is 26-40 min (476 GB of fp8 weights off the NAS + CUDA-graph capture), so variants were kept few and chosen from the community configs rather than swept blindly.

Results

llama-benchy (pp t/s / tg t/s), single pass each:

config d0 c1 d0 c2 d4K c1 d8K c1 d8K c2
baseline (ep-size 4, chunk 2048, ctx 512K) 3,019 / 44.9 2,597 / 52.6 3,038 / 46.6 2,870 / 39.6 2,430 / 53.6
--enable-deepseek-v4-fp4-indexer 2,942 / 45.1 2,128 / 55.5 3,019 / 48.5 2,829 / 43.6 2,132 / 51.3
ep-size 2 3,054 / 61.0 1,089 / 58.8 2,372 / 42.3 1,965 / 41.1 1,753 / 56.9
ep-size 2 (repeat) 2,485 / 43.2 2,607 / 60.5 3,009 / 44.1 2,692 / 48.6 1,874 / 65.5
ep-size 4 + chunk 8192 + ctx 1M 3,275 / 42.3 — (crash, see below) — — —
final: ep-size 2, chunk 2048, ctx 1M (quality run) 2,358 / 45.6 1,916 / 67.8 2,533 / 45.4 2,859 / 45.8 2,677 / 38.5

probe.py (median of 5, cold 2K prompt → decode tok/s and TTFT≈prefill):

config TTFT decode c1
ep-size 2, chunk 2048, ctx 512K 0.754 s 41.7
ep-size 4 + chunk 8192 + ctx 1M 0.711 s 37.8 / 37.5 (two runs)
ep-size 2 + chunk 8192 + ctx 1M 0.702 s 43.0
ep-size 2, chunk 2048, ctx 1M (final) 0.76 s 38.8 / 41.8 (two runs)

Quality (tool-eval-bench v2.8.0, seed 42, hardmode, thinking on):

config score responsiveness median turn failures
baseline EP4 / 512K 87/100 (168/194) 71 1.7 s TC-68/74/88/89/91/92
final EP2 / 1M 87/100 (168/194) 75 1.4 s TC-68/71/74/88/89/92

The two runs swap one scenario each way (TC-91 vs TC-71) — single-trial noise at the same score. Adopted in recipes/dsv41-flash-1017-4x-sglang (commit fe0d386 in the recipe repo): --ep-size 2, --context-length 1048576, --chunked-prefill-size 2048, SPARK_ENGRAM_MAX_IDS=65536.

Findings

  1. --ep-size 2 (TP4/EP2, the knapcio shape) is the one real decode win. Probe: 41.7-43.0 vs 37.5-37.8 at EP4. It halves the per-step MoE dispatch/combine fan-out (2 nodes instead of 4) at the cost of keeping the full expert set on each node pair — memory is fine (KV pool stays ~9.2M tokens, ~12-13 GB GPU headroom left). Adopted as the lane default.
  2. --enable-deepseek-v4-fp4-indexer is neutral-to-negative here. Decode unchanged within noise, c2/8K rows dropped ~15-20%. Reverted; the flag is experimental in this build and the DSA indexer is not our bottleneck.
  3. --chunked-prefill-size 8192 is blocked by an Engram coupling, not by performance. The per-chunk Engram lookup is max_prefill_tokens x hash columns (~2 x chunk x 12): chunk 2048 → 49,152 ids, which is exactly what the recipe's SPARK_ENGRAM_MAX_IDS=65536 covers; chunk 8192 → 190,608 ids and the engine dies mid-request with RuntimeError: engram layer 1: a lookup of 190608 ids exceeds SPARK_ENGRAM_MAX_IDS=131072. Raising the id budget (≥ 196,608, e.g. 262144) is the prerequisite for any larger chunk. Kept at 2048 by choice: per the agent-serving findings, 2048 gives ~2.9x the decode share of 8192 under a concurrent prefill, which is the interactive-agent trade we want. (At a 2048-token prompt the chunk size does not change pp at all — the first cell of the 8192 run was 3,275 vs 3,019 baseline, inside noise.)
  4. --context-length 1048576 costs nothing (the KV pool is sized by --mem-fraction-static, not by the context cap: 9.18M tokens with the 1M cap vs 9.57M at 512K) and matches DeepSeek's 1017 recommendation. Adopted.
  5. The remaining gap to the knapcio number is not in these flags: their stack (dev-dsv41 + Mia adapter: fast loader, L2 prefetch, chunked indexer v3, mxfp8 b12x MoE, loop-abort) is a different overlay. Porting that is a separate project, not a flag flip.

Operational note: slow-NAS boots

One boot attempt died with TP rank 0 could finish the model loading, but there are other ranks that didn't finish loading while ranks 1/3 still had shards in flight — the NAS read rate had collapsed to ~1/6 of the morning's (48 shards took ~95 min instead of 13), rank 0 finished first and the TCPStore closed under the stragglers. Nothing in the config was wrong; a retry on the same config booted cleanly with 13.5 GB GPU headroom. Two things follow: (a) a boot during NAS contention can fail on rank desync, so retry before suspecting the config; (b) the engram/weight pre-staging on node-local storage is what keeps a restart cheap — the 26-95 min here is all NAS weight read.

Depth sweep (final config, llama-benchy pp2048/tg128, 2 runs per point)

ctx rows = prefill the full depth then decode 128; pp2048 rows = a 2048-token suffix at that depth (the agent-shaped case).

depth ctx pp (c1/c2) ctx tg (c1/c2) pp2048 tg (c1/c2)
16,384 3,530 / 2,279 39.0 / 32.7 46.5 / 62.5
65,536 1,532 / 2,151 32.0 / 6.0 47.3 / 62.1
131,072 1,545 / 1,561 36.1 / 2.7 42.1 / 61.2
262,144 1,447 / 1,525 50.5 / 1.4 46.4 / 67.5

Reading: single-stream decode after a 262K prefill is fine (50.5), and the agent-shaped case (short suffix at depth) holds 42-67 tok/s through the whole range. What collapses is two concurrent deep conversations: the c2 ctx rows fall to 6.0 / 2.7 / 1.4 tok/s because both streams' prefills serialize through the 2048-token chunk loop and decode gets the interleaved slots only. This is the documented prefill-vs-decode contention (see dgx-spark-agent-serving-findings.md), not a regression of this config. If long-context multi-session serving becomes the goal, the lever is --chunked-prefill-size (with SPARK_ENGRAM_MAX_IDS raised accordingly) or an admission cap, not more KV.

Reasoning-effort A/B (2026-10-18 evening)

SGLANG_DSV41_REASONING_EFFORT 100 (Aiden's default, our lane) vs 75 (knapcio's default), same EP2/2048/1M config, same harness:

effort score Hard Mode responsiveness median turn tg c1 (d0)
100 87/100 42/56 (75%) 75 1.4 s 45.6
75 86/100 41/56 (73%) 75 1.4 s 40.3

The effort lever does not pay on 1017. The "max costs 3.5x the reasoning tokens" observation belongs to the old V4.1-Flash; 1017 at max already spends ~40% of that (stu.miller, forum 382725 #266), so dropping to 75 buys no measurable turn time here and costs a point. Lane stays at 100.

Not yet measured

  • DSPARK --speculative-dspark-block-size sweep (3/5/7); accept length on the live lane sits at 3.0-5.2 with block 5.

  • The 2x alternative: thread 386121 / x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark (TensorFold + vLLM on EXL3 2.9bpw, decode c1 59-103 tok/s on two Sparks) is a different checkpoint + engine; a dgx1+dgx2 (2nodes-dac) lane is the natural place to try it.

    The TensorFold 2x alternative (measured 2026-10-19)

Upstream x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark (forum 386121) on 2nodes-dac (dgx1 head + dgx2 worker), EXL3 2.9bpw weights on node-local disk, TensorFold 0.6.0 + Jay Leaton patches, 262144 context, thinking on / effort 75. Bring-up detail + artifacts: recipes/dsv41-flash-1017-2x-tensorfold/NOTE.md.

metric TensorFold 2x 4x SGLang (EP2/1M)
quality (v2.8.0, seed 42, hardmode) 89/100 87/100
responsiveness / median turn 23 / 6.7 s 75 / 1.4 s
decode c1 (probe median) 50.4 tok/s 38.8-43.0 tok/s
prefill for the same ~2K prompt 5.4 s (~380 tok/s) 0.75 s (~2,700 tok/s)
nodes 2 4

So the trade is explicit: half the cluster, better decode and no quality loss, at 4-5x the turn latency (prefill is ~7x slower — the harness scores it as responsiveness 23). TensorFold 2x is the long-generation / batch shape; the 4x SGLang lane remains the interactive-agent shape. Not purity-matched: thinking effort 75 vs 100, EXL3 2.9bpw vs fp8.

Status 2026-10-19: the lane is now a real sm121 recipe — recipes/dsv41-flash-1017-2x-tensorfold + docker-images/dsv41-tf-1017 + the tensorfold_1017_prepare built-in host script. sm121 serve --recipe dsv41-flash-1017-2x-tensorfold --cluster 2nodes-dac stages, builds, syncs and launches it end to end (API ready ~4 min after the 24.5 GB image sync), and sm121 stop closes the runlock. The upstream repo's vLLM profile on the same weights is blocked in our environment — the exact failure and the fix options are in the last section of that lane's NOTE.md (the NFS Engram volume carries only NAS symlinks that dangle inside the worker's vLLM container; rsync mode wants 196 GiB free on the worker).

Four-lane comparison (2026-10-20/21 night shift)

lane quality (v2.8.0, seed 42, hardmode) responsiveness / median turn pp c1 tg c1 / c2 / c4 KV pool
1017 SGLang 4x (this lane, EP2/1M) 87/100 75 / 1.4 s 3,019 45.6 / 67.8 / — 9.30M
1017 TensorFold 2x (EXL3 2.9bpw) 89/100 23 / 6.7 s ~380 (2K prompt) 50.4 / — / — 26.6M
1017 FLARE 4x (local-inference-lab gb10 pkg) unmeasurable — crashes — ~2.4k 22–27 / — / — 26.6M
Beam-1.0 NVFP4 4x (different model) 90/100 34 / 4.7 s 2,501 26.2 / 41.1 / 62.3 6.81M

FLARE detail: the official 4-Spark gb10 package validates (sm_121, 1M ctx, 16 concurrency, MTP 1-6, 74.7 GiB/rank, 130 graphs) and runs over RoCE with two rails per peer, but decodes at 22-27 tok/s and dies on the first agent-sized prompt — tensor-parallel peer failed: RoCE proxy failed: two-shot op N abandoned: kernel timed out at seq N (short chat prompts are fine). Full bench: 0/100, every scenario 500/connection-refused. The FLARE runtime itself is fine on GB10 (Qwen3.8-Flash-Next tp1: 43.3 tok/s, 1.3 s TTFT, ~1 s load) with one API gap: tool_choice: "required" returns HTTP 500.

Verdict: the EP2/1M SGLang lane stays the interactive default; Beam is the quality leader for hard batch work; TensorFold remains the 2-node batch shape; FLARE is not viable for us at this package revision.

DeepInfer (deepseek-ai/DeepInfer) — assessed 2026-10-21, not runnable on GB10

DeepSeek open-sourced their production inference framework (14.9k stars, Apache 2.0). It targets exactly our model family — "Supported models: DeepSeek-V4.1-Flash", default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 — with optimised kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, TileLang), attention-DP/CP + MoE EP with EPLB, DSpark with confidence-scheduled verification, SWA bounded replay, async scheduling, batch-invariant inference, deterministic sampling and structured outputs.

Three blockers, in order of severity:

  1. Offline inference only. The README states the release is offline and that a complete service still needs "a KV cache system", "VE/Prefill/Decode disaggregation" and "load balancing across data-parallel replicas" — i.e. no server. There is no HTTP API to point an agent at; it is a Python library (examples/offline_inference.py).
  2. SM100-only. deep_cluster/hardware/device.py::is_sm_major_10() gates on compute capability major 10 (SM100/101/103) and get_bf16_tflops_limit() raises RuntimeError("unknown GPU type") otherwise. GB10 is SM121. The vendored kernels (FlashMLA/DeepGEMM/DeepEP) are SM90/SM100-targeted too.
  3. Build cost. DEEPINFER_ACCELERATOR=nvidia ./build.sh needs GCC >= 13, CUDA 13, Python 3.12, four vendored CUDA projects built from submodules, plus an NCCL 2.32.3 force-reinstall; arm64 is not a documented target.

Verdict: not usable this shift, and a port is a project (kernels to sm121 + write the serving layer), not an evening. What is worth stealing:

  • SWA bounded replay (DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact, ..._KEEP_TOKENS=140): replay only a bounded suffix of the decoder prefill, keeping at least the sliding-window span. This is a prefill-latency idea we could look for in the SGLang/VL stack for our 1017 lane.
  • Confidence-scheduled verification (DEEPINFER_ENABLE_MTP_VARLEN=1 + DEEPINFER_USE_NEXTN_DECODE): schedule verification by draft confidence — the same family as the "adaptive verification" knob Beam's DFlash2 wanted.
  • Batch-invariant inference — determinism across batch composition; relevant if we ever need reproducible evals.

Re-check when the repo grows a serving path or an sm120/sm121 target.

Внешние контракты (2026-10-23): FLARE gb10-пакет и рецепт eugr

local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE

У репозитория есть ветка gb10 (README: /raw/gb10/README.md) — подписанный FLARE-пакет ровно под наши четыре Spark:

  • deepseek-v41-flash-1017-tp4-c16-dspark5.safetensors: metadata-only первичный файл + 17 шардов ≈ 511.3 GB (12 весовых + 5 графовых/вспомогательных, каждый < 64 GB), lossless CSF-чекпойнт; gb202 — отдельный пакет под 4 дискретные GPU на одной машине;
  • заявлено: TP4, 16 одновременных запросов, DSpark до 5 драфтов с adaptive verification, prefill чанками 4096, контекст 1,048,576 на запрос (по KV), >= 515 GB на первом Spark;
  • рекомендованный набор флагов (это то, что надо воспроизвести в наших лейнах):
флаг FLARE наш аналог в vLLM у нас сейчас
--embedding-table-storage disk --engram-config '{"table_memory":"disk"}' ✅ есть
--embedding-table-resident-scales disk_resident_scales: true в том же JSON (≈ 5.72 GiB RAM на V4.1) ❌ не включено — A/B
--embedding-table-cache-size 1GiB строкового кэша строк в EngramConfig нет (только table_memory/disk_resident_scales/projection_tp) —
--max-concurrency 16 --max-num-seqs 16 (у нас 4) ❌ 4
--mtp-draft-tokens 5 --mtp-adaptive-verification --speculative-config dspark k=5 (у нас без enable_adaptive_verification) частично

Вывод: disk_resident_scales — наш аналог «resident scales», он прямо обещан в конфиге стокового vLLM 0.32 (vllm/config/engram.py), и это следующий дешёвый шаг тюнинга энгрэма (скейлы E8M0 не читаются с диска на каждом промахе; ~5.7 GiB RAM на ноде).

Рецепт eugr (eugr/spark-vllm-docker, recipes/deepseek-v4-flash.yaml)

1017-рецепта у него нет (только не-1017 deepseek-v4-flash на TP2), но форма полезна как чек-лист того, чего в нашем лейне нет:

  • --tokenizer-mode deepseek_v4 — ❗ у нас не задан;
  • --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' плюс --default-chat-template-kwargs.thinking=true / .reasoning_effort=high — ❗ у нас только --reasoning-parser;
  • env VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (у него max_model_len 500000 при GMU 0.8) и VLLM_USE_BREAKABLE_CUDAGRAPH=1, DG_JIT_USE_NVRTC=0;
  • спека {"method":"mtp","num_speculative_tokens":2} — у нас dspark k=5;
  • бэкенд ray, у нас mp.

Проверять по одному, с замером: --tokenizer-mode, --default-chat-template-kwargs, disk_resident_scales, затем --max-num-seqs 16.

512K + стандартный алиас (2026-10-23, 20:16 UTC)

Лейн dsv41-flash-1017-4x-vllm032 поднят с --max-model-len 524288 и --served-model-name deepseek-v4.1-flash (было 131072 / dsv41). Проверено: /v1/models → deepseek-v4.1-flash 524288; замер на уникальном промпте 24,005 токенов: TTFT 8.0 s, prefill 2,996 t/s — то есть 512K-окно не сломало префилл (с локальным энгрэмом держится прод-уровень ~3k t/s).