launcher/specs/tech-architecture/qwen38-recipe-state.md

Qwen3.8-27B recipes — measured state

Real, on-hardware performance of the qwen38-* recipe family on a single DGX Spark (dgx1, GB10 sm_121, 128 GB unified memory), measured 2026-08-22. Update this table as you re-benchmark newer images / configs. Each row is a like-for-like measurement unless noted.

Benchmark: tool-eval-bench bench --perf-only --pp 2048 --tg 128 @d0 (free-gen, single stream, concurrency 1) unless noted. SGLang rows used a direct SSE stream timer (tool-eval-bench's bench/run cannot parse SGLang's SSE response shape — see "Measurement caveats").

Clock: all rows since the dgx1 thermal shutdown are measured under the qwen38 before_serve clock lock (nvidia-smi -lgc 0,2000 — 2000 MHz cap vs 3003 MHz max). The earlier vLLM rows (22.8) were at full boost; the lock costs ~5% (22.8 → 21.7).

Engine / spec-decode matrix (decode tok/s, prefill tok/s)

Config	Image	decode	prefill	Notes
unsloth NVFP4 + MTP (k=5)	vllm-gb10 v0.27.1-gb10.2	21.7	2,177	WINNER (21.7 @ lock; 22.8 @ boost)
unsloth NVFP4 + DSpark (k=7)	vllm-gb10 v0.27.1-gb10.2	22.8	2,563	ties decode, better prefill; needs patched draft
RedHatAI NVFP4 + MTP (k=5)	vllm-gb10 v0.27.1-gb10.2	19.2	2,153	slower than unsloth NVFP4
Int4 AutoRound + MTP (k=4)	vllm-gb10 v0.27.1-gb10.2	19.6	921	mixed-4bit, slower prefill
FP8 + DSpark (k=7)	vllm-gb10 v0.27.1-gb10.2	17.7	1,133	uses patched draft
FP8 + MTP (k=5)	vllm-gb10 v0.27.1-gb10.2	14.0	701	surprisingly slow
BF16 + MTP (k=5)	vllm-gb10 v0.27.1-gb10.2	10.1	1,194	quality baseline
SGLang RadixArk NVFP4 + DSpark	lmsysorg/sglang:qwen38-27b	9.30	—	stock image (pre-DSpark... pre-DFlash2 tag)
SGLang RadixArk NVFP4 + DFlash2	...-dflash2-minoverlay	9.22	—	DFlash2 fork (c14312a66)

Agentic / edit-heavy quality (tool-eval-bench run --short)

Config	score	median turn
unsloth NVFP4 + MTP	100/100 (28/28)	4.7 s
unsloth NVFP4 + DSpark	100/100 (30/30)	5.2 s

Winner

qwen38-27b-nvfp4-unsloth — vLLM+MTP on the unsloth NVFP4 checkpoint. Best decode (21.7 tok/s under the lock), 100/100 agentic quality, and the MTP head is in-checkpoint (no extra draft to download). This is what pi is pointed at via ~/.pi/agent/models.json → provider dgx1 → model qwen3.8-27b.

Key decisions & why

• Image: ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2 (latest immutable, vLLM 0.27.1, torch 2.13+cu130, FlashInfer from source). Full qwen3_5 + MTP support. Used over eugr images (unpinned nightly) per the repo rule.
• MTP over vLLM-DSpark for the winner: both tie on decode (22.8), MTP is simpler (no draft download/patch), and MTP has the slightly better agentic turn-around (4.7s vs 5.2s). The vLLM DSpark recipes are kept as experimentals.
• SGLang (DSpark and DFlash2) is NOT faster here. Despite the forum's 52-87 tok/s claims, both measured ~9 tok/s for fresh prompts — 2.3× slower than vLLM+MTP under the same clock lock. The forum numbers are an ndec.py cached-code-context artifact, not per-request generation throughput.
• unsloth NVFP4 beat RedHatAI NVFP4 (21.7 vs 19.2 decode). Both are compressed-tensors NVFP4. RadixArk (modelopt) NVFP4 was only needed for SGLang's quantized-head path (see below).
• num_speculative_tokens: 5 for the MTP recipes (helge/forum peak; the single MTP layer cycles), 4 for Int4 (SlopOps). DSpark uses block 7 = 8 draft tokens (requires max-num-batched-tokens ≥ 16384 to avoid the max_num_scheduled_tokens warning).

Measurement caveats

• SGLang rows used a hand-rolled SSE stream timer (count content + reasoning_content deltas, divide time by token count). tool-eval-bench bench/run cannot parse SGLang's SSE stream (llama-benchy "no usable throughput metrics"; run times out per-turn). vLLM rows used the standard bench --perf-only.
• The forum's high SGLang numbers use bench/ndec.py (2-call net-decode A/B on a large cached code prefix) — measures marginal edit cost, not per-request generation. Not comparable to the table above.
• thinking/reasoning_content is counted as token generation; the winner's 21.7 tok/s is measured with thinking on (default) counting both content and reasoning deltas.

Recipes in this family

Recipe	Target	spec-decode	Status
qwen38-27b-nvfp4-unsloth	unsloth NVFP4	MTP (k=5)	winner — serving on dgx1
qwen38-27b-nvfp4	RedHatAI NVFP4	MTP (k=5)	measured
qwen38-27b-nvfp4-unsloth-dspark	unsloth NVFP4	DSpark (k=7)	experimental
qwen38-27b-fp8-dspark	FP8	DSpark (k=7)	experimental
qwen38-27b-fp8	FP8	MTP (k=5)	measured
qwen38-27b-int4	Pilcothink Int4	MTP (k=4)	measured
qwen38-27b	Qwen BF16	MTP (k=5)	measured baseline
qwen38-27b-nvfp4-sglang-dspark	RadixArk NVFP4	SGLang DSpark	experimental (slow)
qwen38-27b-nvfp4-sglang-dflash2	RadixArk NVFP4	SGLang DFlash2	experimental (slow)

vLLM DSpark draft note

The vLLM method:"dspark" paths (qwen38-27b-fp8-dspark, qwen38-27b-nvfp4-unsloth-dspark) load a dedicated draft model that must live at a local dir with its config.json arch patched:

• RadixArk Qwen3.8-27B-DSpark declares architectures: ["DSparkDraftModel"], which vLLM maps to the DeepSeek-V4 DSpark code path (hc_mult attribute error). It must be re-labelled Qwen3DSparkModel + draft_vocab_size set to route to qwen3_dspark.
• The patched copy lives on dgx1 at /home/kilork/dspark-draft, bind-mounted into the container at /dspark-draft via the recipe's [[extra_mounts]].
• This is fragile; if you only run the winner, drop these two recipes.

SGLang image note

lmsysorg/sglang:qwen38-27b (cookbook) predates DFlash2 (upstream merged c14312a66 on 2026-08-19). DFlash2 is overlaid by hand: lmsysorg/sglang:qwen38-27b-dflash2-minoverlay (5 sha256-verified modules from the MiaAI repo, built on dgx1). That minoverlay already bakes in the quantized-head selector fix, so it serves the modelopt NVFP4 target. SGLang only accelerates modelopt NVFP4 — unsloth/RedHatAI (compressed-tensors) fall back to a slow unquantized path.

Re-benchmarking

1. Serve a recipe on the qwen38 single-node cluster (dgx1) — the before_serve clock lock is applied automatically.
2. For vLLM: tool-eval-bench bench --base-url http://dgx1.lan:30000/v1 --model qwen3.8-27b --perf-only --pp 2048 --tg 128 --concurrency 1.
3. For SGLang: use a direct SSE stream timer (see Measurement caveats).
4. Record the new numbers in the table above; note the image tag + date.