# Qwen3.8-27B recipes — measured state

> Real, on-hardware performance of the `qwen38-*` recipe family on a single
> DGX Spark (dgx1, GB10 sm_121, 128 GB unified memory), measured 2026-08-22.
> Update this table as you re-benchmark newer images / configs. Each row is a
> like-for-like measurement unless noted.
>
> **Benchmark:** `tool-eval-bench bench --perf-only --pp 2048 --tg 128 @d0`
> (free-gen, single stream, concurrency 1) unless noted. SGLang rows used a
> direct SSE stream timer (tool-eval-bench's bench/run cannot parse SGLang's
> SSE response shape — see "Measurement caveats").
>
> **Clock:** all rows since the dgx1 thermal shutdown are measured under the
> qwen38 `before_serve` clock lock (`nvidia-smi -lgc 0,2000` — 2000 MHz cap vs
> 3003 MHz max). The earlier vLLM rows (22.8) were at full boost; the lock costs
> ~5% (22.8 → 21.7).

## Engine / spec-decode matrix (decode tok/s, prefill tok/s)

| Config | Image | decode | prefill | Notes |
|---|---|---|---|---|
| **unsloth NVFP4 + MTP (k=5)** | vllm-gb10 `v0.27.1-gb10.2` | **21.7** | 2,177 | **WINNER** (21.7 @ lock; 22.8 @ boost) |
| unsloth NVFP4 + DSpark (k=7) | vllm-gb10 `v0.27.1-gb10.2` | 22.8 | **2,563** | ties decode, better prefill; needs patched draft |
| RedHatAI NVFP4 + MTP (k=5) | vllm-gb10 `v0.27.1-gb10.2` | 19.2 | 2,153 | slower than unsloth NVFP4 |
| Int4 AutoRound + MTP (k=4) | vllm-gb10 `v0.27.1-gb10.2` | 19.6 | 921 | mixed-4bit, slower prefill |
| FP8 + DSpark (k=7) | vllm-gb10 `v0.27.1-gb10.2` | 17.7 | 1,133 | uses patched draft |
| FP8 + MTP (k=5) | vllm-gb10 `v0.27.1-gb10.2` | 14.0 | 701 | surprisingly slow |
| BF16 + MTP (k=5) | vllm-gb10 `v0.27.1-gb10.2` | 10.1 | 1,194 | quality baseline |
| SGLang RadixArk NVFP4 + DSpark | `lmsysorg/sglang:qwen38-27b` | 9.30 | — | stock image (pre-DSpark... pre-DFlash2 tag) |
| SGLang RadixArk NVFP4 + DFlash2 | `...-dflash2-minoverlay` | 9.22 | — | DFlash2 fork (c14312a66) |

### Agentic / edit-heavy quality (tool-eval-bench `run --short`)

| Config | score | median turn |
|---|---|---|
| unsloth NVFP4 + MTP | **100/100** (28/28) | 4.7 s |
| unsloth NVFP4 + DSpark | 100/100 (30/30) | 5.2 s |

## Winner

**`qwen38-27b-nvfp4-unsloth`** — vLLM+MTP on the unsloth NVFP4 checkpoint. Best
decode (21.7 tok/s under the lock), 100/100 agentic quality, and the MTP head is
in-checkpoint (no extra draft to download). This is what pi is pointed at via
`~/.pi/agent/models.json` → provider `dgx1` → model `qwen3.8-27b`.

## Key decisions & why

- **Image:** `ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2` (latest immutable,
  vLLM 0.27.1, torch 2.13+cu130, FlashInfer from source). Full `qwen3_5` + MTP
  support. Used over `eugr` images (unpinned nightly) per the repo rule.
- **MTP over vLLM-DSpark for the winner:** both tie on decode (22.8), MTP is
  simpler (no draft download/patch), and MTP has the slightly better agentic
  turn-around (4.7s vs 5.2s). The vLLM DSpark recipes are kept as experimentals.
- **SGLang (DSpark and DFlash2) is NOT faster here.** Despite the forum's
  52-87 tok/s claims, both measured ~9 tok/s for fresh prompts — 2.3× slower
  than vLLM+MTP under the same clock lock. The forum numbers are an `ndec.py`
  cached-code-context artifact, not per-request generation throughput.
- **`unsloth` NVFP4 beat `RedHatAI` NVFP4** (21.7 vs 19.2 decode). Both are
  compressed-tensors NVFP4. `RadixArk` (modelopt) NVFP4 was only needed for
  SGLang's quantized-head path (see below).
- **`num_speculative_tokens`:** 5 for the MTP recipes (helge/forum peak; the
  single MTP layer cycles), 4 for Int4 (SlopOps). DSpark uses block 7 = 8 draft
  tokens (requires `max-num-batched-tokens` ≥ 16384 to avoid the
  `max_num_scheduled_tokens` warning).

## Measurement caveats

- **SGLang rows** used a hand-rolled SSE stream timer (count `content` +
  `reasoning_content` deltas, divide time by token count). `tool-eval-bench`
  `bench`/`run` cannot parse SGLang's SSE stream (llama-benchy "no usable
  throughput metrics"; `run` times out per-turn). vLLM rows used the standard
  `bench --perf-only`.
- The forum's high SGLang numbers use `bench/ndec.py` (2-call net-decode A/B on
  a large cached code prefix) — measures *marginal* edit cost, not per-request
  generation. Not comparable to the table above.
- `thinking`/`reasoning_content` is counted as token generation; the winner's
  21.7 tok/s is measured with thinking **on** (default) counting both content
  and reasoning deltas.

## Recipes in this family

| Recipe | Target | spec-decode | Status |
|---|---|---|---|
| `qwen38-27b-nvfp4-unsloth` | unsloth NVFP4 | MTP (k=5) | **winner — serving on dgx1** |
| `qwen38-27b-nvfp4` | RedHatAI NVFP4 | MTP (k=5) | measured |
| `qwen38-27b-nvfp4-unsloth-dspark` | unsloth NVFP4 | DSpark (k=7) | experimental |
| `qwen38-27b-fp8-dspark` | FP8 | DSpark (k=7) | experimental |
| `qwen38-27b-fp8` | FP8 | MTP (k=5) | measured |
| `qwen38-27b-int4` | Pilcothink Int4 | MTP (k=4) | measured |
| `qwen38-27b` | Qwen BF16 | MTP (k=5) | measured baseline |
| `qwen38-27b-nvfp4-sglang-dspark` | RadixArk NVFP4 | SGLang DSpark | experimental (slow) |
| `qwen38-27b-nvfp4-sglang-dflash2` | RadixArk NVFP4 | SGLang DFlash2 | experimental (slow) |

### vLLM DSpark draft note

The vLLM `method:"dspark"` paths (`qwen38-27b-fp8-dspark`,
`qwen38-27b-nvfp4-unsloth-dspark`) load a dedicated draft model that must live
at a **local dir** with its `config.json` arch patched:

- RadixArk `Qwen3.8-27B-DSpark` declares `architectures: ["DSparkDraftModel"]`,
  which vLLM maps to the *DeepSeek-V4* DSpark code path (`hc_mult` attribute
  error). It must be re-labelled `Qwen3DSparkModel` + `draft_vocab_size` set to
  route to `qwen3_dspark`.
- The patched copy lives on dgx1 at `/home/kilork/dspark-draft`, bind-mounted
  into the container at `/dspark-draft` via the recipe's `[[extra_mounts]]`.
- This is fragile; if you only run the winner, drop these two recipes.

### SGLang image note

`lmsysorg/sglang:qwen38-27b` (cookbook) predates DFlash2 (upstream merged
`c14312a66` on 2026-08-19). DFlash2 is overlaid by hand:
`lmsysorg/sglang:qwen38-27b-dflash2-minoverlay` (5 sha256-verified modules from
the MiaAI repo, built on dgx1). That minoverlay already bakes in the
quantized-head selector fix, so it serves the modelopt NVFP4 target. SGLang only
accelerates *modelopt* NVFP4 — `unsloth`/`RedHatAI` (compressed-tensors) fall
back to a slow unquantized path.

## Re-benchmarking

1. Serve a recipe on the `qwen38` single-node cluster (dgx1) — the `before_serve`
   clock lock is applied automatically.
2. For vLLM: `tool-eval-bench bench --base-url http://dgx1.lan:30000/v1 --model qwen3.8-27b --perf-only --pp 2048 --tg 128 --concurrency 1`.
3. For SGLang: use a direct SSE stream timer (see `Measurement caveats`).
4. Record the new numbers in the table above; note the image tag + date.