Add Qwen3.8-27B recipes + measured state doc

- 9 qwen38-* recipes on ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2
  (latest immutable, vLLM 0.27.1, full qwen3_5 + MTP). Target the 5 cached
  Qwen3.8-27B variants (BF16, FP8, unsloth+RedHatAI NVFP4, Int4 AutoRound)
  plus vLLM-DSpark and SGLang-DSpark/DFlash2 experimentals.
- Winner: qwen38-27b-nvfp4-unsloth (vLLM+MTP k=5, ~21.7 t/s decode @ 2000 MHz
  clock lock, 100/100 agentic tool-eval). Served on dgx1; pi points at it via
  ~/.pi/agent/models.json -> dgx1 -> qwen3.8-27b.
- specs/tech-architecture/qwen38-recipe-state.md: the measured matrix, why
  each decision, SGLang-not-faster finding, and re-benchmark steps.
- AGENTS.md: link the qwen38 recipe-state doc under Serving notes.

AGENTS.md

@@ -395,6 +395,17 @@ concurrent prefill, for 7.7% prefill throughput. Use the
 `deepseek-v4-flash-eugr-agent` recipe (2048) for interactive agent traffic;
 use `deepseek-v4-flash-eugr` (8192) for exclusive long-context batch jobs.
 
+### Qwen3.8-27B recipes (measured)
+
+`qwen38-*` recipes serve the Qwen3.8-27B family on a single DGX Spark (dgx1,
+`qwen38` cluster). The tested winner is **`qwen38-27b-nvfp4-unsloth`** (vLLM
+`v0.27.1` + MTP k=5 on unsloth NVFP4): ~21.7 tok/s decode, 100/100 agentic
+tool-eval, and the MTP head is in-checkpoint. SGLang (DSpark *and* the DFlash2
+fork) measured ~2.3× slower on this box despite the forum's 52-87 tok/s claims
+(an `ndec.py` cached-context artifact). See
+`specs/tech-architecture/qwen38-recipe-state.md` for the full measured matrix,
+why each decision was made, and how to re-benchmark.
+
 ## Config fields
 
 ```toml

recipes/qwen38-27b-fp8-dspark/recipe.toml

@@ -0,0 +1,92 @@
+# Qwen3.8-27B (FP8 + DSpark draft) — 1× DGX Spark / GB10 (sm_121), TP=1.
+#
+# DSpark: a dedicated semi-autoregressive draft model (~2.6 GB, 5 layers,
+# block_size 7) proposes blocks of tokens that the full model verifies in one
+# weight-read. 3× cheaper per guess than the in-checkpoint MTP head, so it wins
+# on edit-heavy / structured workloads (blog: 45-48 t/s edit, 17 t/s free-gen,
+# 66 t/s aggregate @ c10 on the FP8 target). DSpark blocks are verifier-rejected,
+# so no quality risk.
+#
+# Target: Qwen/Qwen3.8-27B-FP8 (official FP8 quant, 28.75 GB).
+# Draft:  RadixArk/Qwen3.8-27B-DSpark (arch DSparkDraftModel, bf16, 5 layers,
+#         dspark_block_size 7). Downloaded via hfrs into the node HF cache.
+#
+# Image: ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2
+#   vLLM 0.27.1 (installed 0.27.2.dev0+g6e448d0ea), supports method "dspark"
+#   (SpeculativeMethod Literal includes dspark) + Qwen3DSparkModel draft.
+#
+# PREREQUISITES:
+#   Target + draft cached on the node (dgx1 ~/.cache/huggingface → NAS):
+#       hfrs download Qwen/Qwen3.8-27B-FP8
+#       hfrs download RadixArk/Qwen3.8-27B-DSpark
+#
+# REFERENCE:
+#   https://blog.kubesimplify.com/qwen3-8-27b-on-dgx-spark  (DSpark section)
+#   https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-on-dual-sparks/380350
+
+engine = "vllm"
+name = "Qwen3.8-27B (FP8 + DSpark)"
+docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2"
+model = "Qwen/Qwen3.8-27B-FP8"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["vllm", "serve"]
+
+args = [
+    # KV cache & memory
+    "--kv-cache-dtype", "fp8",
+    "--max-model-len", "262144",
+    "--max-num-seqs", "6",
+    "--max-num-batched-tokens", "8192",
+    "--gpu-memory-utilization", "0.75",
+    "--served-model-name", "qwen3.8-27b",
+    # Engine backends (dense GQA — FlashInfer)
+    "--attention-backend", "flashinfer",
+    # Compilation / perf
+    "--optimization-level", "3",
+    "--performance-mode", "throughput",
+    # Chunked prefill + prefix caching
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--enable-flashinfer-autotune",
+    # DSpark speculative decoding — dedicated draft model (not the in-checkpoint
+    # MTP head). num_speculative_tokens must equal the draft's block size (7).
+    # The draft lives in a local dir (arch-patched copy of RadixArk/Qwen3.8-27B-DSpark
+    # re-labelled `Qwen3DSparkModel` so vLLM routes it to qwen3_dspark, not the
+    # DeepSeek-V4 DSpark code path) bind-mounted at /dspark-draft below.
+    "--speculative-config", """
+'{"method":"dspark","model":"/dspark-draft","num_speculative_tokens":7}'""",
+    # Tokenizer / reasoning (Qwen3 family)
+    "--tool-call-parser", "qwen3_xml",
+    "--reasoning-parser", "qwen3",
+    "--enable-auto-tool-choice",
+    "--generation-config", "auto",
+    "--default-chat-template-kwargs.preserve_thinking=true",
+    # Proven generation defaults (thinking mode; Qwen3.8 model-card best
+    # practices)
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'""",
+    "--trust-remote-code",
+]
+
+# DSpark draft model — arch-patched local copy of RadixArk/Qwen3.8-27B-DSpark,
+# mounted at /dspark-draft so vLLM loads it via the qwen3_dspark draft class.
+[[extra_mounts]]
+host = "/home/kilork/dspark-draft"
+container = "/dspark-draft"
+
+
+[env]
+HF_HUB_OFFLINE = "1"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+CUDA_VISIBLE_DEVICES = "0"
+CUDA_DEVICE_ORDER = "PCI_BUS_ID"
+CUDA_CACHE_MAXSIZE = "4294967296"

recipes/qwen38-27b-fp8/recipe.toml

@@ -0,0 +1,74 @@
+# Qwen3.8-27B (FP8) — 1× DGX Spark / GB10 (sm_121), TP=1.
+#
+# Official Qwen FP8 quant (29 GB). Dense 27B, GQA, hybrid DeltaNet+Attention,
+# MTP-1. Text + vision. As fast as NVFP4 typically, with full FP8 quality.
+#
+# Image: ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2
+#   vLLM 0.27.1, torch 2.13+cuda130, FlashInfer built from source,
+#   instanttensor. Stock vLLM — supports the qwen3_5 arch + MTP.
+#
+# Model: Qwen/Qwen3.8-27B-FP8
+#   https://huggingface.co/Qwen/Qwen3.8-27B-FP8
+#
+# PREREQUISITES:
+#   Weights cached on every node (dgx1 ~/.cache/huggingface → NAS):
+#       huggingface-cli download Qwen/Qwen3.8-27B-FP8
+#
+# REFERENCE:
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-mixedint4-autoround-optimized-for-a-single-dgx-spark/380248
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-on-dual-sparks/380350
+
+engine = "vllm"
+name = "Qwen3.8-27B (FP8)"
+docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2"
+model = "Qwen/Qwen3.8-27B-FP8"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["vllm", "serve"]
+
+args = [
+    # KV cache & memory
+    "--kv-cache-dtype", "fp8",
+    "--max-model-len", "262144",
+    "--max-num-seqs", "6",
+    "--max-num-batched-tokens", "8192",
+    "--gpu-memory-utilization", "0.75",
+    "--served-model-name", "qwen3.8-27b",
+    # Engine backends (dense GQA — FlashInfer)
+    "--attention-backend", "flashinfer",
+    # Compilation / perf
+    "--optimization-level", "3",
+    "--performance-mode", "throughput",
+    # Chunked prefill + prefix caching
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--enable-flashinfer-autotune",
+    # MTP speculative decoding (in-checkpoint draft)
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":5}'""",
+    # Tokenizer / reasoning (Qwen3 family)
+    "--tool-call-parser", "qwen3_xml",
+    "--reasoning-parser", "qwen3",
+    "--enable-auto-tool-choice",
+    "--generation-config", "auto",
+    "--default-chat-template-kwargs.preserve_thinking=true",
+    # Proven generation defaults (thinking mode; Qwen3.8 model-card best
+    # practices)
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'""",
+    "--trust-remote-code",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+CUDA_VISIBLE_DEVICES = "0"
+CUDA_DEVICE_ORDER = "PCI_BUS_ID"
+CUDA_CACHE_MAXSIZE = "4294967296"

recipes/qwen38-27b-int4/recipe.toml

@@ -0,0 +1,96 @@
+# Qwen3.8-27B (Pilcothink Int4 AutoRound + MTP) — 1× DGX Spark / GB10 (sm_121),
+# TP=1.
+#
+# Mixed 4-bit AutoRound quant (20 GB) tuned for a single DGX Spark. Sensitive
+# layers kept FP8/FP16; vision layers unquantized. MTP quantized + included.
+# Dense 27B, GQA, hybrid DeltaNet+Attention. Text + vision. The community has
+# heavily tuned vLLM flags for this exact build (thread 380248 post #39).
+#
+# Image: ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2
+#   vLLM 0.27.1, torch 2.13+cuda130, FlashInfer built from source,
+#   instanttensor. Stock vLLM — supports the qwen3_5 arch + MTP.
+#
+# Model: Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound
+#   https://huggingface.co/Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound
+#
+# PREREQUISITES:
+#   Weights cached on every node (dgx1 ~/.cache/huggingface → NAS):
+#       huggingface-cli download Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound
+#
+# REFERENCE:
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-mixedint4-autoround-optimized-for-a-single-dgx-spark/380248
+
+engine = "vllm"
+name = "Qwen3.8-27B (Int4 AutoRound)"
+docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2"
+model = "Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["vllm", "serve"]
+
+args = [
+    # KV cache & memory
+    "--kv-cache-dtype", "fp8",
+    "--max-model-len", "262144",
+    "--max-num-seqs", "6",
+    "--max-num-batched-tokens", "8192",
+    "--gpu-memory-utilization", "0.75",
+    "--served-model-name", "qwen3.8-27b",
+    # Engine backends (dense GQA — FlashInfer)
+    "--attention-backend", "flashinfer",
+    # Compilation / perf (AutoRound community tuning)
+    "--optimization-level", "3",
+    "--performance-mode", "throughput",
+    # Chunked prefill + prefix caching
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--enable-flashinfer-autotune",
+    # MTP speculative decoding (quantized draft)
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":4}'""",
+    # Tokenizer / reasoning (Qwen3 family)
+    "--tool-call-parser", "qwen3_xml",
+    "--reasoning-parser", "qwen3",
+    "--enable-auto-tool-choice",
+    "--generation-config", "auto",
+    "--default-chat-template-kwargs.preserve_thinking=true",
+    # Proven generation defaults (thinking mode; Qwen3.8 model-card best
+    # practices)
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'""",
+    "--trust-remote-code",
+]
+
+[env]
+# AutoRound community tuning (thread 380248 post #39): faster load + higher
+# compile level for the 4-bit kernels.
+VLLM_TORCH_COMPILE_LEVEL = "3"
+VLLM_TRITON_COMPILE_ON_LOAD = "1"
+VLLM_WARMUP_BEFORE_SERVING = "1"
+VLLM_SKIP_WARMUP = "false"
+TORCH_MATMUL_PRECISION = "high"
+NVIDIA_FORWARD_COMPAT = "1"
+NVIDIA_DISABLE_REQUIRE = "1"
+CUDA_DEVICE_MAX_CONNECTIONS = "1"
+VLLM_MARLIN_USE_ATOMIC_ADD = "1"
+FLASHINFER_DISABLE_VERSION_CHECK = "1"
+
+# Offline mode
+HF_HUB_OFFLINE = "1"
+
+# Architecture
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"
+
+# GPU memory management
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+
+# GPU device isolation
+CUDA_VISIBLE_DEVICES = "0"
+CUDA_DEVICE_ORDER = "PCI_BUS_ID"
+CUDA_CACHE_MAXSIZE = "4294967296"

recipes/qwen38-27b-nvfp4-sglang-dflash2/recipe.toml

@@ -0,0 +1,75 @@
+# Qwen3.8-27B (RadixArk NVFP4 + DFlash2 on SGLang) — 1× DGX Spark / GB10,
+# TP=1.
+#
+# SGLang + DFlash2 block-diffusion speculative decoding. Per the community
+# (MiaAI / forum 380350), DFlash2 on NVFP4 ties DSpark on code and *beats it*
+# on prose/essay and short-chat once counted right. This is the fork/engine
+# the forum's high tok/s numbers (52-61 code single, 87 TP=2) came from — NOT
+# the stock DSpark path.
+#
+# Image: lmsysorg/sglang:qwen38-27b-dflash2-minoverlay — a locally-built overlay
+#   of the 5 DFlash2 modules onto the cookbook qwen38-27b image. The cookbook
+#   tag itself predates DFlash2 (upstream merged commit c14312a66 on
+#   2026-08-19, after every published tag), so the modules are overlaid.
+#   The minoverlay modules already bake in the quantized-head selector fix
+#   (should_apply_lm_head_quant_method) so DFLASH runs the NVFP4 lm_head in
+#   place — no dense dequant.
+#
+# Draft: z-lab/Qwen3.8-27B-DFlash2 @ 50307d4 (block-diffusion drafter, 3.6 GB).
+#
+# Model: RadixArk/Qwen3.8-27B-NVFP4 (modelopt)
+#
+# PREREQUISITES (on the node):
+#   hfrs download RadixArk/Qwen3.8-27B-NVFP4
+#   hfrs download z-lab/Qwen3.8-27B-DFlash2
+#   docker build the minoverlay image (or use the recipe's docker_image as-is)
+#
+# REFERENCE:
+#   https://github.com/MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark  (start-dflash.sh)
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-on-dual-sparks/380350
+
+engine = "sglang"
+name = "Qwen3.8-27B (RadixArk NVFP4, SGLang DFlash2)"
+docker_image = "lmsysorg/sglang:qwen38-27b-dflash2-minoverlay"
+model = "RadixArk/Qwen3.8-27B-NVFP4"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["python3", "-m", "sglang.launch_server"]
+
+args = [
+    # Memory — NVFP4 needs 0.90 (0.95 hard-rebooted the box at graph capture)
+    "--mem-fraction-static", "0.90",
+    "--max-running-requests", "10",
+    "--max-mamba-cache-size", "30",
+    # Attention / KV
+    "--attention-backend", "flashinfer",
+    "--kv-cache-dtype", "fp8_e4m3",
+    # Mamba GDN hybrid — DFLASH requires extra_buffer (not extra_buffer_lazy)
+    "--mamba-ssm-dtype", "bfloat16",
+    "--mamba-full-memory-ratio", "4.21",
+    "--mamba-radix-cache-strategy", "extra_buffer",
+    # Chunked prefill
+    "--chunked-prefill-size", "8192",
+    # DFlash2 block-diffusion speculative decoding
+    "--speculative-algorithm", "DFLASH",
+    "--speculative-draft-model-path", "z-lab/Qwen3.8-27B-DFlash2",
+    "--speculative-draft-model-revision", "50307d4c4cde6860d4eee73e2547cd786fe8e8a4",
+    "--speculative-num-draft-tokens", "8",
+    # Tokenizer / reasoning / tool calling (Qwen3 family)
+    "--reasoning-parser", "qwen3",
+    "--tool-call-parser", "qwen3_coder",
+    # Observability
+    "--enable-metrics",
+    "--enable-cache-report",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK = "1"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"

recipes/qwen38-27b-nvfp4-sglang-dspark/recipe.toml

@@ -0,0 +1,89 @@
+# Qwen3.8-27B (RadixArk NVFP4 + DSpark on SGLang) — 1× DGX Spark / GB10 (sm_121),
+# TP=1.
+#
+# SGLang + DSpark block-diffusion speculative decoding. The community (MiaAI)
+# measured DSpark on this box at ~51.5 tok/s on code vs vLLM+MTP's 22.8 — the
+# drafter is 3× cheaper per guess and edits/rewrites much faster. Uses the
+# pre-built cookbook image (no source build): lmsysorg/sglang:qwen38-27b.
+#
+# TARGET MUST BE RadixArk MODELOPT NVFP4: the cookbook image accelerates
+# modelopt-quantized NVFP4. The unsloth/RedHatAI compressed-tensors builds
+# fall back to a slow unquantized path ("Falling back to UnquantizedLinear-
+# Method") — prefill crawls at ~0.5 tok/s. Use RadixArk here.
+#
+# Draft: RadixArk/Qwen3.8-27B-DSpark (bf16, block 7). SGLang loads it natively
+# from the HF cache — no arch patch needed (unlike the vLLM qwen3_dspark path).
+#
+# Image: lmsysorg/sglang:qwen38-27b  (multi-arch, incl. arm64; cookbook's
+#   DGX Spark Qwen3.8 cell, pinned image digest febfb971). Pre-built — no
+#   source build.
+#
+# Model: RadixArk/Qwen3.8-27B-NVFP4  (modelopt, cached locally on the node)
+#   https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4
+#
+# PREREQUISITES:
+#   Target + draft cached on the node (dgx1 ~/.cache/huggingface → NAS):
+#       hfrs download RadixArk/Qwen3.8-27B-NVFP4
+#       hfrs download RadixArk/Qwen3.8-27B-DSpark
+#
+# REFERENCE:
+#   https://github.com/MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark  (start-dspark.sh)
+#   https://blog.kubesimplify.com/qwen3-8-27b-on-dgx-spark
+
+engine = "sglang"
+name = "Qwen3.8-27B (RadixArk NVFP4, SGLang DSpark)"
+docker_image = "lmsysorg/sglang:qwen38-27b"
+model = "RadixArk/Qwen3.8-27B-NVFP4"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["python3", "-m", "sglang.launch_server"]
+
+args = [
+    # Memory — larger KV pool (mamba GDN state pool sized from concurrency)
+    "--mem-fraction-static", "0.90",
+    "--max-running-requests", "10",
+    "--max-mamba-cache-size", "30",
+    # Attention / KV
+    "--attention-backend", "flashinfer",
+    "--kv-cache-dtype", "fp8_e4m3",
+    # Mamba GDN hybrid
+    "--mamba-ssm-dtype", "bfloat16",
+    "--mamba-full-memory-ratio", "4.21",
+    "--mamba-radix-cache-strategy", "extra_buffer_lazy",
+    # Chunked prefill
+    "--chunked-prefill-size", "8192",
+    # DSpark speculative decoding (block 7 / 8 draft tokens, unquant bf16 draft)
+    "--speculative-algorithm", "DSPARK",
+    "--speculative-draft-model-path", "RadixArk/Qwen3.8-27B-DSpark",
+    "--speculative-dspark-block-size", "7",
+    "--speculative-draft-model-quantization", "unquant",
+    "--speculative-num-draft-tokens", "8",
+    # Compile + decode graphs
+    "--enable-torch-compile", "--torch-compile-max-bs", "4",
+    "--cuda-graph-max-bs-decode", "4",
+    "--num-continuous-decode-steps", "2",
+    # Tokenizer / reasoning / tool calling (Qwen3 family)
+    "--reasoning-parser", "qwen3",
+    "--tool-call-parser", "qwen3_coder",
+    # Observability
+    "--enable-metrics",
+    "--enable-cache-report",
+]
+
+[env]
+# Offline — target + draft are both cached on the node
+HF_HUB_OFFLINE = "1"
+
+# Pin to GB10's Cortex-X5 big cores (scheduler/tokenizer off the A725 efficiency
+# cores — +2-7% decode per the MiaAI build)
+# CPUSET handled at the container layer; this is the SGLang-side hint.
+SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK = "1"
+
+# Architecture
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"

recipes/qwen38-27b-nvfp4-unsloth-dspark/recipe.toml

@@ -0,0 +1,90 @@
+# Qwen3.8-27B (unsloth NVFP4 + DSpark) — 1× DGX Spark / GB10 (sm_121), TP=1.
+#
+# NVFP4 compressed-tensors quant (22 GB) with a dedicated DSpark draft model
+# (RadixArk, 2.6 GB, block_size 7) for speculative decoding. Dense 27B, GQA,
+# hybrid DeltaNet+Attention. Text + vision. Unsloth Dynamic V3.0 (preview)
+# quant — SOTA quality for NVFP4. vLLM-only (lm_head FP8).
+#
+# DSpark vs the in-checkpoint MTP head (qwen38-27b-nvfp4-unsloth recipe): the
+# draft is 3× cheaper per guess and edits/rewrites much faster, at the cost of
+# a 2.6 GB draft + a local arch-patched copy. Benchmark both.
+#
+# Image: ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2
+#   vLLM 0.27.1, torch 2.13+cuda130, FlashInfer built from source,
+#   instanttensor. Stock vLLM — supports qwen3_5 arch + method "dspark".
+#
+# Model: unsloth/Qwen3.8-27B-NVFP4
+#   https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4
+#
+# PREREQUISITES:
+#   Weights cached on every node (dgx1 ~/.cache/huggingface → NAS):
+#       huggingface-cli download unsloth/Qwen3.8-27B-NVFP4
+#       hfrs download RadixArk/Qwen3.8-27B-DSpark  (then cp -aL + patch arch → /home/kilork/dspark-draft)
+#
+# REFERENCE:
+#   https://blog.kubesimplify.com/qwen3-8-27b-on-dgx-spark  (DSpark section)
+
+engine = "vllm"
+name = "Qwen3.8-27B (NVFP4 unsloth + DSpark)"
+docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2"
+model = "unsloth/Qwen3.8-27B-NVFP4"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["vllm", "serve"]
+
+args = [
+    # KV cache & memory
+    "--kv-cache-dtype", "fp8",
+    "--max-model-len", "262144",
+    "--max-num-seqs", "6",
+    "--max-num-batched-tokens", "16384",
+    "--gpu-memory-utilization", "0.75",
+    "--served-model-name", "qwen3.8-27b",
+    # Engine backends (dense GQA — FlashInfer; NVFP4 route via compressed-tensors)
+    "--attention-backend", "flashinfer",
+    # Compilation / perf
+    "--optimization-level", "3",
+    "--performance-mode", "throughput",
+    # Chunked prefill + prefix caching
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--enable-flashinfer-autotune",
+    # DSpark speculative decoding — dedicated draft model (arch-patched local
+    # copy of RadixArk/Qwen3.8-27B-DSpark, re-labelled `Qwen3DSparkModel` so
+    # vLLM routes it to qwen3_dspark not the DeepSeek-V4 path). block_size 7.
+    # batched-tokens raised to 16384 to fit the draft token slots (warn:
+    # max_num_scheduled_tokens=8156 at 8192).
+    "--speculative-config", """
+'{"method":"dspark","model":"/dspark-draft","num_speculative_tokens":7}'""",
+    # Tokenizer / reasoning (Qwen3 family)
+    "--tool-call-parser", "qwen3_xml",
+    "--reasoning-parser", "qwen3",
+    "--enable-auto-tool-choice",
+    "--generation-config", "auto",
+    "--default-chat-template-kwargs.preserve_thinking=true",
+    # Proven generation defaults (thinking mode; Qwen3.8 model-card best
+    # practices)
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'""",
+    "--trust-remote-code",
+]
+
+# DSpark draft model — arch-patched local copy of RadixArk/Qwen3.8-27B-DSpark.
+[[extra_mounts]]
+host = "/home/kilork/dspark-draft"
+container = "/dspark-draft"
+
+
+[env]
+HF_HUB_OFFLINE = "1"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+CUDA_VISIBLE_DEVICES = "0"
+CUDA_DEVICE_ORDER = "PCI_BUS_ID"
+CUDA_CACHE_MAXSIZE = "4294967296"

recipes/qwen38-27b-nvfp4-unsloth/recipe.toml

@@ -0,0 +1,80 @@
+# Qwen3.8-27B (unsloth NVFP4 + MTP) — 1× DGX Spark / GB10 (sm_121), TP=1.
+#
+# NVFP4 compressed-tensors quant (22 GB) with a separate MTP draft in
+# `model_mtp.safetensors`. Dense 27B, GQA, hybrid DeltaNet+Attention. Text +
+# vision. Unsloth Dynamic V3.0 (preview) quant — SOTA quality for NVFP4.
+# vLLM-only (lm_head FP8).
+#
+# Nearly identical to RedHatAI's NVFP4 build (qwen38-27b-nvfp4 recipe);
+# benchmark both to pick the faster / higher-quality one.
+#
+# Image: ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2
+#   vLLM 0.27.1, torch 2.13+cuda130, FlashInfer built from source,
+#   instanttensor. Stock vLLM — supports the qwen3_5 arch + MTP.
+#
+# Model: unsloth/Qwen3.8-27B-NVFP4
+#   https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4
+#
+# PREREQUISITES:
+#   Weights cached on every node (dgx1 ~/.cache/huggingface → NAS):
+#       huggingface-cli download unsloth/Qwen3.8-27B-NVFP4
+#
+# REFERENCE:
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-on-dual-sparks/380350
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-nvfp4-on-a-single-dgx-spark-up-to-1m-context-vllm-mtp-measurements/380244
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-mixedint4-autoround-optimized-for-a-single-dgx-spark/380248
+
+engine = "vllm"
+name = "Qwen3.8-27B (NVFP4, unsloth)"
+docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2"
+model = "unsloth/Qwen3.8-27B-NVFP4"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["vllm", "serve"]
+
+args = [
+    # KV cache & memory
+    "--kv-cache-dtype", "fp8",
+    "--max-model-len", "262144",
+    "--max-num-seqs", "6",
+    "--max-num-batched-tokens", "8192",
+    "--gpu-memory-utilization", "0.75",
+    "--served-model-name", "qwen3.8-27b",
+    # Engine backends (dense GQA — FlashInfer; NVFP4 route via compressed-tensors)
+    "--attention-backend", "flashinfer",
+    # Compilation / perf
+    "--optimization-level", "3",
+    "--performance-mode", "throughput",
+    # Chunked prefill + prefix caching
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--enable-flashinfer-autotune",
+    # MTP speculative decoding (in-checkpoint draft, model_mtp.safetensors)
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":5}'""",
+    # Tokenizer / reasoning (Qwen3 family)
+    "--tool-call-parser", "qwen3_xml",
+    "--reasoning-parser", "qwen3",
+    "--enable-auto-tool-choice",
+    "--generation-config", "auto",
+    "--default-chat-template-kwargs.preserve_thinking=true",
+    # Proven generation defaults (thinking mode; Qwen3.8 model-card best
+    # practices)
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'""",
+    "--trust-remote-code",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+CUDA_VISIBLE_DEVICES = "0"
+CUDA_DEVICE_ORDER = "PCI_BUS_ID"
+CUDA_CACHE_MAXSIZE = "4294967296"

recipes/qwen38-27b-nvfp4/recipe.toml

@@ -0,0 +1,81 @@
+# Qwen3.8-27B (RedHatAI NVFP4 + MTP) — 1× DGX Spark / GB10 (sm_121), TP=1.
+#
+# NVFP4 compressed-tensors quant (22 GB) with a separate MTP draft in
+# `model_mtp.safetensors`. Dense 27B, GQA, hybrid DeltaNet+Attention. Text +
+# vision. The vLLM-only quant (lm_head FP8) — SGLang can't load it.
+#
+# Two nearly-identical NVFP4 builds are available:
+#   RedHatAI/Qwen3.8-27B-NVFP4  (this recipe)
+#   unsloth/Qwen3.8-27B-NVFP4   (qwen38-27b-nvfp4-unsloth recipe)
+# Speed/quality are close; benchmark both in the experiment.
+#
+# Image: ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2
+#   vLLM 0.27.1, torch 2.13+cuda130, FlashInfer built from source,
+#   instanttensor. Stock vLLM — supports the qwen3_5 arch + MTP.
+#
+# Model: RedHatAI/Qwen3.8-27B-NVFP4
+#   https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
+#
+# PREREQUISITES:
+#   Weights cached on every node (dgx1 ~/.cache/huggingface → NAS):
+#       huggingface-cli download RedHatAI/Qwen3.8-27B-NVFP4
+#
+# REFERENCE:
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-on-dual-sparks/380350
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-nvfp4-on-a-single-dgx-spark-up-to-1m-context-vllm-mtp-measurements/380244
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-mixedint4-autoround-optimized-for-a-single-dgx-spark/380248
+
+engine = "vllm"
+name = "Qwen3.8-27B (NVFP4, RedHatAI)"
+docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2"
+model = "RedHatAI/Qwen3.8-27B-NVFP4"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["vllm", "serve"]
+
+args = [
+    # KV cache & memory
+    "--kv-cache-dtype", "fp8",
+    "--max-model-len", "262144",
+    "--max-num-seqs", "6",
+    "--max-num-batched-tokens", "8192",
+    "--gpu-memory-utilization", "0.75",
+    "--served-model-name", "qwen3.8-27b",
+    # Engine backends (dense GQA — FlashInfer; NVFP4 route via compressed-tensors)
+    "--attention-backend", "flashinfer",
+    # Compilation / perf
+    "--optimization-level", "3",
+    "--performance-mode", "throughput",
+    # Chunked prefill + prefix caching
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--enable-flashinfer-autotune",
+    # MTP speculative decoding (in-checkpoint draft, model_mtp.safetensors)
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":5}'""",
+    # Tokenizer / reasoning (Qwen3 family)
+    "--tool-call-parser", "qwen3_xml",
+    "--reasoning-parser", "qwen3",
+    "--enable-auto-tool-choice",
+    "--generation-config", "auto",
+    "--default-chat-template-kwargs.preserve_thinking=true",
+    # Proven generation defaults (thinking mode; Qwen3.8 model-card best
+    # practices)
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'""",
+    "--trust-remote-code",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+CUDA_VISIBLE_DEVICES = "0"
+CUDA_DEVICE_ORDER = "PCI_BUS_ID"
+CUDA_CACHE_MAXSIZE = "4294967296"

recipes/qwen38-27b/recipe.toml

@@ -0,0 +1,88 @@
+# Qwen3.8-27B (BF16) — 1× DGX Spark / GB10 (sm_121), TP=1.
+#
+# Native BF16 dense 27B (52 GB). Quality reference baseline; the quantized
+# variants (FP8 / NVFP4 / Int4) are the speed candidates. Runs text + vision.
+#
+# Image: ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2
+#   vLLM 0.27.1, torch 2.13+cuda130, FlashInfer built from source,
+#   instanttensor. Stock vLLM — supports the qwen3_5 arch + MTP.
+#
+# Model: Qwen/Qwen3.8-27B
+#   https://huggingface.co/Qwen/Qwen3.8-27B
+#   Dense 27B, 64 layers, hidden 5120, GQA (4 KV heads × 256), Gated
+#   DeltaNet + Gated Attention hybrid. MTP-1. Native 262,144 ctx (YaRN to 1M).
+#   Multimodal (vision tower, `language_model_only: False`).
+#
+# PREREQUISITES:
+#   Weights cached on every node (dgx1 ~/.cache/huggingface → NAS):
+#       huggingface-cli download Qwen/Qwen3.8-27B
+#
+# REFERENCE:
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-mixedint4-autoround-optimized-for-a-single-dgx-spark/380248
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-on-dual-sparks/380350
+#   https://forums.developer.nvidia.com/t/qwen3-8-27b-at-34-38-tok-s-on-dgx-spark-open-source-one-command-setup-sglang-nvfp4-dspark/380257
+
+engine = "vllm"
+name = "Qwen3.8-27B (BF16)"
+docker_image = "ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2"
+model = "Qwen/Qwen3.8-27B"
+required_nodes = 1
+tensor_parallel_size = 1
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+entrypoint = ["vllm", "serve"]
+
+args = [
+    # KV cache & memory
+    "--kv-cache-dtype", "fp8",
+    "--max-model-len", "262144",
+    "--max-num-seqs", "6",
+    "--max-num-batched-tokens", "8192",
+    "--gpu-memory-utilization", "0.75",
+    "--served-model-name", "qwen3.8-27b",
+    # Engine backends (dense GQA — FlashInfer, not the DeepSeek MLA path)
+    "--attention-backend", "flashinfer",
+    # Compilation / perf
+    "--optimization-level", "3",
+    "--performance-mode", "throughput",
+    # Chunked prefill + prefix caching
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--enable-flashinfer-autotune",
+    # MTP speculative decoding (in-checkpoint draft, mtp_num_hidden_layers=1
+    # cycled across steps)
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":5}'""",
+    # Tokenizer / reasoning (Qwen3 family)
+    "--tool-call-parser", "qwen3_xml",
+    "--reasoning-parser", "qwen3",
+    "--enable-auto-tool-choice",
+    "--generation-config", "auto",
+    "--default-chat-template-kwargs.preserve_thinking=true",
+    # Proven generation defaults (thinking mode; Qwen3.8 model-card best
+    # practices). repetition_penalty 1.0 = identity: accepted but never
+    # deviates from no-op during req handling (DSpark/MLA bug caveat — see
+    # findings doc for the DeepSeek path; irrelevant to dense Qwen).
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'""",
+    "--trust-remote-code",
+]
+
+[env]
+# Offline mode — weights must be cached on each node
+HF_HUB_OFFLINE = "1"
+
+# Architecture
+TORCH_CUDA_ARCH_LIST = "12.1a"
+CUTE_DSL_ARCH = "sm_121a"
+OMP_NUM_THREADS = "4"
+
+# GPU memory management
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+
+# GPU device isolation
+CUDA_VISIBLE_DEVICES = "0"
+CUDA_DEVICE_ORDER = "PCI_BUS_ID"
+CUDA_CACHE_MAXSIZE = "4294967296"

specs/tech-architecture/qwen38-recipe-state.md

@@ -0,0 +1,124 @@
+# Qwen3.8-27B recipes — measured state
+
+> Real, on-hardware performance of the `qwen38-*` recipe family on a single
+> DGX Spark (dgx1, GB10 sm_121, 128 GB unified memory), measured 2026-08-22.
+> Update this table as you re-benchmark newer images / configs. Each row is a
+> like-for-like measurement unless noted.
+>
+> **Benchmark:** `tool-eval-bench bench --perf-only --pp 2048 --tg 128 @d0`
+> (free-gen, single stream, concurrency 1) unless noted. SGLang rows used a
+> direct SSE stream timer (tool-eval-bench's bench/run cannot parse SGLang's
+> SSE response shape — see "Measurement caveats").
+>
+> **Clock:** all rows since the dgx1 thermal shutdown are measured under the
+> qwen38 `before_serve` clock lock (`nvidia-smi -lgc 0,2000` — 2000 MHz cap vs
+> 3003 MHz max). The earlier vLLM rows (22.8) were at full boost; the lock costs
+> ~5% (22.8 → 21.7).
+
+## Engine / spec-decode matrix (decode tok/s, prefill tok/s)
+
+| Config | Image | decode | prefill | Notes |
+|---|---|---|---|---|
+| **unsloth NVFP4 + MTP (k=5)** | vllm-gb10 `v0.27.1-gb10.2` | **21.7** | 2,177 | **WINNER** (21.7 @ lock; 22.8 @ boost) |
+| unsloth NVFP4 + DSpark (k=7) | vllm-gb10 `v0.27.1-gb10.2` | 22.8 | **2,563** | ties decode, better prefill; needs patched draft |
+| RedHatAI NVFP4 + MTP (k=5) | vllm-gb10 `v0.27.1-gb10.2` | 19.2 | 2,153 | slower than unsloth NVFP4 |
+| Int4 AutoRound + MTP (k=4) | vllm-gb10 `v0.27.1-gb10.2` | 19.6 | 921 | mixed-4bit, slower prefill |
+| FP8 + DSpark (k=7) | vllm-gb10 `v0.27.1-gb10.2` | 17.7 | 1,133 | uses patched draft |
+| FP8 + MTP (k=5) | vllm-gb10 `v0.27.1-gb10.2` | 14.0 | 701 | surprisingly slow |
+| BF16 + MTP (k=5) | vllm-gb10 `v0.27.1-gb10.2` | 10.1 | 1,194 | quality baseline |
+| SGLang RadixArk NVFP4 + DSpark | `lmsysorg/sglang:qwen38-27b` | 9.30 | — | stock image (pre-DSpark... pre-DFlash2 tag) |
+| SGLang RadixArk NVFP4 + DFlash2 | `...-dflash2-minoverlay` | 9.22 | — | DFlash2 fork (c14312a66) |
+
+### Agentic / edit-heavy quality (tool-eval-bench `run --short`)
+
+| Config | score | median turn |
+|---|---|---|
+| unsloth NVFP4 + MTP | **100/100** (28/28) | 4.7 s |
+| unsloth NVFP4 + DSpark | 100/100 (30/30) | 5.2 s |
+
+## Winner
+
+**`qwen38-27b-nvfp4-unsloth`** — vLLM+MTP on the unsloth NVFP4 checkpoint. Best
+decode (21.7 tok/s under the lock), 100/100 agentic quality, and the MTP head is
+in-checkpoint (no extra draft to download). This is what pi is pointed at via
+`~/.pi/agent/models.json` → provider `dgx1` → model `qwen3.8-27b`.
+
+## Key decisions & why
+
+- **Image:** `ghcr.io/timothystewart6/vllm-gb10:v0.27.1-gb10.2` (latest immutable,
+  vLLM 0.27.1, torch 2.13+cu130, FlashInfer from source). Full `qwen3_5` + MTP
+  support. Used over `eugr` images (unpinned nightly) per the repo rule.
+- **MTP over vLLM-DSpark for the winner:** both tie on decode (22.8), MTP is
+  simpler (no draft download/patch), and MTP has the slightly better agentic
+  turn-around (4.7s vs 5.2s). The vLLM DSpark recipes are kept as experimentals.
+- **SGLang (DSpark and DFlash2) is NOT faster here.** Despite the forum's
+  52-87 tok/s claims, both measured ~9 tok/s for fresh prompts — 2.3× slower
+  than vLLM+MTP under the same clock lock. The forum numbers are an `ndec.py`
+  cached-code-context artifact, not per-request generation throughput.
+- **`unsloth` NVFP4 beat `RedHatAI` NVFP4** (21.7 vs 19.2 decode). Both are
+  compressed-tensors NVFP4. `RadixArk` (modelopt) NVFP4 was only needed for
+  SGLang's quantized-head path (see below).
+- **`num_speculative_tokens`:** 5 for the MTP recipes (helge/forum peak; the
+  single MTP layer cycles), 4 for Int4 (SlopOps). DSpark uses block 7 = 8 draft
+  tokens (requires `max-num-batched-tokens` ≥ 16384 to avoid the
+  `max_num_scheduled_tokens` warning).
+
+## Measurement caveats
+
+- **SGLang rows** used a hand-rolled SSE stream timer (count `content` +
+  `reasoning_content` deltas, divide time by token count). `tool-eval-bench`
+  `bench`/`run` cannot parse SGLang's SSE stream (llama-benchy "no usable
+  throughput metrics"; `run` times out per-turn). vLLM rows used the standard
+  `bench --perf-only`.
+- The forum's high SGLang numbers use `bench/ndec.py` (2-call net-decode A/B on
+  a large cached code prefix) — measures *marginal* edit cost, not per-request
+  generation. Not comparable to the table above.
+- `thinking`/`reasoning_content` is counted as token generation; the winner's
+  21.7 tok/s is measured with thinking **on** (default) counting both content
+  and reasoning deltas.
+
+## Recipes in this family
+
+| Recipe | Target | spec-decode | Status |
+|---|---|---|---|
+| `qwen38-27b-nvfp4-unsloth` | unsloth NVFP4 | MTP (k=5) | **winner — serving on dgx1** |
+| `qwen38-27b-nvfp4` | RedHatAI NVFP4 | MTP (k=5) | measured |
+| `qwen38-27b-nvfp4-unsloth-dspark` | unsloth NVFP4 | DSpark (k=7) | experimental |
+| `qwen38-27b-fp8-dspark` | FP8 | DSpark (k=7) | experimental |
+| `qwen38-27b-fp8` | FP8 | MTP (k=5) | measured |
+| `qwen38-27b-int4` | Pilcothink Int4 | MTP (k=4) | measured |
+| `qwen38-27b` | Qwen BF16 | MTP (k=5) | measured baseline |
+| `qwen38-27b-nvfp4-sglang-dspark` | RadixArk NVFP4 | SGLang DSpark | experimental (slow) |
+| `qwen38-27b-nvfp4-sglang-dflash2` | RadixArk NVFP4 | SGLang DFlash2 | experimental (slow) |
+
+### vLLM DSpark draft note
+
+The vLLM `method:"dspark"` paths (`qwen38-27b-fp8-dspark`,
+`qwen38-27b-nvfp4-unsloth-dspark`) load a dedicated draft model that must live
+at a **local dir** with its `config.json` arch patched:
+
+- RadixArk `Qwen3.8-27B-DSpark` declares `architectures: ["DSparkDraftModel"]`,
+  which vLLM maps to the *DeepSeek-V4* DSpark code path (`hc_mult` attribute
+  error). It must be re-labelled `Qwen3DSparkModel` + `draft_vocab_size` set to
+  route to `qwen3_dspark`.
+- The patched copy lives on dgx1 at `/home/kilork/dspark-draft`, bind-mounted
+  into the container at `/dspark-draft` via the recipe's `[[extra_mounts]]`.
+- This is fragile; if you only run the winner, drop these two recipes.
+
+### SGLang image note
+
+`lmsysorg/sglang:qwen38-27b` (cookbook) predates DFlash2 (upstream merged
+`c14312a66` on 2026-08-19). DFlash2 is overlaid by hand:
+`lmsysorg/sglang:qwen38-27b-dflash2-minoverlay` (5 sha256-verified modules from
+the MiaAI repo, built on dgx1). That minoverlay already bakes in the
+quantized-head selector fix, so it serves the modelopt NVFP4 target. SGLang only
+accelerates *modelopt* NVFP4 — `unsloth`/`RedHatAI` (compressed-tensors) fall
+back to a slow unquantized path.
+
+## Re-benchmarking
+
+1. Serve a recipe on the `qwen38` single-node cluster (dgx1) — the `before_serve`
+   clock lock is applied automatically.
+2. For vLLM: `tool-eval-bench bench --base-url http://dgx1.lan:30000/v1 --model qwen3.8-27b --perf-only --pp 2048 --tg 128 --concurrency 1`.
+3. For SGLang: use a direct SSE stream timer (see `Measurement caveats`).
+4. Record the new numbers in the table above; note the image tag + date.