recipes/recipes/dsv41-flash-1017-4x-vllm032/NOTE.md

DeepSeek-V4.1-Flash-1017 on stock-era vLLM (eugr nightly 20261015)

Purpose: the vLLM side of the "how far has the stock stack come on OUR model" question (specs/tech-architecture/stock-stack-gaps-1017.md). Same checkpoint, same four Sparks (dgx5–8, TP4), same cluster as the production SGLang lane.

Baseline, no speculation (2026-10-23)

Boot: 476 GB checkpoint streamed from the NAS with the plain safetensors loader (~10 min), GPU KV cache size: 8,726,823 tokens at a 131072 window (fp8 KV, --gpu-memory-utilization 0.88, engram tables in disk mode).

measure	stock vLLM (vLLM main 8244d1c44)	production SGLang EP2 4x
decode c1	29.7 t/s (20.0 on the first, cold turn)	47.8–50.7 t/s
decode c2 (aggregate)	45.6 t/s	64.2–68.3 t/s
prefill (~4K prompt)	~750 t/s	2,800–3,127 t/s
KV pool	8.73M tokens @131072	—

So the engine alone, un-speculated and untuned, lands at ~60–70% of the production lane's decode and ~25% of its prefill. Production runs DSpark K5 speculation on a tuned SGLang build; the next boot adds DSpark to this lane to see how much of the gap is speculation and how much is the engine.

Boot notes

• --load-format instanttensor fails on this image (Installed InstantTensor does not expose the metadata layout needed for index-aware loading (missing: _determine_buffer_size)) — use the default loader.
• --engram-config '{"cpu_offload":false,"table_memory":"disk"}' is accepted and the engine starts with the tables streamed from the checkpoint.

With DSpark K5 (2026-10-23) — decode reaches production, prefill does not

--speculative-config '{"method":"dspark","num_speculative_tokens":5}' (MTP heads in-checkpoint, no drafter repo). KV pool drops 8.73M → 7.07M tokens.

measure	stock vLLM + DSpark k5	stock vLLM, no spec	production SGLang EP2 4x
decode c1	52.9 t/s (25.5 cold)	29.7	47.8–50.7
decode c2 (agg)	66.4 t/s	45.6	64.2–68.3
prefill (~4K)	605 t/s	~750	2,800–3,127
acceptance	4.79 of 5	—	—

So on the same checkpoint and the same four Sparks, upstream vLLM main plus DSpark matches and slightly beats the production lane's decode (52.9 vs 47.8–50.7, 66.4 vs 64.2–68.3) — while its prefill is 4–5× slower (605 vs 2,800–3,127 t/s), which is the whole remaining gap for agentic traffic where TTFT dominates.

Next levers for the prefill side, in order of cheapness: --max-num-batched-tokens (4096 now — SGLang chunks larger), the lab fork's --prefill-compute-share, and the b12x A4-prefill patch (todo #23) if B12X_W4A16_MXFP4_PREFILL_MIN_TOKENS exists in this image.

Engram locality (2026-10-23) — the prefill gap was I/O, not the engine

The Engram n-gram tables are read at ~1 MB per generated token and live in exactly two shards (model-00047/48-of-00048.safetensors, 101.5 GB each). Serving them off the shared NAS cost the lane its prefill:

setup	prefill (12.6K unique prompt)	shared-NFS reads during a run
Engram on the NAS	750 t/s	+258 MB per short completion
Engram staged locally	3,957 t/s	0 MB

Decode c1 also improved (52.9 → 59.7 t/s). Production's 2,800–3,100 t/s was mostly this, not a different engine stack.

The recipe now declares it, and serve does the rest:

[[staging]]
kind = "local-view"
repo = "deepseek-ai/DeepSeek-V4.1-Flash-1017"
dst = "/home/kilork/models/dsv41-1017-local"
local = ["model-00047-of-00048.safetensors", "model-00048-of-00048.safetensors"]
min_free = "10G"

serve measures every launch node first (need vs free, and refuses before starting anything if one is short), then builds the view: real copies for the two engram shards, symlinks into the shared cache for the other 46 files, and mounts it at /root/.cache/hf-view (the model path is rewritten to it). The view is verified by sampled content hashes, so the second serve is verification-only, and a tree already rsync'd by hand is adopted.

Notes for the next lane: 190 GB per node is the cost — check df -h /home first (dgx7 sits at ~49 GB free with the shards in place, so it is the binding node). First staging pass over the NAS took ~8 min per node at ~470 MB/s (fabric, node-to-node).