# DGX Spark agent-serving benchmark findings > Measured properties of the DeepSeek-V4-Flash-0731 / DSpark / 2× DGX Spark > stack that inform recipe design. Source: **SvangenStudios > `dgx-spark-agent-serving-benchmarks` v0.1.1** (measured 2026-08-01/02, this > exact stack — DS4-Flash-0731, DSpark k=5, TP=2, RoCE, GB10 sm_121, 128 GB > unified memory). Repo: . > > The benchmark is methodologically rigorous: pre-registered hypotheses, a > correction document (`results/CORRECTION-2026-08-02.md`) that overturned one > of its own published conclusions, 27 regression tests for the measurement > scripts, and a threats-to-validity doc. The headline chunk-size finding was > re-measured after the authors found three bugs in their own instrument. ## Why this matters here The `sm121` launcher exists to serve **agents** (per `AGENTS.md`: "a model serving this very agent may be running on the cluster"). Agent traffic is multi-turn, interleaved prefill+decode, and latency-sensitive — exactly the workload this benchmark characterizes. Several findings directly contradict defaults we were shipping; they are now applied (see "Applied changes" below). The benchmark measured the `tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark` recipe (a sibling of our `deepseek-v4-flash-dspark` family) on vLLM `0.21.1rc1.dev339` + DSpark overlay — the same vLLM version as `aidendle94/sparkrun-vllm-ds4-gb10:production-3.8`. ## The headline finding — chunk size is a fairness lever `--max-num-batched-tokens` is **both** a jitter lever and a fairness lever for agent workloads. The authors originally concluded it was jitter-only; re-measurement (correcting three instrument bugs) overturned that. **A/B: 262K-token prefill against an ongoing decode stream, 2048 vs 8192:** | Metric | 8192 | 2048 | Delta | |---|---|---|---| | Decode share during prefill | 1.7% | 5.0% | **2.9×** | | Decode tokens delivered | 106 | 332 | 3.1× | | Prefill throughput | 1,584 tok/s | 1,462 tok/s | −7.7% | | p95 output-chunk gap | 6.10 s | 1.64 s | −73% | | KV pool | 1,598,763 tok | 2,669,829 tok | +67% | | Max concurrency @1M | 1.59× | 2.55× | | **Mental model** (falsifiable, predicts scaling with 1/chunk_size): ``` decode_tokens ≈ (prefill_tokens / chunk_size) × accepted_tokens_per_decode_step ``` Smaller chunk → 4× more scheduling slots for decode between prefill chunks → ~4× more decode tokens (measured 3.1× because per-step accepted-token yield is slightly higher at 8192). Scales as ~1/chunk_size until accepted-token yield saturates. Undisturbed decode rate is identical in both profiles (36.3–39.0 tok/s) — chunk size only affects decode *during* a concurrent prefill. **What it does NOT fix:** new-request admission stays blocked during a large prefill (~145 s at 8192 / ~159 s at 2048 wait for a short request submitted mid-prefill, vs 1.66 s normal warm TTFT — 221×). Chunk size helps *ongoing* decode, KV capacity, and jitter; it does not restore admission. That needs scheduling/preemption changes. ## The three-engines systems finding An LLM server is not one resource but three nearly-independent ones: | Engine | Bounded by | Scales with concurrency? | |---|---|---| | **Prefill** | compute | **No** — aggregate flat (pure serialization) | | **Decode** | memory bandwidth + batching | **Yes** — to ~N=4 saturation | | **Admission** | scheduling | **No** during long prefill | - A single 32K prefill saturates the box (prefill aggregate flat N=1→6). - Decode scales: Swedish prose +125% to N=6, code **126 tok/s aggregate at N=4**. Speculation acceptance holds ~71% under concurrency — does not degrade with parallel streams. - Admission: three short requests started 25 s into a 516K prefill all waited ~367 s (released when prefill finished). Short prompts at N=4 → TTFT 0.4 s — blocking is specifically tied to long prefill saturating compute. Implication: `--max-num-seqs` beyond ~4 inflates worst-case latency without throughput gain. The agent profile ships `max_num_seqs=6` (headroom, not saturation). ## GMU is a KV lever, not a safety lever (on GB10) The authors corrected their own GMU mental model: *"GPU-mem-util 0.70 as a 'safety measure' was destructive. Weights take 77.7 GiB of a ~119.6 GiB node; at 0.70 the KV budget fell to 2.39 GiB — insufficient for even 32K. When weights dominate memory, GMU is a KV lever, not a safety lever."* Operational ceiling: **0.80 boots but dies under traffic** — **0.78 is the safe ceiling**. Raising GMU to "be safe" *reduces* usable KV capacity. ## UMA memory gate — vLLM #48140 On GB10 (unified memory), vLLM's startup KV-cache budget check effectively reads Linux `MemFree`, so ~20 GB of reclaimable page cache (created by *reading the weights*) is booked as unavailable. The page cache has a dual role: the same pages *speed up* weight loading (8.75 s/shard warm vs 11.5 cold) *and* lower the reported KV budget. Reproduced across three boots (4.22 / 5.18 / 5.34 GiB reported available KV regardless of process history / JIT state): - 32K–64K boots normally (keep the page cache). - 128K–262K+ **deterministically refuses** (262K requires 18.35 GiB). Mitigations (the `drop_caches` workaround remains **unverified** on this deployment): - `echo 3 > /proc/sys/vm/drop_caches` before start (or eugr's `drop_caches` mod as a `before_serve` hook); - a local UMA-aware patch of `gpu_worker.py` to use `MemAvailable`. ## `repetition_penalty` crashes the DSpark path A concrete gotcha for any gateway in front of an `sm121`-served DSpark model: `repetition_penalty` triggers an **illegal-memory-access crash** that kills the serving container. The recipe documents it; the benchmark did not repeat the destructive test. `presence_penalty` and `frequency_penalty` are verified safe. Gateways **must filter `repetition_penalty`**. ## Prefix-cache locality (agent frameworks are already optimal) Real Hermes v0.19.0 agent runs (bug-fix task, 10 model calls, 20 tool definitions, ~15–18K growing prompt): per-turn prefix-cache reuse **95.7–98.4%**, first divergence at the *last token* of the previous prompt (append-only framework — no mutating timestamps, no reordered tool lists). Each agent step re-prefills only ~270–660 tokens (≈0.2–0.4 s) vs ~10 s for the full context. `--enable-prefix-caching` is doing exactly the right thing. Contrast (synthetic): moving a mutating field from bottom→top of the prompt increased per-turn cost **39.6× on DS4** (13.7× on Laguna S 2.1) — 0% vs ~98% reusable prefix. A divergence in the first cache block → 0% reuse on both stacks. Lesson: keep dynamic prompt fields **last**. ## Cold vs warm start (Laguna S 2.1, measured) - FP4 JIT, cold kernel cache: ~24 min (one-time; persistent in `~/.cache/vllm`). - Weight loading: ~12–13 min regardless of start state (single-threaded, 101% CPU — the floor for every restart; first ~10 shards slower via UVM ramp). - Profiling/capture/autotune: ~2–4 min. - **Cold total ~39 min; warm to API ~13 min 16 s; first generation +5.1 s.** Readiness should be measured to the **first successful generation**, not the open port. ## NFS weight loading saturates the 10G management link (load-time only) Operational finding measured on the `sm121` cluster itself (4nodes-switch, GLM-5.2-MXFP4-Experts-GPTQ serving, 2026-08-03). The nodes have two data planes: a 200G RoCE/IB fabric (`mlx5_core`, `enp1s0f0np0` @ 192.168.200.x) and a 10G copper management link (`r8127`, `enP7s7` @ 192.168.88.x). During `sm121 serve`, the 10G interface is heavily active — ~1.3 TB `rx_bytes` over one serving session. **This is not an NCCL misconfiguration.** NCCL is correctly pinned to the 200G IB fabric (`NCCL_NET=IB`, `NCCL_IB_HCA=rocep1s0f0,...`, `NCCL_SOCKET_IFNAME=enp1s0f0np0`); the running container holds **zero** TCP sockets on the 10G interface, and tensor-parallel traffic goes over RDMA/IB, not TCP. Confirmed via `docker exec ... ss -tn` and interface counters. **The 10G traffic is NFS — the HuggingFace cache mount.** `nas.lan:/sna/ai/huggingface` is bind-mounted into the container at `/root/.cache/huggingface`; every model-weight and tokenizer/config read streams over the 10G copper to the NAS (`192.168.88.16:2049`, port 2049 = NFS, `nconnect=8`, 512K rsize/wsize). **The vLLM JIT compile cache is NOT on NFS** — it lives on local NVMe (`/dev/nvme0n1p2` ext4, mounted at `/root/.cache/jit`), so the 10G link carries only weights + configs, not compiled kernels. The recipes also bind-mount `~/.cache/vllm-*` (the persistent compile artifact dir) to a host path under `~`, which is on local NVMe, not the NAS. **Weight pages are evicted by size arithmetic, not a tuning bug.** `mincore` on all 78 shards of GLM-5.2-MXFP4-Experts-GPTQ showed **0.0 GB of 406 GB (0.0%) in the page cache** during steady-state serving. The cause is pure size: the model is **406 GB**, the node has **121 GB RAM**, and `MemAvailable` was 4.1 GB under UVM pressure (`vm.swappiness=60`, `vfs_cache_pressure=100`, 3.1 GB already swapped). The weights literally cannot fit in the page cache, so every `sm121 serve` re-reads the full ~406 GB from the NAS over 10G — at ~1.1 GB/s practical NFS throughput, that is ~370 s of pure 10G saturation per node per serve (the ~1.3 TB `rx_bytes` = 3+ loads × 406 GB). This is the load-time cost the benchmark measured above (~12–13 min weight loading, single-threaded). It is a **per-serve tax, not a steady-state cost** — once weights are in GPU memory, inference runs over IB (NCCL) and the API port; steady-state 10G traffic is low (tokenizer/config reads only). **What helps (and what doesn't):** - **Do nothing for steady state.** The 10G saturation is load-time only. - **A `before_serve` warm-read hook does NOT help here** — the 406 GB model does not fit in 121 GB RAM, so pre-reading would just thrash the page cache and evict itself. (It *would* help for a model smaller than ~RAM, and is the standard mitigation for the UMA #48140 gate on smaller checkpoints.) - **Context-parallel sharding** (the glm52 recipe uses `--decode-context-parallel-size 4`) means each rank loads only ~406/4 ≈ 100 GB, which *does* nearly fit in RAM. Pairing CP with a lazy/tensor-selective loader (`--load-format instanttensor`, as the eugr recipe does, or the `instanttensor-hybrid-draft-loader` mod) maximizes the page-cache reuse for the per-rank shard. - **Mounting the NAS over IB instead of 10G** is the only structural fix that removes the contention entirely — a site-infrastructure change, not a recipe one. ## Retrieval at 1M (context-depth verification) 3 needles per depth (8%, 50%, 92%) with distractor codes: **12/12 recovered** at 32K / 128K / 512K / 900K. Prefill decays with depth (attention's O(n²)): 1,810 → 1,758 → 1,308 → 1,017 tok/s. 1M is **working capacity for batch analysis, not interactive mode** — a 900K prefill takes ~15 min and starves everything. ## Measurement discipline (learned the hard way) 1. On this DSpark build one SSE chunk carries ~2.5 accepted tokens — **counting chunks is not counting tokens**. Use `"stream": false`, or stream with `"return_token_ids": true` and count `token_ids`. The authors documented this rule, then broke it in one script anyway — a regression test now enforces it. 2. Warm up with **long** generations; three short calls are not "warm" and the effect decays after ~30 min idle. For interference measurement, use one stream as its own baseline (before vs during). 3. Measure p50/p95 token intervals, not just the largest gap — partial starvation looks "green" otherwise. 4. Salt every prompt uniquely, or you measure the prefix cache. 5. Do not judge model loading / memory from process RSS on GB10 — CUDA/UVM allocations don't appear there. Read system `free` and the engine logs. 6. Do not read an instantaneous ETA as a trend; compare the same phase between runs (early-shard vs steady-state rate → false 2.6× conclusion). 7. `pkill -f`/`pgrep -f` self-match across SSH — use `[b]racket` patterns or PID files. ## Applied changes in this repo These findings drove concrete recipe/doc changes: - **`recipes/deepseek-v4-flash-eugr-agent/`** — new recipe applying the validated `agent-2048` profile (`configs/agent-2048.env`): `max-num-batched-tokens 2048`, `max-num-seqs 6`, `max-cudagraph-capture-size 36` (=6×(5+1)), `gpu-memory-utilization 0.78`, `max-model-len 1048576`. Reuses the built `eugr/spark-vllm-b12x:sm121` image (no new build). - **`recipes/deepseek-v4-flash-eugr/`** and **`recipes/deepseek-v4-flash-src/`** — `gpu-memory-utilization` lowered to 0.78 (from 0.85 / 0.80) with citing comments. - **`AGENTS.md` → "Serving notes"** — `repetition_penalty` crash warning, UMA #48140 `drop_caches` workaround, GMU-is-a-KV-lever, the three-engines model, and the 2048-vs-8192 agent-vs-batch guidance. The throughput-oriented `deepseek-v4-flash` (3.75 image, `max-num-batched-tokens 16536`, `gpu-memory-utilization 0.84`) is left unchanged — it is the known-working baseline; the GMU/chunk findings apply to it too, but it should not be modified without a boot test on a free node. ## Open questions (pending node availability) - Does the `drop_caches` `before_serve` hook actually let a 128K–262K profile boot on GB10? (benchmark: unverified on their deployment) - 72-hour soak on the final 2048 agent profile (benchmark: pending). - The scheduling-deviation minimal reproduction (`repro/`) — should an upstream vLLM issue be filed, and does it reproduce *without* the DSpark patches? - 262K `drop_caches` causality test for #48140 (benchmark: pending). ## Sources - Repo (v0.1.1 tag): - Results: `results/RESULTS.md` - Correction (overturned the chunk-size conclusion): `results/CORRECTION-2026-08-02.md` - Serve profiles: `configs/agent-2048.env`, `configs/batch-8192.env` - Forum post: - UMA issue: (closed, "not planned")