Domů Procházet Nápověda
Přihlásit se
sm121
/
launcher
2
0
Rozštěpit 0
Soubory Úkoly 0 Pull Requesty 0 Wiki
Strom: ef7298e1ee
Větve Značky
launcher
/
specs
/
tech-architecture
/
dgx-spark-agent-serving-findings.md

dgx-spark-agent-serving-findings.md 14 KB
Historie Surový

DGX Spark agent-serving benchmark findings

Measured properties of the DeepSeek-V4-Flash-0731 / DSpark / 2× DGX Spark stack that inform recipe design. Source: SvangenStudios dgx-spark-agent-serving-benchmarks v0.1.1 (measured 2026-08-01/02, this exact stack — DS4-Flash-0731, DSpark k=5, TP=2, RoCE, GB10 sm_121, 128 GB unified memory). Repo: https://github.com/SvangenStudios/dgx-spark-agent-serving-benchmarks.

The benchmark is methodologically rigorous: pre-registered hypotheses, a correction document (results/CORRECTION-2026-08-02.md) that overturned one of its own published conclusions, 27 regression tests for the measurement scripts, and a threats-to-validity doc. The headline chunk-size finding was re-measured after the authors found three bugs in their own instrument.

Why this matters here

The sm121 launcher exists to serve agents (per AGENTS.md: "a model serving this very agent may be running on the cluster"). Agent traffic is multi-turn, interleaved prefill+decode, and latency-sensitive — exactly the workload this benchmark characterizes. Several findings directly contradict defaults we were shipping; they are now applied (see "Applied changes" below).

The benchmark measured the tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark recipe (a sibling of our deepseek-v4-flash-dspark family) on vLLM 0.21.1rc1.dev339 + DSpark overlay — the same vLLM version as aidendle94/sparkrun-vllm-ds4-gb10:production-3.8.

The headline finding — chunk size is a fairness lever

--max-num-batched-tokens is both a jitter lever and a fairness lever for agent workloads. The authors originally concluded it was jitter-only; re-measurement (correcting three instrument bugs) overturned that.

A/B: 262K-token prefill against an ongoing decode stream, 2048 vs 8192:

Metric 8192 2048 Delta
Decode share during prefill 1.7% 5.0% 2.9×
Decode tokens delivered 106 332 3.1×
Prefill throughput 1,584 tok/s 1,462 tok/s −7.7%
p95 output-chunk gap 6.10 s 1.64 s −73%
KV pool 1,598,763 tok 2,669,829 tok +67%
Max concurrency @1M 1.59× 2.55×

Mental model (falsifiable, predicts scaling with 1/chunk_size):

decode_tokens ≈ (prefill_tokens / chunk_size) × accepted_tokens_per_decode_step

Smaller chunk → 4× more scheduling slots for decode between prefill chunks → ~4× more decode tokens (measured 3.1× because per-step accepted-token yield is slightly higher at 8192). Scales as ~1/chunk_size until accepted-token yield saturates. Undisturbed decode rate is identical in both profiles (36.3–39.0 tok/s) — chunk size only affects decode during a concurrent prefill.

What it does NOT fix: new-request admission stays blocked during a large prefill (~145 s at 8192 / ~159 s at 2048 wait for a short request submitted mid-prefill, vs 1.66 s normal warm TTFT — 221×). Chunk size helps ongoing decode, KV capacity, and jitter; it does not restore admission. That needs scheduling/preemption changes.

The three-engines systems finding

An LLM server is not one resource but three nearly-independent ones:

Engine Bounded by Scales with concurrency?
Prefill compute No — aggregate flat (pure serialization)
Decode memory bandwidth + batching Yes — to ~N=4 saturation
Admission scheduling No during long prefill
  • A single 32K prefill saturates the box (prefill aggregate flat N=1→6).
  • Decode scales: Swedish prose +125% to N=6, code 126 tok/s aggregate at N=4. Speculation acceptance holds ~71% under concurrency — does not degrade with parallel streams.
  • Admission: three short requests started 25 s into a 516K prefill all waited ~367 s (released when prefill finished). Short prompts at N=4 → TTFT 0.4 s — blocking is specifically tied to long prefill saturating compute.

Implication: --max-num-seqs beyond ~4 inflates worst-case latency without throughput gain. The agent profile ships max_num_seqs=6 (headroom, not saturation).

GMU is a KV lever, not a safety lever (on GB10)

The authors corrected their own GMU mental model: "GPU-mem-util 0.70 as a 'safety measure' was destructive. Weights take 77.7 GiB of a ~119.6 GiB node; at 0.70 the KV budget fell to 2.39 GiB — insufficient for even 32K. When weights dominate memory, GMU is a KV lever, not a safety lever."

Operational ceiling: 0.80 boots but dies under traffic — 0.78 is the safe ceiling. Raising GMU to "be safe" reduces usable KV capacity.

UMA memory gate — vLLM #48140

On GB10 (unified memory), vLLM's startup KV-cache budget check effectively reads Linux MemFree, so ~20 GB of reclaimable page cache (created by reading the weights) is booked as unavailable. The page cache has a dual role: the same pages speed up weight loading (8.75 s/shard warm vs 11.5 cold) and lower the reported KV budget.

Reproduced across three boots (4.22 / 5.18 / 5.34 GiB reported available KV regardless of process history / JIT state):

  • 32K–64K boots normally (keep the page cache).
  • 128K–262K+ deterministically refuses (262K requires 18.35 GiB).

Mitigations (the drop_caches workaround remains unverified on this deployment):

  • echo 3 > /proc/sys/vm/drop_caches before start (or eugr's drop_caches mod as a before_serve hook);
  • a local UMA-aware patch of gpu_worker.py to use MemAvailable.

repetition_penalty crashes the DSpark path

A concrete gotcha for any gateway in front of an sm121-served DSpark model: repetition_penalty triggers an illegal-memory-access crash that kills the serving container. The recipe documents it; the benchmark did not repeat the destructive test. presence_penalty and frequency_penalty are verified safe. Gateways must filter repetition_penalty.

Prefix-cache locality (agent frameworks are already optimal)

Real Hermes v0.19.0 agent runs (bug-fix task, 10 model calls, 20 tool definitions, ~15–18K growing prompt): per-turn prefix-cache reuse 95.7–98.4%, first divergence at the last token of the previous prompt (append-only framework — no mutating timestamps, no reordered tool lists). Each agent step re-prefills only ~270–660 tokens (≈0.2–0.4 s) vs ~10 s for the full context. --enable-prefix-caching is doing exactly the right thing.

Contrast (synthetic): moving a mutating field from bottom→top of the prompt increased per-turn cost 39.6× on DS4 (13.7× on Laguna S 2.1) — 0% vs ~98% reusable prefix. A divergence in the first cache block → 0% reuse on both stacks. Lesson: keep dynamic prompt fields last.

Cold vs warm start (Laguna S 2.1, measured)

  • FP4 JIT, cold kernel cache: ~24 min (one-time; persistent in ~/.cache/vllm).
  • Weight loading: ~12–13 min regardless of start state (single-threaded, 101% CPU — the floor for every restart; first ~10 shards slower via UVM ramp).
  • Profiling/capture/autotune: ~2–4 min.
  • Cold total ~39 min; warm to API ~13 min 16 s; first generation +5.1 s.

Readiness should be measured to the first successful generation, not the open port.

NFS weight loading saturates the 10G management link (load-time only)

Operational finding measured on the sm121 cluster itself (4nodes-switch, GLM-5.2-MXFP4-Experts-GPTQ serving, 2026-08-03). The nodes have two data planes: a 200G RoCE/IB fabric (mlx5_core, enp1s0f0np0 @ 192.168.200.x) and a 10G copper management link (r8127, enP7s7 @ 192.168.88.x). During sm121 serve, the 10G interface is heavily active — ~1.3 TB rx_bytes over one serving session.

This is not an NCCL misconfiguration. NCCL is correctly pinned to the 200G IB fabric (NCCL_NET=IB, NCCL_IB_HCA=rocep1s0f0,..., NCCL_SOCKET_IFNAME=enp1s0f0np0); the running container holds zero TCP sockets on the 10G interface, and tensor-parallel traffic goes over RDMA/IB, not TCP. Confirmed via docker exec ... ss -tn and interface counters.

The 10G traffic is NFS — the HuggingFace cache mount. nas.lan:/sna/ai/huggingface is bind-mounted into the container at /root/.cache/huggingface; every model-weight and tokenizer/config read streams over the 10G copper to the NAS (192.168.88.16:2049, port 2049 = NFS, nconnect=8, 512K rsize/wsize).

The vLLM JIT compile cache is NOT on NFS — it lives on local NVMe (/dev/nvme0n1p2 ext4, mounted at /root/.cache/jit), so the 10G link carries only weights + configs, not compiled kernels. The recipes also bind-mount ~/.cache/vllm-* (the persistent compile artifact dir) to a host path under ~, which is on local NVMe, not the NAS.

Weight pages are evicted by size arithmetic, not a tuning bug. mincore on all 78 shards of GLM-5.2-MXFP4-Experts-GPTQ showed 0.0 GB of 406 GB (0.0%) in the page cache during steady-state serving. The cause is pure size: the model is 406 GB, the node has 121 GB RAM, and MemAvailable was 4.1 GB under UVM pressure (vm.swappiness=60, vfs_cache_pressure=100, 3.1 GB already swapped). The weights literally cannot fit in the page cache, so every sm121 serve re-reads the full ~406 GB from the NAS over 10G — at ~1.1 GB/s practical NFS throughput, that is ~370 s of pure 10G saturation per node per serve (the ~1.3 TB rx_bytes = 3+ loads × 406 GB).

This is the load-time cost the benchmark measured above (~12–13 min weight loading, single-threaded). It is a per-serve tax, not a steady-state cost — once weights are in GPU memory, inference runs over IB (NCCL) and the API port; steady-state 10G traffic is low (tokenizer/config reads only).

What helps (and what doesn't):

  • Do nothing for steady state. The 10G saturation is load-time only.
  • A before_serve warm-read hook does NOT help here — the 406 GB model does not fit in 121 GB RAM, so pre-reading would just thrash the page cache and evict itself. (It would help for a model smaller than ~RAM, and is the standard mitigation for the UMA #48140 gate on smaller checkpoints.)
  • Context-parallel sharding (the glm52 recipe uses --decode-context-parallel-size 4) means each rank loads only ~406/4 ≈ 100 GB, which does nearly fit in RAM. Pairing CP with a lazy/tensor-selective loader (--load-format instanttensor, as the eugr recipe does, or the instanttensor-hybrid-draft-loader mod) maximizes the page-cache reuse for the per-rank shard.
  • Mounting the NAS over IB instead of 10G is the only structural fix that removes the contention entirely — a site-infrastructure change, not a recipe one.

Retrieval at 1M (context-depth verification)

3 needles per depth (8%, 50%, 92%) with distractor codes: 12/12 recovered at 32K / 128K / 512K / 900K. Prefill decays with depth (attention's O(n²)): 1,810 → 1,758 → 1,308 → 1,017 tok/s. 1M is working capacity for batch analysis, not interactive mode — a 900K prefill takes ~15 min and starves everything.

Measurement discipline (learned the hard way)

  1. On this DSpark build one SSE chunk carries ~2.5 accepted tokens — counting chunks is not counting tokens. Use "stream": false, or stream with "return_token_ids": true and count token_ids. The authors documented this rule, then broke it in one script anyway — a regression test now enforces it.
  2. Warm up with long generations; three short calls are not "warm" and the effect decays after ~30 min idle. For interference measurement, use one stream as its own baseline (before vs during).
  3. Measure p50/p95 token intervals, not just the largest gap — partial starvation looks "green" otherwise.
  4. Salt every prompt uniquely, or you measure the prefix cache.
  5. Do not judge model loading / memory from process RSS on GB10 — CUDA/UVM allocations don't appear there. Read system free and the engine logs.
  6. Do not read an instantaneous ETA as a trend; compare the same phase between runs (early-shard vs steady-state rate → false 2.6× conclusion).
  7. pkill -f/pgrep -f self-match across SSH — use [b]racket patterns or PID files.

Applied changes in this repo

These findings drove concrete recipe/doc changes:

  • recipes/deepseek-v4-flash-eugr-agent/ — new recipe applying the validated agent-2048 profile (configs/agent-2048.env): max-num-batched-tokens 2048, max-num-seqs 6, max-cudagraph-capture-size 36 (=6×(5+1)), gpu-memory-utilization 0.78, max-model-len 1048576. Reuses the built eugr/spark-vllm-b12x:sm121 image (no new build).
  • recipes/deepseek-v4-flash-eugr/ and recipes/deepseek-v4-flash-src/ — gpu-memory-utilization lowered to 0.78 (from 0.85 / 0.80) with citing comments.
  • AGENTS.md → "Serving notes" — repetition_penalty crash warning, UMA #48140 drop_caches workaround, GMU-is-a-KV-lever, the three-engines model, and the 2048-vs-8192 agent-vs-batch guidance.

The throughput-oriented deepseek-v4-flash (3.75 image, max-num-batched-tokens 16536, gpu-memory-utilization 0.84) is left unchanged — it is the known-working baseline; the GMU/chunk findings apply to it too, but it should not be modified without a boot test on a free node.

Open questions (pending node availability)

  • Does the drop_caches before_serve hook actually let a 128K–262K profile boot on GB10? (benchmark: unverified on their deployment)
  • 72-hour soak on the final 2048 agent profile (benchmark: pending).
  • The scheduling-deviation minimal reproduction (repro/) — should an upstream vLLM issue be filed, and does it reproduce without the DSpark patches?
  • 262K drop_caches causality test for #48140 (benchmark: pending).

Sources