feat(recipes): agent-serving profile + GMU/env fixes from benchmark findings

Applies findings from the SvangenStudios dgx-spark-agent-serving-benchmarks
v0.1.1 (measured on this exact stack: DS4-Flash-0731, DSpark k=5, 2x DGX
Spark, TP=2). The benchmark overturned its own prior conclusion: chunk size
IS a fairness lever for agent workloads.

- recipes/deepseek-v4-flash-eugr-agent/: new agent-serving variant applying
  the validated agent-2048 profile — max-num-batched-tokens 2048 (2.9x the
  decode share under concurrent prefill vs 8192), max-num-seqs 6,
  max-cudagraph-capture-size 36 (=6*(k+1)), gpu-memory-utilization 0.78,
  max-model-len 1048576. Reuses the built eugr image (no new build).
- recipes/deepseek-v4-flash-eugr + -src: gpu-memory-utilization 0.85/0.80
  -> 0.78 (benchmark: "0.80 boots but dies under traffic"; GMU is a KV
  lever, not a safety lever, when weights dominate memory).
- AGENTS.md: new "## Serving notes" section — repetition_penalty crashes
  the DSpark path (gateways must filter it), UMA #48140 drop_caches
  workaround, GMU-is-a-KV-lever, the three-engines model, 2048-vs-8192
  agent-vs-batch guidance.
- specs/tech-architecture/dgx-spark-agent-serving-findings.md: full
  capture of the benchmark findings + an operational finding measured on
  the sm121 cluster (NFS weight-loading saturates the 10G mgmt link at
  load time; 0% page-cache residency because the 406 GB model > 121 GB
  RAM — eviction is size arithmetic, not a tuning bug; NCCL is correctly
  on 200G IB, vLLM JIT cache is on local NVMe not NFS).

make check passes (fmt, clippy, 164 tests, line limits).

AGENTS.md

@@ -330,6 +330,51 @@ Allowed characters (glob + exclude): alphanumeric, `* ? . _ - /`.
 
 The composed dir is cleaned up automatically on `sm121 stop`.
 
+## Serving notes (DeepSeek-V4-Flash / DSpark on GB10)
+
+Operational findings from the SvangenStudios agent-serving benchmark (v0.1.1,
+measured on this exact stack: DS4-Flash-0731, DSpark k=5, 2× DGX Spark,
+TP=2). See `specs/tech-architecture/dgx-spark-agent-serving-findings.md`
+for full detail.
+
+**`repetition_penalty` crashes the DSpark path** — an illegal-memory-access
+that kills the serving container. Any gateway/proxy in front of an
+`sm121`-served DSpark model **must filter `repetition_penalty`** from
+requests before forwarding. `presence_penalty` and `frequency_penalty` are
+verified safe. This is a DSpark-stack bug, not model behavior; the benchmark
+explicitly did not repeat the destructive test.
+
+**UMA memory gate (vLLM #48140)** — on GB10 (unified memory), vLLM's startup
+KV-cache budget check effectively reads Linux `MemFree`, so ~20 GB of
+reclaimable page cache (created by *reading the weights*) is booked as
+unavailable. The effect: large `--max-model-len` profiles (128K–262K+) may
+**deterministically refuse to boot** even when the memory should fit, while
+32K–64K boots normally. Two candidate mitigations:
+- run `echo 3 > /proc/sys/vm/drop_caches` (or the built-in `drop_caches`
+  mod from eugr's repo) as a `before_serve` hook before the engine starts;
+- a local UMA-aware patch of `gpu_worker.py` to use `MemAvailable` instead
+  of `MemFree`.
+The `drop_caches` workaround remains unverified on this deployment — test
+before relying on it for a large-context recipe.
+
+**`--gpu-memory-utilization` is a KV lever, not a safety lever** on GB10:
+weights take ~77.7 GiB of a ~119.6 GiB node, so the GMU ceiling directly
+squeezes the KV-cache budget. The benchmark found **0.80 boots but dies
+under traffic**; **0.78** is the safe ceiling. Do not raise GMU to "be safe" —
+that *reduces* usable KV capacity. The deepseek-v4-flash-eugr / -src / -eugr-agent
+recipes all ship 0.78.
+
+**Three engines** — an LLM server is three nearly-independent resources:
+prefill (compute-bound, serializes; one 32K prefill saturates the box),
+decode (bandwidth-bound, scales to ~N=4 concurrency), and admission
+(scheduler-bound, **blocked during long prefill** — a short request
+submitted mid-prefill waits until prefill completes, ~221× normal TTFT).
+`--max-num-batched-tokens` governs how often decode gets a scheduling slot
+between prefill chunks; **2048 gives 2.9× the decode share of 8192** under a
+concurrent prefill, for 7.7% prefill throughput. Use the
+`deepseek-v4-flash-eugr-agent` recipe (2048) for interactive agent traffic;
+use `deepseek-v4-flash-eugr` (8192) for exclusive long-context batch jobs.
+
 ## Config fields
 
 ```toml

recipes/deepseek-v4-flash-eugr-agent/recipe.toml

@@ -0,0 +1,167 @@
+# DeepSeek-V4-Flash-0731 DSpark — AGENT-SERVING profile (2× GB10, TP=2).
+#
+# Fork of deepseek-v4-flash-eugr tuned for interactive agent traffic, applying
+# the validated "agent-2048" profile from the SvangenStudios DGX-Spark
+# agent-serving benchmarks (v0.1.1, measured 2026-08-02 on this exact stack:
+# DeepSeek-V4-Flash-0731, DSpark k=5, 2× DGX Spark, TP=2, RoCE):
+#   https://github.com/SvangenStudios/dgx-spark-agent-serving-benchmarks
+#   configs/agent-2048.env  /  results/RESULTS.md §4
+#
+# The benchmark's headline finding: --max-num-batched-tokens is a FAIRNESS
+# lever for agent workloads, not just a jitter lever. Against an ongoing
+# decode stream, a 262K-token prefill at 2048 delivers 2.9× the decode share
+# (5.0% vs 1.7%), 3.1× the decode tokens, 73% lower p95 output-chunk jitter,
+# and +67% KV-pool capacity — for 7.7% prefill throughput. The launcher's
+# whole purpose is agent serving (a model may be serving this very agent), so
+# 2048 is the more correct default here.
+#
+# Mental model (benchmark §3): an LLM server is three nearly-independent
+# engines — prefill (compute-bound, serializes), decode (bandwidth-bound,
+# scales to ~N=4), admission (scheduler-bound, BLOCKED during long prefill).
+# Chunk size governs how often decode gets a scheduling slot between prefill
+# chunks; smaller = more slots. It does NOT fix new-request admission (a
+# short request submitted during a large prefill waits until prefill
+# completes — 221× normal TTFT in both profiles).
+#
+# Knob changes vs deepseek-v4-flash-eugr (everything else identical — same
+# eugr image, same backends, same env):
+#   --max-num-batched-tokens  8192 -> 2048   (the fairness lever)
+#   --max-num-seqs            8    -> 6      (decode saturates ~N=4; 6 gives
+#                                             headroom without inflating
+#                                             worst-case latency)
+#   --max-cudagraph-capture-size 64 -> 36    (= max_num_seqs * (k+1) = 6*6;
+#                                             the k+1 multiple rule — the
+#                                             benchmark notes k=5 rejects
+#                                             MAX_NUM_SEQS=1 at boot)
+#   --gpu-memory-utilization  0.85 -> 0.78   (benchmark: "0.80 boots but dies
+#                                             under traffic"; GMU is a KV
+#                                             lever, not a safety lever, when
+#                                             weights dominate memory)
+#   --max-model-len           auto -> 1048576  (benchmark's 1M working capacity
+#                                             — verified 12/12 needles at
+#                                             32K/128K/512K/900K)
+#
+# Trade-off: this profile costs ~7.7% prefill throughput vs the 8192 batch
+# profile. Use deepseek-v4-flash-eugr (8192) for exclusive long-context batch
+# jobs (512K–1M) with no concurrent agent traffic. See the batch-8192 profile
+# in the benchmark repo for the opposite extreme.
+#
+# 3× Sparks (TP=3): bump tensor_parallel_size + required_nodes to 3 (the eugr
+# image supports it per forum comment #28). k=5 unchanged (model-native).
+
+engine = "vllm"
+name = "DeepSeek V4 Flash DSpark (eugr b12x, agent 2048)"
+docker_image = "eugr/spark-vllm-b12x:sm121"
+model = "deepseek-ai/DeepSeek-V4-Flash-0731"
+required_nodes = 2
+tensor_parallel_size = 2
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+docker_image_dir = "deepseek-v4-flash-eugr"
+
+entrypoint = ["vllm", "serve"]
+
+args = [
+    "--kv-cache-dtype", "fp8",
+    "--block-size", "256",
+    "--load-format", "instanttensor",
+    # 1M working capacity (benchmark §2: 12/12 needles at 32K–900K). A 900K
+    # prefill takes ~15 min and starves everything — batch capacity, not an
+    # interactive mode.
+    "--max-model-len", "1048576",
+    # Decode saturates around N≈4 (benchmark §3); 6 gives headroom without
+    # inflating worst-case latency. cudagraph capture = max_num_seqs*(k+1)=36.
+    "--max-num-seqs", "6",
+    # FAIRNESS LEVER (benchmark §4): 2048 gives 2.9× the decode share under a
+    # concurrent prefill vs 8192, +67% KV pool, 73% lower jitter, for 7.7%
+    # prefill throughput. Use 8192 only for exclusive long-context batch jobs.
+    "--max-num-batched-tokens", "2048",
+    # = max_num_seqs * (k+1) = 6 * 6 (the k+1 multiple rule).
+    "--max-cudagraph-capture-size", "36",
+    # Benchmark finding: "0.80 boots but dies under traffic" — 0.78 is the
+    # safe ceiling. GMU is a KV lever, not a safety lever, when weights
+    # dominate memory (~77.7 GiB of ~119.6 GiB node).
+    "--gpu-memory-utilization", "0.78",
+    "--served-model-name", "deepseek-v4-flash",
+    # Engine backends — b12x across attention / MoE / linear
+    "--attention-backend", "B12X_MLA_SPARSE",
+    "--moe-backend", "b12x",
+    "--linear-backend", "b12x",
+    # Prefix caching — agent frameworks build prompts append-only, so reuse is
+    # 95.7–98.4% per turn (benchmark §9: divergence at the last token of the
+    # previous prompt; only ~270–660 tokens re-prefilled per agent step).
+    "--enable-prefix-caching",
+    # Compilation
+    "--compilation-config", """
+'{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'""",
+    # DSpark speculative decoding — k=5 (DeepSeek-V4-Flash native
+    # dspark_block_size). draft_sample_method per eugr recipe; draft attention
+    # backend B12X_MLA_SPARSE. The baked mod makes the embedded draft load via
+    # lazy safetensors while the target stays on InstantTensor.
+    "--speculative-config", """
+'{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","attention_backend":"B12X_MLA_SPARSE"}'""",
+    # Tokenizer / reasoning — reasoning-config pins the parser + empty
+    # reasoning markers (deepseek_v4 tokenizer mode emits them itself).
+    "--tokenizer-mode", "deepseek_v4",
+    "--tool-call-parser", "deepseek_v4",
+    "--reasoning-parser", "deepseek_v4",
+    "--reasoning-config", """
+'{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}'""",
+    "--enable-auto-tool-choice",
+    "--generation-config", "vllm",
+    "--default-chat-template-kwargs.thinking=true",
+    "--default-chat-template-kwargs.reasoning_effort=high",
+    "--disable-custom-all-reduce",
+    "--trust-remote-code",
+]
+
+# Persistent vLLM compile cache (speeds up restarts). Mounted at vLLM's
+# default cache root for root (/root/.cache/vllm); the HF cache is already
+# mounted at /root/.cache/huggingface by the launcher. Shares the eugr
+# recipe's cache dir (same image/backends — compile artifacts are reusable).
+[[extra_mounts]]
+host = "~/.cache/vllm-ds4-eugr"
+container = "/root/.cache/vllm"
+
+[env]
+# --- b12x + compile (from eugr's recipe env, verbatim) ---
+CUTE_DSL_ARCH = "sm_121a"
+VLLM_USE_AOT_COMPILE = "1"
+VLLM_USE_BREAKABLE_CUDAGRAPH = "0"
+VLLM_USE_MEGA_AOT_ARTIFACT = "-1"
+VLLM_MEMORY_PROFILE_INCLUDE_ATTN = "1"
+VLLM_USE_FLASHINFER_SAMPLER = "1"
+VLLM_USE_B12X_WO_PROJECTION = "1"
+VLLM_USE_B12X_MHC = "1"
+VLLM_USE_B12X_FP8_GEMM = "1"
+VLLM_USE_B12X_MOE = "1"
+VLLM_USE_B12X_SPARSE_INDEXER = "1"
+VLLM_USE_V2_MODEL_RUNNER = "1"
+B12X_MLA_SM120_UNIFIED = "1"
+B12X_MOE_FORCE_A8 = "1"
+
+# --- arch / memory / GPU isolation (operational on the sm121 cluster) ---
+TORCH_CUDA_ARCH_LIST = "12.1a"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+OMP_NUM_THREADS = "4"
+CUDA_VISIBLE_DEVICES = "0"
+CUDA_DEVICE_ORDER = "PCI_BUS_ID"
+CUDA_CACHE_MAXSIZE = "4294967296"
+
+# --- offline ---
+HF_HUB_OFFLINE = "1"
+
+# --- NCCL overrides (GB10 single-NIC tuning; image-agnostic — mirrors
+#     recipes/deepseek-v4-flash. The launcher still injects IB/HCA/socket
+#     vars from the cluster config; these are recipe-level extras.) ---
+NCCL_P2P_LEVEL = "SYS"
+NCCL_CUMEM_ENABLE = "0"
+NCCL_IB_TIMEOUT = "24"
+NCCL_IB_QPS_PER_CONNECTION = "8"
+NCCL_BUFFERSIZE = "16777216"
+NCCL_SOCKET_FAMILY = "inet"
+
+# PCIe allreduce not needed on GB10 single-NIC
+VLLM_ENABLE_PCIE_ALLREDUCE = "0"
+VLLM_PCIE_ALLREDUCE_BACKEND = "cpp"

recipes/deepseek-v4-flash-eugr/recipe.toml

@@ -55,7 +55,13 @@ args = [
     "--max-num-seqs", "8",
     "--max-num-batched-tokens", "8192",
     "--max-cudagraph-capture-size", "64",
-    "--gpu-memory-utilization", "0.85",
+    # 0.78, not higher: the SvangenStudios agent-serving benchmark (v0.1.1) on
+    # this exact stack found "0.80 boots but dies under traffic". GMU is a KV
+    # lever, not a safety lever, when weights dominate memory (~77.7 GiB of
+    # ~119.6 GiB node). For interactive agent traffic prefer the
+    # deepseek-v4-flash-eugr-agent profile (2048 chunk). See
+    # specs/tech-architecture/dgx-spark-agent-serving-findings.md.
+    "--gpu-memory-utilization", "0.78",
     "--served-model-name", "deepseek-v4-flash",
     # Engine backends — b12x across attention / MoE / linear
     "--attention-backend", "B12X_MLA_SPARSE",

recipes/deepseek-v4-flash-src/recipe.toml

@@ -36,7 +36,11 @@ args = [
     "--max-num-batched-tokens", "8192",
     # fork rule: max_num_seqs * (k+1) = 4 * 6
     "--max-cudagraph-capture-size", "24",
-    "--gpu-memory-utilization", "0.80",
+    # 0.78, not 0.80: the SvangenStudios agent-serving benchmark (v0.1.1) on
+    # this exact stack found "0.80 boots but dies under traffic". GMU is a KV
+    # lever, not a safety lever, when weights dominate memory. See
+    # specs/tech-architecture/dgx-spark-agent-serving-findings.md.
+    "--gpu-memory-utilization", "0.78",
     "--served-model-name", "deepseek-v4-flash", "default",
     "--attention-backend", "B12X_MLA_SPARSE",
     "--moe-backend", "b12x",

specs/tech-architecture/dgx-spark-agent-serving-findings.md

@@ -0,0 +1,267 @@
+# DGX Spark agent-serving benchmark findings
+
+> Measured properties of the DeepSeek-V4-Flash-0731 / DSpark / 2× DGX Spark
+> stack that inform recipe design. Source: **SvangenStudios
+> `dgx-spark-agent-serving-benchmarks` v0.1.1** (measured 2026-08-01/02, this
+> exact stack — DS4-Flash-0731, DSpark k=5, TP=2, RoCE, GB10 sm_121, 128 GB
+> unified memory). Repo: <https://github.com/SvangenStudios/dgx-spark-agent-serving-benchmarks>.
+>
+> The benchmark is methodologically rigorous: pre-registered hypotheses, a
+> correction document (`results/CORRECTION-2026-08-02.md`) that overturned one
+> of its own published conclusions, 27 regression tests for the measurement
+> scripts, and a threats-to-validity doc. The headline chunk-size finding was
+> re-measured after the authors found three bugs in their own instrument.
+
+## Why this matters here
+
+The `sm121` launcher exists to serve **agents** (per `AGENTS.md`: "a model
+serving this very agent may be running on the cluster"). Agent traffic is
+multi-turn, interleaved prefill+decode, and latency-sensitive — exactly the
+workload this benchmark characterizes. Several findings directly contradict
+defaults we were shipping; they are now applied (see "Applied changes" below).
+
+The benchmark measured the `tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark`
+recipe (a sibling of our `deepseek-v4-flash-dspark` family) on vLLM
+`0.21.1rc1.dev339` + DSpark overlay — the same vLLM version as
+`aidendle94/sparkrun-vllm-ds4-gb10:production-3.8`.
+
+## The headline finding — chunk size is a fairness lever
+
+`--max-num-batched-tokens` is **both** a jitter lever and a fairness lever
+for agent workloads. The authors originally concluded it was jitter-only;
+re-measurement (correcting three instrument bugs) overturned that.
+
+**A/B: 262K-token prefill against an ongoing decode stream, 2048 vs 8192:**
+
+| Metric | 8192 | 2048 | Delta |
+|---|---|---|---|
+| Decode share during prefill | 1.7% | 5.0% | **2.9×** |
+| Decode tokens delivered | 106 | 332 | 3.1× |
+| Prefill throughput | 1,584 tok/s | 1,462 tok/s | −7.7% |
+| p95 output-chunk gap | 6.10 s | 1.64 s | −73% |
+| KV pool | 1,598,763 tok | 2,669,829 tok | +67% |
+| Max concurrency @1M | 1.59× | 2.55× | |
+
+**Mental model** (falsifiable, predicts scaling with 1/chunk_size):
+```
+decode_tokens ≈ (prefill_tokens / chunk_size) × accepted_tokens_per_decode_step
+```
+Smaller chunk → 4× more scheduling slots for decode between prefill chunks →
+~4× more decode tokens (measured 3.1× because per-step accepted-token yield is
+slightly higher at 8192). Scales as ~1/chunk_size until accepted-token yield
+saturates. Undisturbed decode rate is identical in both profiles (36.3–39.0
+tok/s) — chunk size only affects decode *during* a concurrent prefill.
+
+**What it does NOT fix:** new-request admission stays blocked during a large
+prefill (~145 s at 8192 / ~159 s at 2048 wait for a short request submitted
+mid-prefill, vs 1.66 s normal warm TTFT — 221×). Chunk size helps *ongoing*
+decode, KV capacity, and jitter; it does not restore admission. That needs
+scheduling/preemption changes.
+
+## The three-engines systems finding
+
+An LLM server is not one resource but three nearly-independent ones:
+
+| Engine | Bounded by | Scales with concurrency? |
+|---|---|---|
+| **Prefill** | compute | **No** — aggregate flat (pure serialization) |
+| **Decode** | memory bandwidth + batching | **Yes** — to ~N=4 saturation |
+| **Admission** | scheduling | **No** during long prefill |
+
+- A single 32K prefill saturates the box (prefill aggregate flat N=1→6).
+- Decode scales: Swedish prose +125% to N=6, code **126 tok/s aggregate at
+  N=4**. Speculation acceptance holds ~71% under concurrency — does not
+  degrade with parallel streams.
+- Admission: three short requests started 25 s into a 516K prefill all waited
+  ~367 s (released when prefill finished). Short prompts at N=4 → TTFT 0.4 s
+  — blocking is specifically tied to long prefill saturating compute.
+
+Implication: `--max-num-seqs` beyond ~4 inflates worst-case latency without
+throughput gain. The agent profile ships `max_num_seqs=6` (headroom, not
+saturation).
+
+## GMU is a KV lever, not a safety lever (on GB10)
+
+The authors corrected their own GMU mental model: *"GPU-mem-util 0.70 as a
+'safety measure' was destructive. Weights take 77.7 GiB of a ~119.6 GiB node;
+at 0.70 the KV budget fell to 2.39 GiB — insufficient for even 32K. When
+weights dominate memory, GMU is a KV lever, not a safety lever."*
+
+Operational ceiling: **0.80 boots but dies under traffic** — **0.78 is the
+safe ceiling**. Raising GMU to "be safe" *reduces* usable KV capacity.
+
+## UMA memory gate — vLLM #48140
+
+On GB10 (unified memory), vLLM's startup KV-cache budget check effectively
+reads Linux `MemFree`, so ~20 GB of reclaimable page cache (created by
+*reading the weights*) is booked as unavailable. The page cache has a dual
+role: the same pages *speed up* weight loading (8.75 s/shard warm vs 11.5
+cold) *and* lower the reported KV budget.
+
+Reproduced across three boots (4.22 / 5.18 / 5.34 GiB reported available KV
+regardless of process history / JIT state):
+- 32K–64K boots normally (keep the page cache).
+- 128K–262K+ **deterministically refuses** (262K requires 18.35 GiB).
+
+Mitigations (the `drop_caches` workaround remains **unverified** on this
+deployment):
+- `echo 3 > /proc/sys/vm/drop_caches` before start (or eugr's `drop_caches`
+  mod as a `before_serve` hook);
+- a local UMA-aware patch of `gpu_worker.py` to use `MemAvailable`.
+
+## `repetition_penalty` crashes the DSpark path
+
+A concrete gotcha for any gateway in front of an `sm121`-served DSpark model:
+`repetition_penalty` triggers an **illegal-memory-access crash** that kills
+the serving container. The recipe documents it; the benchmark did not repeat
+the destructive test. `presence_penalty` and `frequency_penalty` are verified
+safe. Gateways **must filter `repetition_penalty`**.
+
+## Prefix-cache locality (agent frameworks are already optimal)
+
+Real Hermes v0.19.0 agent runs (bug-fix task, 10 model calls, 20 tool
+definitions, ~15–18K growing prompt): per-turn prefix-cache reuse
+**95.7–98.4%**, first divergence at the *last token* of the previous prompt
+(append-only framework — no mutating timestamps, no reordered tool lists).
+Each agent step re-prefills only ~270–660 tokens (≈0.2–0.4 s) vs ~10 s for
+the full context. `--enable-prefix-caching` is doing exactly the right thing.
+
+Contrast (synthetic): moving a mutating field from bottom→top of the prompt
+increased per-turn cost **39.6× on DS4** (13.7× on Laguna S 2.1) — 0% vs ~98%
+reusable prefix. A divergence in the first cache block → 0% reuse on both
+stacks. Lesson: keep dynamic prompt fields **last**.
+
+## Cold vs warm start (Laguna S 2.1, measured)
+
+- FP4 JIT, cold kernel cache: ~24 min (one-time; persistent in `~/.cache/vllm`).
+- Weight loading: ~12–13 min regardless of start state (single-threaded, 101%
+  CPU — the floor for every restart; first ~10 shards slower via UVM ramp).
+- Profiling/capture/autotune: ~2–4 min.
+- **Cold total ~39 min; warm to API ~13 min 16 s; first generation +5.1 s.**
+
+Readiness should be measured to the **first successful generation**, not the
+open port.
+
+## NFS weight loading saturates the 10G management link (load-time only)
+
+Operational finding measured on the `sm121` cluster itself (4nodes-switch,
+GLM-5.2-MXFP4-Experts-GPTQ serving, 2026-08-03). The nodes have two data
+planes: a 200G RoCE/IB fabric (`mlx5_core`, `enp1s0f0np0` @ 192.168.200.x)
+and a 10G copper management link (`r8127`, `enP7s7` @ 192.168.88.x). During
+`sm121 serve`, the 10G interface is heavily active — ~1.3 TB `rx_bytes`
+over one serving session.
+
+**This is not an NCCL misconfiguration.** NCCL is correctly pinned to the
+200G IB fabric (`NCCL_NET=IB`, `NCCL_IB_HCA=rocep1s0f0,...`,
+`NCCL_SOCKET_IFNAME=enp1s0f0np0`); the running container holds **zero** TCP
+sockets on the 10G interface, and tensor-parallel traffic goes over RDMA/IB,
+not TCP. Confirmed via `docker exec ... ss -tn` and interface counters.
+
+**The 10G traffic is NFS — the HuggingFace cache mount.** `nas.lan:/sna/ai/huggingface`
+is bind-mounted into the container at `/root/.cache/huggingface`; every
+model-weight and tokenizer/config read streams over the 10G copper to the NAS
+(`192.168.88.16:2049`, port 2049 = NFS, `nconnect=8`, 512K rsize/wsize).
+
+**The vLLM JIT compile cache is NOT on NFS** — it lives on local NVMe
+(`/dev/nvme0n1p2` ext4, mounted at `/root/.cache/jit`), so the 10G link
+carries only weights + configs, not compiled kernels. The recipes also
+bind-mount `~/.cache/vllm-*` (the persistent compile artifact dir) to a
+host path under `~`, which is on local NVMe, not the NAS.
+
+**Weight pages are evicted by size arithmetic, not a tuning bug.** `mincore`
+on all 78 shards of GLM-5.2-MXFP4-Experts-GPTQ showed **0.0 GB of 406 GB
+(0.0%) in the page cache** during steady-state serving. The cause is pure
+size: the model is **406 GB**, the node has **121 GB RAM**, and `MemAvailable`
+was 4.1 GB under UVM pressure (`vm.swappiness=60`, `vfs_cache_pressure=100`,
+3.1 GB already swapped). The weights literally cannot fit in the page cache,
+so every `sm121 serve` re-reads the full ~406 GB from the NAS over 10G — at
+~1.1 GB/s practical NFS throughput, that is ~370 s of pure 10G saturation per
+node per serve (the ~1.3 TB `rx_bytes` = 3+ loads × 406 GB).
+
+This is the load-time cost the benchmark measured above (~12–13 min weight
+loading, single-threaded). It is a **per-serve tax, not a steady-state cost**
+— once weights are in GPU memory, inference runs over IB (NCCL) and the API
+port; steady-state 10G traffic is low (tokenizer/config reads only).
+
+**What helps (and what doesn't):**
+
+- **Do nothing for steady state.** The 10G saturation is load-time only.
+- **A `before_serve` warm-read hook does NOT help here** — the 406 GB model
+  does not fit in 121 GB RAM, so pre-reading would just thrash the page cache
+  and evict itself. (It *would* help for a model smaller than ~RAM, and is
+  the standard mitigation for the UMA #48140 gate on smaller checkpoints.)
+- **Context-parallel sharding** (the glm52 recipe uses `--decode-context-parallel-size 4`)
+  means each rank loads only ~406/4 ≈ 100 GB, which *does* nearly fit in RAM.
+  Pairing CP with a lazy/tensor-selective loader (`--load-format instanttensor`,
+  as the eugr recipe does, or the `instanttensor-hybrid-draft-loader` mod)
+  maximizes the page-cache reuse for the per-rank shard.
+- **Mounting the NAS over IB instead of 10G** is the only structural fix that
+  removes the contention entirely — a site-infrastructure change, not a recipe one.
+
+## Retrieval at 1M (context-depth verification)
+
+3 needles per depth (8%, 50%, 92%) with distractor codes: **12/12 recovered**
+at 32K / 128K / 512K / 900K. Prefill decays with depth (attention's O(n²)):
+1,810 → 1,758 → 1,308 → 1,017 tok/s. 1M is **working capacity for batch
+analysis, not interactive mode** — a 900K prefill takes ~15 min and starves
+everything.
+
+## Measurement discipline (learned the hard way)
+
+1. On this DSpark build one SSE chunk carries ~2.5 accepted tokens — **counting
+   chunks is not counting tokens**. Use `"stream": false`, or stream with
+   `"return_token_ids": true` and count `token_ids`. The authors documented
+   this rule, then broke it in one script anyway — a regression test now
+   enforces it.
+2. Warm up with **long** generations; three short calls are not "warm" and the
+   effect decays after ~30 min idle. For interference measurement, use one
+   stream as its own baseline (before vs during).
+3. Measure p50/p95 token intervals, not just the largest gap — partial
+   starvation looks "green" otherwise.
+4. Salt every prompt uniquely, or you measure the prefix cache.
+5. Do not judge model loading / memory from process RSS on GB10 — CUDA/UVM
+   allocations don't appear there. Read system `free` and the engine logs.
+6. Do not read an instantaneous ETA as a trend; compare the same phase between
+   runs (early-shard vs steady-state rate → false 2.6× conclusion).
+7. `pkill -f`/`pgrep -f` self-match across SSH — use `[b]racket` patterns or
+   PID files.
+
+## Applied changes in this repo
+
+These findings drove concrete recipe/doc changes:
+
+- **`recipes/deepseek-v4-flash-eugr-agent/`** — new recipe applying the
+  validated `agent-2048` profile (`configs/agent-2048.env`): `max-num-batched-tokens 2048`,
+  `max-num-seqs 6`, `max-cudagraph-capture-size 36` (=6×(5+1)),
+  `gpu-memory-utilization 0.78`, `max-model-len 1048576`. Reuses the built
+  `eugr/spark-vllm-b12x:sm121` image (no new build).
+- **`recipes/deepseek-v4-flash-eugr/`** and **`recipes/deepseek-v4-flash-src/`** —
+  `gpu-memory-utilization` lowered to 0.78 (from 0.85 / 0.80) with citing
+  comments.
+- **`AGENTS.md` → "Serving notes"** — `repetition_penalty` crash warning,
+  UMA #48140 `drop_caches` workaround, GMU-is-a-KV-lever, the three-engines
+  model, and the 2048-vs-8192 agent-vs-batch guidance.
+
+The throughput-oriented `deepseek-v4-flash` (3.75 image, `max-num-batched-tokens 16536`,
+`gpu-memory-utilization 0.84`) is left unchanged — it is the known-working
+baseline; the GMU/chunk findings apply to it too, but it should not be
+modified without a boot test on a free node.
+
+## Open questions (pending node availability)
+
+- Does the `drop_caches` `before_serve` hook actually let a 128K–262K
+  profile boot on GB10? (benchmark: unverified on their deployment)
+- 72-hour soak on the final 2048 agent profile (benchmark: pending).
+- The scheduling-deviation minimal reproduction (`repro/`) — should an
+  upstream vLLM issue be filed, and does it reproduce *without* the DSpark
+  patches?
+- 262K `drop_caches` causality test for #48140 (benchmark: pending).
+
+## Sources
+
+- Repo (v0.1.1 tag): <https://github.com/SvangenStudios/dgx-spark-agent-serving-benchmarks>
+- Results: `results/RESULTS.md`
+- Correction (overturned the chunk-size conclusion): `results/CORRECTION-2026-08-02.md`
+- Serve profiles: `configs/agent-2048.env`, `configs/batch-8192.env`
+- Forum post: <https://forums.developer.nvidia.com/t/378890>
+- UMA issue: <https://github.com/vllm-project/vllm/issues/48140> (closed, "not planned")