|
|
@@ -105,12 +105,33 @@ contention can fail on rank desync, so retry before suspecting the config;
|
|
|
(b) the engram/weight pre-staging on node-local storage is what keeps a
|
|
|
*restart* cheap — the 26-95 min here is all NAS weight read.
|
|
|
|
|
|
+## Depth sweep (final config, llama-benchy pp2048/tg128, 2 runs per point)
|
|
|
+
|
|
|
+`ctx` rows = prefill the full depth then decode 128; `pp2048` rows = a 2048-token
|
|
|
+suffix at that depth (the agent-shaped case).
|
|
|
+
|
|
|
+| depth | ctx pp (c1/c2) | ctx tg (c1/c2) | pp2048 tg (c1/c2) |
|
|
|
+|---|---|---|---|
|
|
|
+| 16,384 | 3,530 / 2,279 | 39.0 / 32.7 | 46.5 / 62.5 |
|
|
|
+| 65,536 | 1,532 / 2,151 | 32.0 / **6.0** | 47.3 / 62.1 |
|
|
|
+| 131,072 | 1,545 / 1,561 | 36.1 / **2.7** | 42.1 / 61.2 |
|
|
|
+| 262,144 | 1,447 / 1,525 | 50.5 / **1.4** | 46.4 / 67.5 |
|
|
|
+
|
|
|
+Reading: single-stream decode after a 262K prefill is fine (50.5), and the
|
|
|
+agent-shaped case (short suffix at depth) holds 42-67 tok/s through the whole
|
|
|
+range. What collapses is **two concurrent deep conversations**: the c2 `ctx`
|
|
|
+rows fall to 6.0 / 2.7 / 1.4 tok/s because both streams' prefills serialize
|
|
|
+through the 2048-token chunk loop and decode gets the interleaved slots only.
|
|
|
+This is the documented prefill-vs-decode contention (see
|
|
|
+`dgx-spark-agent-serving-findings.md`), not a regression of this config. If
|
|
|
+long-context multi-session serving becomes the goal, the lever is
|
|
|
+`--chunked-prefill-size` (with `SPARK_ENGRAM_MAX_IDS` raised accordingly) or
|
|
|
+an admission cap, not more KV.
|
|
|
+
|
|
|
## Not yet measured
|
|
|
|
|
|
-- `SGLANG_DSV41_REASONING_EFFORT=75` (knapcio default; stu.miller measured 1017
|
|
|
- at `max` using ~40% of the old model's reasoning tokens). Changes token volume
|
|
|
- and therefore responsiveness far more than any flag above — needs its own
|
|
|
- quality + responsiveness A/B.
|
|
|
+- The depth sweep above is now measured (see the table); the concurrent
|
|
|
+ deep-context failure mode is documented rather than fixed.
|
|
|
- DSPARK `--speculative-dspark-block-size` sweep (3/5/7); accept length on the
|
|
|
live lane sits at 3.0-5.2 with block 5.
|
|
|
- `--depth` sweep (16K..256K) on the final config; the 1M context cap is adopted
|
