|
|
@@ -121,6 +121,13 @@ Measured 2026-10-21 against the lane as served (DFlash2 k=7, FLASHINFER, fp8 KV)
|
|
|
tool-call parsing is `--tool-call-parser beam` with
|
|
|
`--enable-auto-tool-choice`; `--reasoning-parser beam` splits the channels.
|
|
|
|
|
|
+### Real-workload shape (measured with prefix caching, the V1 default)
|
|
|
+
|
|
|
+Multi-turn on a 4.7K-token document: **TTFT 0.41 s / 0.51 s** (turn 1 / turn 2 on
|
|
|
+the same prefix — the prefill is served from the prefix cache; at 2.4K tok/s
|
|
|
+uncached it would cost ~2 s), 11.4 s for 580 output tokens, 6.7 s for 384. So a
|
|
|
+real agent turn on Beam costs ~0.4 s to first token plus ~30 ms/token.
|
|
|
+
|
|
|
### Where it stands versus the prod lane
|
|
|
|
|
|
| | Beam NVFP4 4x (spec) | 1017 SGLang 4x (prod) |
|
