|
@@ -99,6 +99,41 @@ interactive use. `allow_missing_thinking: false` (vendor's
|
|
|
serving recommendation) is deliberately NOT set: our harness replays assistant
|
|
serving recommendation) is deliberately NOT set: our harness replays assistant
|
|
|
turns whose reasoning the agent stripped, and the template raises on those.
|
|
turns whose reasoning the agent stripped, and the template raises on those.
|
|
|
|
|
|
|
|
|
|
+## Using it for real work — operational contract
|
|
|
|
|
+
|
|
|
|
|
+Measured 2026-10-21 against the lane as served (DFlash2 k=7, FLASHINFER, fp8 KV):
|
|
|
|
|
+
|
|
|
|
|
+- **Thinking counts against `max_tokens`.** Document-QA requests over a ~4.7K
|
|
|
|
|
+ token spec burned 600-900 tokens of reasoning before emitting any `content`;
|
|
|
|
|
+ with `max_tokens: 600` the answer arrived empty (all budget spent thinking).
|
|
|
|
|
+ Use **`max_tokens >= 1500`** for anything non-trivial, or the client sees an
|
|
|
|
|
+ empty `content` with `finish_reason: length`.
|
|
|
|
|
+- With that, long-context QA is accurate and grounded: three verifiable
|
|
|
|
|
+ questions over a 4.7K-token document were answered in 14.7 / 15.3 / 17.5 s
|
|
|
|
|
+ (847-1065 completion tokens, ~200 tok/s effective including thinking), and all
|
|
|
|
|
+ three quoted the document's numbers exactly (KV pool `9.30M`, the
|
|
|
|
|
+ `SPARK_ENGRAM_MAX_IDS` rule with both chunk sizes).
|
|
|
|
|
+- `chat_template_kwargs.reasoning_effort` (8/32/64/128/256/512) is a *prompt-side
|
|
|
|
|
+ hint*, not a hard cap: on a hard triage task the model still spent 380-900
|
|
|
|
|
+ reasoning tokens at every setting, so lowering it does not reliably cut
|
|
|
|
|
+ latency — it does not hurt either. Leave the server default (max) for quality.
|
|
|
|
|
+- Reasoning arrives in the `reasoning` field (not `reasoning_content`), and
|
|
|
|
|
+ tool-call parsing is `--tool-call-parser beam` with
|
|
|
|
|
+ `--enable-auto-tool-choice`; `--reasoning-parser beam` splits the channels.
|
|
|
|
|
+
|
|
|
|
|
+### Where it stands versus the prod lane
|
|
|
|
|
+
|
|
|
|
|
+| | Beam NVFP4 4x (spec) | 1017 SGLang 4x (prod) |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| quality | **90/100**, Hard Mode 48/56, Safety 92% | 87/100, Hard Mode 42/56 |
|
|
|
|
|
+| median turn | 2.6 s | **1.4 s** |
|
|
|
|
|
+| decode c1 / c2 | 36.8 / **55.1** | 45.6 / 67.8 |
|
|
|
|
|
+| context | 262K (20x concurrency) | **1M** (9.3M-token pool) |
|
|
|
|
|
+
|
|
|
|
|
+Beam is the better *thinker* and a stronger decoder under load; 1017 is faster
|
|
|
|
|
+per turn and has the big context. A prod switch is an operator decision, not a
|
|
|
|
|
+technical blocker — the lane is ready to take traffic.
|
|
|
|
|
+
|
|
|
## Verdict
|
|
## Verdict
|
|
|
|
|
|
|
|
Beam is the **quality leader** (90/100, the strongest Hard Mode result of the
|
|
Beam is the **quality leader** (90/100, the strongest Hard Mode result of the
|