Browse Source

beam lane: spec-k7 quality/perf reports + real-work operational contract

Quality with DFlash2 k=7 is unchanged (90/100, 175 pts, Hard Mode 48/56) while
responsiveness scores 56 (2.6 s median turn) vs 34 (4.7 s) without speculation.

Operational findings recorded for real use: thinking counts against max_tokens
(a 4.7K-token doc-QA request burns 600-900 reasoning tokens, so max_tokens 600
returns empty content with finish_reason=length — use >= 1500); reasoning_effort
is a prompt-side hint that does not reliably shorten thinking on hard tasks;
three verifiable long-context questions answered correctly in 15-17 s.
Alexander Korolev 2 days ago
parent
commit
a3c43d9bb1
1 changed files with 35 additions and 0 deletions
  1. 35 0
      recipes/beam-1-0-nvfp4-4x/NOTE.md

+ 35 - 0
recipes/beam-1-0-nvfp4-4x/NOTE.md

@@ -99,6 +99,41 @@ interactive use. `allow_missing_thinking: false` (vendor's
 serving recommendation) is deliberately NOT set: our harness replays assistant
 turns whose reasoning the agent stripped, and the template raises on those.
 
+## Using it for real work — operational contract
+
+Measured 2026-10-21 against the lane as served (DFlash2 k=7, FLASHINFER, fp8 KV):
+
+- **Thinking counts against `max_tokens`.** Document-QA requests over a ~4.7K
+  token spec burned 600-900 tokens of reasoning before emitting any `content`;
+  with `max_tokens: 600` the answer arrived empty (all budget spent thinking).
+  Use **`max_tokens >= 1500`** for anything non-trivial, or the client sees an
+  empty `content` with `finish_reason: length`.
+- With that, long-context QA is accurate and grounded: three verifiable
+  questions over a 4.7K-token document were answered in 14.7 / 15.3 / 17.5 s
+  (847-1065 completion tokens, ~200 tok/s effective including thinking), and all
+  three quoted the document's numbers exactly (KV pool `9.30M`, the
+  `SPARK_ENGRAM_MAX_IDS` rule with both chunk sizes).
+- `chat_template_kwargs.reasoning_effort` (8/32/64/128/256/512) is a *prompt-side
+  hint*, not a hard cap: on a hard triage task the model still spent 380-900
+  reasoning tokens at every setting, so lowering it does not reliably cut
+  latency — it does not hurt either. Leave the server default (max) for quality.
+- Reasoning arrives in the `reasoning` field (not `reasoning_content`), and
+  tool-call parsing is `--tool-call-parser beam` with
+  `--enable-auto-tool-choice`; `--reasoning-parser beam` splits the channels.
+
+### Where it stands versus the prod lane
+
+| | Beam NVFP4 4x (spec) | 1017 SGLang 4x (prod) |
+|---|---|---|
+| quality | **90/100**, Hard Mode 48/56, Safety 92% | 87/100, Hard Mode 42/56 |
+| median turn | 2.6 s | **1.4 s** |
+| decode c1 / c2 | 36.8 / **55.1** | 45.6 / 67.8 |
+| context | 262K (20x concurrency) | **1M** (9.3M-token pool) |
+
+Beam is the better *thinker* and a stronger decoder under load; 1017 is faster
+per turn and has the big context. A prod switch is an operator decision, not a
+technical blocker — the lane is ready to take traffic.
+
 ## Verdict
 
 Beam is the **quality leader** (90/100, the strongest Hard Mode result of the