|
@@ -57,8 +57,14 @@ helped (karmic's native b12x MoE experts: 19.2 vs 19.0 t/s). Speculation does:
|
|
|
| config | decode c1 | decode c2 (aggregate) | acceptance len |
|
|
| config | decode c1 | decode c2 (aggregate) | acceptance len |
|
|
|
|---|---|---|---|
|
|
|---|---|---|---|
|
|
|
| no spec (first boot) | 19.0 t/s | — | — |
|
|
| no spec (first boot) | 19.0 t/s | — | — |
|
|
|
-| **MTP k=3** | **33.4–34.9 t/s** | **47.6 t/s** | 3.2–3.6 / 3 |
|
|
|
|
|
-| MTP k=5 | see below | | |
|
|
|
|
|
|
|
+| **MTP k=3** | **33.4–34.9 t/s** | **47.6 t/s** | 3.2–3.6 of 3 |
|
|
|
|
|
+| **MTP k=5** | **33.1–37.3 t/s** | **47.9 t/s** | 4.36–4.66 of 5 |
|
|
|
|
|
+
|
|
|
|
|
+k=5 buys nothing over k=3 in throughput (the drafts are accepted 87–93% of the
|
|
|
|
|
+time, so the ladder is not the limit — verification is). Both roughly **double**
|
|
|
|
|
+the un-speculated decode: 19.0 → ~35 t/s single request. The recipe keeps k=5
|
|
|
|
|
+because it is what was last measured; k=3 is the safer default if memory is
|
|
|
|
|
+tighter. |
|
|
|
|
|
|
|
|
`--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` — the MTP
|
|
`--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` — the MTP
|
|
|
head is in the checkpoint (`num_nextn_predict_layers=1`), so no draft model is
|
|
head is in the checkpoint (`num_nextn_predict_layers=1`), so no draft model is
|