|
|
@@ -17,6 +17,7 @@ All three share served name **`glm53`** — clients never changed between runs.
|
|
|
| `glm53-flash-nvfp4-lab` | local-inference-lab mixed NVFP4/MXFP8-MTP | **92** (eager) / 89 (graphs) | 162/176 | **28.4** | 40.5 | 5.6 s | **72** | ✅ k=3, accept len ~2.9–3.5, ~63–83 % |
|
|
|
| `glm53-flash-nvfp4-redhat` | RedHatAI W4A4 compressed-tensors | **90** | 159/176 | 13.9 | 13.8 | 2.2 s | 67 | ❌ (no MTP weights in checkpoint) |
|
|
|
| `glm53-flash-nvfp4` | LibertAIDAI NVFP4-A16 | **88** | 154/176 | 26.2 | 22.7 | 6.3 s | 69 | ✅ k=5 |
|
|
|
+| `glm53-flash-nvfp4-4p67` | 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 | **87** | 153/176 | 24.4 | 39.2 | 1.4 s | 67 | ✅ accept ~90% |
|
|
|
|
|
|
(lab scored /174 — TC-53 was excluded from its run as an infrastructure
|
|
|
failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s
|
|
|
@@ -125,3 +126,35 @@ retry serve (or the launcher refuses with "only 1 nodes").
|
|
|
the lab's MXFP8 MTP as an external `speculative-config.model` draft is
|
|
|
theoretically possible but untested in vLLM — don't chase before the
|
|
|
official drop.
|
|
|
+
|
|
|
+## 4p67: --enforce-eager vs CUDA graphs (2026-08-29, same seed/invocation)
|
|
|
+
|
|
|
+Same recipe both arms (MTP k=3 included), only `--enforce-eager` removed;
|
|
|
+graphs arm verified to capture (FULL 5/5 + PIECEWISE 11/11), KV identical
|
|
|
+(1,164,369 tok pinned).
|
|
|
+
|
|
|
+| metric | eager | graphs |
|
|
|
+|---|---|---|
|
|
|
+| quality | **87** (153/176) | **88** (154/176) |
|
|
|
+| deployability | 67 | 68 |
|
|
|
+| tg t/s c1 (d0/d4096/d8192) | 21.1 / 23.2 / **24.4** | **23.6** / 22.6 / 22.9 |
|
|
|
+| tg t/s c2 (d0/d4096) | **39.2** / 22.9 | 33.1 / **24.0** |
|
|
|
+| TTFT (d0 / d8192) | 1.45 s / 6.17 s | 1.41 s / 6.10 s |
|
|
|
+| MTP draft accept | 90.0–92.2 % | 89.0–97.0 % |
|
|
|
+
|
|
|
+Findings:
|
|
|
+
|
|
|
+* **No graphs+MTP quality regression on 4p67** (88 vs 87 = 1 point, run-to-run
|
|
|
+ noise; the failure sets differ only by 5 shuffled partials). The lab
|
|
|
+ 92→89 graphs regression (forum #44) does NOT reproduce on this quant —
|
|
|
+ so the bug is build/quant-specific, not universal.
|
|
|
+* **Graphs buy nothing on decode here**: every cell within ±5%, and the
|
|
|
+ headline c2 number is *worse* with graphs (39.2→33.1). MTP already eats
|
|
|
+ the launch-gap that graphs would recover.
|
|
|
+* Prefill rows are prefix-cache-polluted (recipe enables prefix caching and
|
|
|
+ the perf probe repeats prompts); cold prefill ~1.4–1.5 K t/s both arms.
|
|
|
+* **Decision: `--enforce-eager` stays in `glm53-flash-nvfp4-4p67`** — it is
|
|
|
+ measurably free here and stays immune to the lab graph bug class.
|
|
|
+
|
|
|
+Reports: `/tmp/runs/2026/08/2026-08-29T09-09-14…--(eager)`,
|
|
|
+`…10-27-35.818420Z_181bbd18.md` (graphs).
|