Forráskód Böngészése

4p67: eager vs CUDA-graphs A/B measured (tool-eval-bench, seed 42)

Same recipe both arms (incl. MTP k=3); graphs arm capture-verified.
Quality 87 (eager) vs 88 (graphs) = noise — the lab graphs+MTP regression
does NOT reproduce on this quant. Decode within ±5% everywhere, c2
actually worse with graphs (39.2 -> 33.1 t/s). enforce-eager stays:
measurably free, faster boots, immune to the bug class. Full table in
specs/tech-architecture/glm53-recipe-state.md; 4p67 added to the quant
comparison table.
kilork 1 hónapja
szülő
commit
caae1b0ead

+ 5 - 0
recipes/glm53-flash-nvfp4-4p67/recipe.toml

@@ -83,6 +83,11 @@ args = [
 '{"method":"mtp","num_speculative_tokens":3}'""",
     "--language-model-only",
     # MTP + CUDA graphs degrades OUTPUT QUALITY on the glm53 day-0 dev build
+    # (lab quant: graphs+MTP 89 vs eager 92). On the 4p67 quant the bug does
+    # NOT reproduce (measured 2026-08-29: graphs 88 vs eager 87, tg within
+    # ±5%) — but graphs buy nothing either, so eager stays: measurably free,
+    # boots ~2 min faster, immune to the bug class. See
+    # specs/tech-architecture/glm53-recipe-state.md for the A/B table.
     # (forum #381350/#44, CONFIRMED by A/B bench 2026-08-28: same seed/flags,
     # graphs on = 89/100, eager = 92/100). Same build → same guard here;
     # re-A/B only if the base image is ever refreshed past the fix.

+ 33 - 0
specs/tech-architecture/glm53-recipe-state.md

@@ -17,6 +17,7 @@ All three share served name **`glm53`** — clients never changed between runs.
 | `glm53-flash-nvfp4-lab` | local-inference-lab mixed NVFP4/MXFP8-MTP | **92** (eager) / 89 (graphs) | 162/176 | **28.4** | 40.5 | 5.6 s | **72** | ✅ k=3, accept len ~2.9–3.5, ~63–83 % |
 | `glm53-flash-nvfp4-redhat` | RedHatAI W4A4 compressed-tensors | **90** | 159/176 | 13.9 | 13.8 | 2.2 s | 67 | ❌ (no MTP weights in checkpoint) |
 | `glm53-flash-nvfp4` | LibertAIDAI NVFP4-A16 | **88** | 154/176 | 26.2 | 22.7 | 6.3 s | 69 | ✅ k=5 |
+| `glm53-flash-nvfp4-4p67` | 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 | **87** | 153/176 | 24.4 | 39.2 | 1.4 s | 67 | ✅ accept ~90% |
 
 (lab scored /174 — TC-53 was excluded from its run as an infrastructure
 failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s
@@ -125,3 +126,35 @@ retry serve (or the launcher refuses with "only 1 nodes").
   the lab's MXFP8 MTP as an external `speculative-config.model` draft is
   theoretically possible but untested in vLLM — don't chase before the
   official drop.
+
+## 4p67: --enforce-eager vs CUDA graphs (2026-08-29, same seed/invocation)
+
+Same recipe both arms (MTP k=3 included), only `--enforce-eager` removed;
+graphs arm verified to capture (FULL 5/5 + PIECEWISE 11/11), KV identical
+(1,164,369 tok pinned).
+
+| metric | eager | graphs |
+|---|---|---|
+| quality | **87** (153/176) | **88** (154/176) |
+| deployability | 67 | 68 |
+| tg t/s c1 (d0/d4096/d8192) | 21.1 / 23.2 / **24.4** | **23.6** / 22.6 / 22.9 |
+| tg t/s c2 (d0/d4096) | **39.2** / 22.9 | 33.1 / **24.0** |
+| TTFT (d0 / d8192) | 1.45 s / 6.17 s | 1.41 s / 6.10 s |
+| MTP draft accept | 90.0–92.2 % | 89.0–97.0 % |
+
+Findings:
+
+* **No graphs+MTP quality regression on 4p67** (88 vs 87 = 1 point, run-to-run
+  noise; the failure sets differ only by 5 shuffled partials). The lab
+  92→89 graphs regression (forum #44) does NOT reproduce on this quant —
+  so the bug is build/quant-specific, not universal.
+* **Graphs buy nothing on decode here**: every cell within ±5%, and the
+  headline c2 number is *worse* with graphs (39.2→33.1). MTP already eats
+  the launch-gap that graphs would recover.
+* Prefill rows are prefix-cache-polluted (recipe enables prefix caching and
+  the perf probe repeats prompts); cold prefill ~1.4–1.5 K t/s both arms.
+* **Decision: `--enforce-eager` stays in `glm53-flash-nvfp4-4p67`** — it is
+  measurably free here and stays immune to the lab graph bug class.
+
+Reports: `/tmp/runs/2026/08/2026-08-29T09-09-14…--(eager)`,
+`…10-27-35.818420Z_181bbd18.md` (graphs).