Просмотр исходного кода

mxfp4 probe: KV pin 4->8 GiB - engine needs 6.58 GiB/rank for one 131072-token request (51.6 KB/token/rank measured)

Alexander Korolev 1 месяц назад
Родитель
Сommit
9cf76da709
1 измененных файлов с 5 добавлено и 3 удалено
  1. 5 3
      recipes/glm53-full-mxfp4-4x-probe/recipe.toml

+ 5 - 3
recipes/glm53-full-mxfp4-4x-probe/recipe.toml

@@ -171,9 +171,11 @@ args = [
     # 0.88 (107.0) is the Flash lane's value and is too tight here — the full
     # 0.88 (107.0) is the Flash lane's value and is too tight here — the full
     # model's weights leave ~24 GiB before the engine's own ~12 GiB.
     # model's weights leave ~24 GiB before the engine's own ~12 GiB.
     "--gpu-memory-utilization", "0.85",
     "--gpu-memory-utilization", "0.85",
-    # 4 GiB/rank probe pin (16 GiB aggregate) — deliberately the lightest pool
-    # that still serves long contexts; the DCP4/512K sibling pins 8 GiB.
-    "--kv-cache-memory-bytes", "4294967296",
+    # 8 GiB/rank pin (measured 2026-09-13: the engine needs 6.58 GiB/rank for
+    # ONE 131072-token request on this model - 51.6 KB/token/rank, fp8_ds_mla
+    # KV - so the earlier 4 GiB probe pin could not hold a single full-length
+    # request; 8 GiB leaves headroom and matches the DCP4/512K sibling).
+    "--kv-cache-memory-bytes", "8589934592",
     # PROBE: no DCP (the sibling has DCP4). Bare TP4: each rank holds the full
     # PROBE: no DCP (the sibling has DCP4). Bare TP4: each rank holds the full
     # KV sequence, 4 GiB pinned per rank is plenty for a 131K probe.
     # KV sequence, 4 GiB pinned per rank is plenty for a 131K probe.
     # No --kv-cache-dtype: let the r35 fork pick its native default for
     # No --kv-cache-dtype: let the r35 fork pick its native default for