Explorar o código

glm53-kindling-8x: 48 GiB hot KV pin (measured 24-27 GiB/node headroom at 30G)

Alexander Korolev hai 1 semana
pai
achega
606fe37cb4
Modificáronse 1 ficheiros con 1 adicións e 1 borrados
  1. 1 1
      recipes/glm53-kindling-8x/recipe.toml

+ 1 - 1
recipes/glm53-kindling-8x/recipe.toml

@@ -56,7 +56,7 @@ args = [
     "--max-model-len", "524288",
     "--kv-cache-dtype", "fp8_e4m3",
     "--block-size", "1536",
-    "--kv-cache-memory", "27917287424",
+    "--kv-cache-memory", "51539607552",
     "--trust-remote-code",
     "--enable-prefix-caching",
     "--max-num-batched-tokens", "16384",