Bläddra i källkod

fix: reduce gpu-memory-utilization to 0.75 for deepseek-v4-flash

From author: 0.8+ over-commits the ~121 GiB unified pool → OOM on GB10
Alexander Korolev 3 månader sedan
förälder
incheckning
ae1285dfdf
1 ändrade filer med 1 tillägg och 1 borttagningar
  1. 1 1
      recipes/deepseek-v4-flash/recipe.toml

+ 1 - 1
recipes/deepseek-v4-flash/recipe.toml

@@ -23,7 +23,7 @@ args = [
     "--moe-backend", "flashinfer_cutlass",
     "--linear-backend", "b12x",
     "--enable-chunked-prefill",
-    "--gpu-memory-utilization", "0.85",
+    "--gpu-memory-utilization", "0.75",
     "--compilation-config", """
 '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'""",
     "--enable-prefix-caching",