|
|
@@ -125,6 +125,25 @@ Report to the fork owner; keep `--load-format instanttensor` for GLM-5.3
|
|
|
NVFP4 meanwhile. Test recipe kept: `glm53-flash-r28-4x-dcp4-b12xload`
|
|
|
(re-test once the fork fix lands, then delete). (2) Discord env vars — TESTED: `--mamba-cache-mode align` + `VLLM_GLM53_SPLIT_TARGET_BLOCK_SIZE=2048` / `_MAMBA_BLOCK_SIZE=256` boots clean (split2048 recipe): KV 8,269,149 tokens (+2.6% vs 8,060,928 standard at the same 16 GiB pin — better page packing), bench 156/176 / 3.1 s — inside the standard-r28 noise band (155/159), so neutral for sequential serving; the win (256-token recurrent checkpoints → finer prefix reuse) only shows under high prefix-cache hit workloads. Keep OFF by default; it is the #646 page geometry the DFlash2 path expects.
|
|
|
|
|
|
+## GLM-5.3-NVFP4-Spark 386B boot test (operator request) — NOT 4x-capable
|
|
|
+
|
|
|
+`local-inference-lab/GLM-5.3-NVFP4-Spark` (386B params, `GlmMoeDsaForCausalLM`
|
|
|
+— registered in the fork under `vllm/models/deepseek_v32`, 78 layers,
|
|
|
+hidden 6144, 8 experts/tok, 1M context): **406 GiB of safetensors** on disk
|
|
|
+(84 shards), ~101.5 GiB per node on 4x. First load pass completed (294 s at
|
|
|
+1.5 GB/s), the engine then restarted and began a SECOND full load while the
|
|
|
+first copy was still resident → 2×101.5 > 121.63 GiB → all four nodes went
|
|
|
+OOM-thrashing (0 GiB available, sshd unreachable on 3 of 4). Kill + wait for
|
|
|
+the OOM killer; nodes recovering slowly (swap thrash).
|
|
|
+
|
|
|
+**Answer to the operator: 4 nodes is not enough** — a single copy already
|
|
|
+leaves ~20 GiB/node, which the runtime (vLLM processes ~12 GiB + KV +
|
|
|
+activations + JIT) consumes to zero. 6+ nodes is the safe shape
|
|
|
+(67.6 GiB/node weights → 54 GiB headroom); 5 nodes (dgx5, currently off,
|
|
|
+88.43 in config) would fit at ~81 GiB/node with ~40 GiB headroom if it joins
|
|
|
+the RoCE cluster. Keep the `glm53-spark-nvfp4-4x` recipe as a reference only
|
|
|
+(6 GiB KV pin, r28 image) — do not run on 4x.
|
|
|
+
|
|
|
## Ops notes / lessons
|
|
|
|
|
|
- Node clocks on the Sparks run +2 h ahead of UTC; the launcher's own INFO
|
