Home Esplora Aiuto
Accedi
sm121
/
launcher
2
0
Forka 0
File Problemi 0 Pull Requests 0 Wiki
Albero (Tree): 79d1a76c74
Rami (Branch) Tag
launcher
/
specs
/
tech-architecture
/
glm53-recipe-state.md

glm53-recipe-state.md 7.1 KB
Cronologia Originale

GLM-5.3-Flash NVFP4 — three quant variants measured on 2× DGX Spark (ds4)

Measured 2026-08-27/28 on cluster ds4 (dgx3+dgx4, switched fabric), all three served by sm121 through the launcher, same benchmark invocation:

tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53 \
  --seed 42 --hardmode --perf --concurrency 1,2

All three share served name glm53 — clients never changed between runs.

Results

recipe quant quality points decode tg t/s (c1) tg c2 TTFT deployability MTP
glm53-flash-nvfp4-lab local-inference-lab mixed NVFP4/MXFP8-MTP 92 (eager) / 89 (graphs) 162/176 28.4 40.5 5.6 s 72 ✅ k=3, accept len ~2.9–3.5, ~63–83 %
glm53-flash-nvfp4-redhat RedHatAI W4A4 compressed-tensors 90 159/176 13.9 13.8 2.2 s 67 ❌ (no MTP weights in checkpoint)
glm53-flash-nvfp4 LibertAIDAI NVFP4-A16 88 154/176 26.2 22.7 6.3 s 69 ✅ k=5

(lab scored /174 — TC-53 was excluded from its run as an infrastructure failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s total — harness drops timed-out scenarios from the denominator; had it scored like redhat's 2/2, lab would be 157/176 = 89 — no change). Quality % is comparable. LibertAIDAI is the only variant whose MTP regresses under c2 (26.2→22.7) — lab's k=3 draft scales, LibertAIDAI's k=5 does not.)

Prefill (pp2048) 0.3–1.7 K t/s depending on depth/concurrency — noisy, all three in the same band (NAS load-influenced); decode is the stable discriminator.

Read of the numbers

  • lab (eager) wins outright: 92/100 best quality AND 2× decode (28–40 t/s vs ~14 baseline). Acceptance lengths matched upstream's 2.55–3.5 range.
  • redhat (90/100 with graphs) is a baseline-decode dead-end on this box: its checkpoint ships zero MTP tensors (verified index: 146,745 tensors, none under layer 45), so it is stuck at ~14 t/s — no longer the quality pick either.
  • lab --enforce-eager = day-driver; redhat = fallback; LibertAIDAI = proven first but k=5 MTP regresses at c2 (eager might lift it too — untested).

Images & patches

image dir base patches
glm53-flash-nvfp4 vllm/vllm-openai:glm53-flash-arm64-cu130 kingjones30 NoPE-MLA mod (4-part, anchor-asserted)
glm53-flash-nvfp4-lab same + 4 FujitsuPolycom patches (Apache-2.0, see patches/UPSTREAM.md): model.py naming shim, modelopt MTP namespace fix, persistent_topk CC-12.x guards ×2
redhat recipe reuses glm53-flash-nvfp4:local (mod is quant-independent)

Boot markers [quantprobe] layer=RoutedExperts prefix=model.layers.45.mlp.experts algo=MXFP8 confirm the MTP fix is live; algo=None there = broken load.

MTP + CUDA graphs quality bug (forum #381350/#44, CONFIRMED 2026-08-28)

0rand: “mtp or dflash plus cuda graphs seriously hit quality — try disabling cuda graph capture”. A/B with identical seed/flags on lab:

arm quality points tg c1 tg c2 TTFT
graphs (CUDA graphs ON) 89 155/174 28.4 43.4 3.9 s
--enforce-eager 92 162/176 28.4 40.5 5.6 s

+3 quality points for −7 % c2 decode — eager is now the lab default. Accept metrics were healthy in BOTH arms (71.6 %, len 3.15 eager), so this is not an acceptance collapse — graph capture corrupts sampled output silently. Corroborating: randomllama (DFlash2 recipe) already flagged “not claiming bit-identical greedy outputs” with graphs on. The b12x glm52 stack runs graphs + MTP k=4 FULL_AND_PIECEWISE benches clean (100/100) — build-specific bug in the glm53 day-0 dev build, do NOT propagate --enforce-eager to glm52. Untested follow-ups: LibertAIDAI + eager (k=5; expect ~88→9x), redhat is graph-neutral (no MTP, stays 90 with graphs).

Operational notes (measured)

redhat @ 524288 ctx (2026-08-28)

262144 booted on the base image; 512K needed three changes (each verified by a real boot that died without it):

  1. --kernel-config {"enable_flashinfer_autotune":false,"enable_cutedsl_warmup":false}
    • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — autotune scratch at 512K shapes OOMs the driver (NVRM NV_ERR_NO_MEMORY _memdescAllocInternal); the sparse-MLA kernels' non-autotuned fallback is fine (lab proved it).
  2. --max-num-batched-tokens 4096 + --block-size 256 (lab's memory shape).
  3. The lab image — the killer was launch_persistent_topk ... total_ctas=62 > num_sms*occupancy=48 ... fallback requires >=128KB smem (have 101376), i.e. exactly what the FujitsuPolycom sparse_attn_indexer(_kpool) CC-12.0 guards fix. Indexer CTAs scale with ctx: 262K fit 48 SMs, 512K needs 62. The patches are arch-level → safe for the compressed-tensors checkpoint (quantprobe is a no-op without modelopt; output verified correct).

Result: KV pool 1,334,265 tokens / 2.54× @ 524288 (10 GiB pin; lab gets 1,164,369 from the same pin — its MTP speculator reserves memory before KV sizing), ~17 min clean boot, correct greedy output.

Note: an orphaned rank-1 headless worker survives rank-0 death and keeps the node occupied — always docker rm -f label-filtered leftovers before a retry serve (or the launcher refuses with "only 1 nodes").

  • lab weight load: 92.19 GiB / 700 s sequential from the NAS HF cache; --kv-cache-memory-bytes 10737418240 → KV 1,164,369 tokens / 2.22× @ 512K (matches upstream to the digit — do NOT switch back to GMU sizing).
  • First launcher serve of lab died on both ranks at ~54 s (exit 1, logs lost to --rm); re-run was clean twice. Suspected transient (cold imports + simultaneous launch). If it recurs: re-launch without --rm and keep logs.
  • sm121 serve immediately after sm121 stop can fail with "cluster has only 1 nodes" — occupancy race, nodes not yet released; wait ~10 s and retry.
  • LibertAIDAI/redhat recipes still run FlashInfer autotune (60-min possible boot, VLLM_ENGINE_READY_TIMEOUT_S=3600 set); the lab recipe disables it via --kernel-config per upstream's proven boot.
  • Chat-template caveat: reasoning_effort:"low" still emits a reasoning trace (budget ≠ 0); max_tokens:48 can yield content: null, finish: length — use ≥256 for smoke tests.

Follow-ups

  • Vendor instanttensor loader into the lab image (~30 s vs 700 s weight load).
  • Consider adding --kernel-config autotune-off to the LibertAIDAI recipe (boot minutes shorter; redhat now has it, see 512K section — verify before trusting on LibertAIDAI).
  • MTP for redhat: RedHatAI ships MTP as separate variant repos (proven pattern: RedHatAI/GLM-5.2-NVFP4-FP8-mtp-FP8 = full quant + mtp.safetensors alongside main weights). Nothing for 5.3-Flash yet (org search 2026-08-28: only NVFP4 + BF16). If/when GLM-5.3-Flash-NVFP4-mtp-* lands, the recipe change is: swap model + add --speculative-config (k=3). Cross-grafting the lab's MXFP8 MTP as an external speculative-config.model draft is theoretically possible but untested in vLLM — don't chase before the official drop.