# GLM-5.3-Flash NVFP4 — three quant variants measured on 2× DGX Spark (ds4)

Measured 2026-08-27/28 on cluster `ds4` (dgx3+dgx4, switched fabric), all three
served by sm121 through the launcher, same benchmark invocation:

```
tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53 \
  --seed 42 --hardmode --perf --concurrency 1,2
```

All three share served name **`glm53`** — clients never changed between runs.

## Results

| recipe | quant | quality | points | decode tg t/s (c1) | tg c2 | TTFT | deployability | MTP |
|---|---|---|---|---|---|---|---|---|
| `glm53-flash-nvfp4-lab` | local-inference-lab mixed NVFP4/MXFP8-MTP | **92** (eager) / 89 (graphs) | 162/176 | **28.4** | 40.5 | 5.6 s | **72** | ✅ k=3, accept len ~2.9–3.5, ~63–83 % |
| `glm53-flash-nvfp4-redhat` | RedHatAI W4A4 compressed-tensors | **90** | 159/176 | 13.9 | 13.8 | 2.2 s | 67 | ❌ (no MTP weights in checkpoint) |
| `glm53-flash-nvfp4` | LibertAIDAI NVFP4-A16 | **88** | 154/176 | 26.2 | 22.7 | 6.3 s | 69 | ✅ k=5 |
| `glm53-flash-nvfp4-4p67` | 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 | **87** | 153/176 | 24.4 | 39.2 | 1.4 s | 67 | ✅ accept ~90% |
| `glm53-flash-sparkrun` | community sparkrun image (aidendle94), NVFP4, eager, NO MTP (image lacks the fix) | **89** | 154/176 | 14.3 | 23.8 | 1.4 s | 66 | ❌ (n/a in image) |

(lab scored /174 — TC-53 was excluded from its run as an infrastructure
failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s
total — harness drops timed-out scenarios from the denominator; had it scored
like redhat's 2/2, lab would be 157/176 = 89 — no change). Quality % is
comparable. LibertAIDAI is the only variant whose MTP *regresses* under c2
(26.2→22.7) — lab's k=3 draft scales, LibertAIDAI's k=5 does not.)

Prefill (pp2048) 0.3–1.7 K t/s depending on depth/concurrency — noisy, all three
in the same band (NAS load-influenced); decode is the stable discriminator.

## Read of the numbers

- **lab (eager) wins outright**: 92/100 best quality AND 2× decode (28–40 t/s
  vs ~14 baseline). Acceptance lengths matched upstream's 2.55–3.5 range.
- **redhat** (90/100 with graphs) is a **baseline-decode dead-end on this box:
  its checkpoint ships zero MTP tensors** (verified index: 146,745 tensors,
  none under layer 45), so it is stuck at ~14 t/s — no longer the quality
  pick either.
- **lab --enforce-eager = day-driver; redhat = fallback; LibertAIDAI =**
  proven first but k=5 MTP regresses at c2 (eager might lift it too — untested).

## Images & patches

| image dir | base | patches |
|---|---|---|
| `glm53-flash-nvfp4` | vllm/vllm-openai:glm53-flash-arm64-cu130 | kingjones30 NoPE-MLA mod (4-part, anchor-asserted) |
| `glm53-flash-nvfp4-lab` | same | + 4 FujitsuPolycom patches (Apache-2.0, see patches/UPSTREAM.md): model.py naming shim, modelopt MTP namespace fix, persistent_topk CC-12.x guards ×2 |
| redhat recipe reuses `glm53-flash-nvfp4:local` (mod is quant-independent) | | |

Boot markers `[quantprobe] layer=RoutedExperts prefix=model.layers.45.mlp.experts
algo=MXFP8` confirm the MTP fix is live; `algo=None` there = broken load.

## MTP + CUDA graphs quality bug (forum #381350/#44, CONFIRMED 2026-08-28)

`0rand`: “mtp or dflash plus cuda graphs seriously hit quality — try
disabling cuda graph capture”. A/B with identical seed/flags on lab:

| arm | quality | points | tg c1 | tg c2 | TTFT |
|---|---|---|---|---|---|
| graphs (CUDA graphs ON) | 89 | 155/174 | 28.4 | 43.4 | 3.9 s |
| **`--enforce-eager`** | **92** | **162/176** | 28.4 | 40.5 | 5.6 s |

+3 quality points for −7 % c2 decode — eager is now the lab default. Accept
metrics were healthy in BOTH arms (71.6 %, len 3.15 eager), so this is not an
acceptance collapse — graph capture corrupts sampled output silently.
Corroborating: randomllama (DFlash2 recipe) already flagged “not claiming
bit-identical greedy outputs” with graphs on. The b12x glm52 stack runs
graphs + MTP k=4 FULL_AND_PIECEWISE benches clean (100/100) — build-specific
bug in the glm53 day-0 dev build, do NOT propagate --enforce-eager to glm52.
Untested follow-ups: LibertAIDAI + eager (k=5; expect ~88→9x), redhat is
graph-neutral (no MTP, stays 90 with graphs).

## Operational notes (measured)

### redhat @ 524288 ctx (2026-08-28)

262144 booted on the base image; **512K needed three changes** (each verified
by a real boot that died without it):
1. `--kernel-config {"enable_flashinfer_autotune":false,"enable_cutedsl_warmup":false}`
   + `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` — autotune scratch at
   512K shapes OOMs the driver (`NVRM NV_ERR_NO_MEMORY _memdescAllocInternal`);
   the sparse-MLA kernels' non-autotuned fallback is fine (lab proved it).
2. `--max-num-batched-tokens 4096` + `--block-size 256` (lab's memory shape).
3. **The lab image** — the killer was `launch_persistent_topk ... total_ctas=62
   > num_sms*occupancy=48 ... fallback requires >=128KB smem (have 101376)`,
   i.e. exactly what the FujitsuPolycom `sparse_attn_indexer(_kpool)` CC-12.0
   guards fix. Indexer CTAs scale with ctx: 262K fit 48 SMs, 512K needs 62.
   The patches are arch-level → safe for the compressed-tensors checkpoint
   (quantprobe is a no-op without modelopt; output verified correct).

Result: KV pool **1,334,265 tokens / 2.54× @ 524288** (10 GiB pin; lab gets
1,164,369 from the same pin — its MTP speculator reserves memory before KV
sizing), ~17 min clean boot, correct greedy output.

Note: an orphaned **rank-1 headless worker survives rank-0 death** and keeps
the node occupied — always `docker rm -f` label-filtered leftovers before a
retry serve (or the launcher refuses with "only 1 nodes").

- lab weight load: 92.19 GiB / 700 s sequential from the NAS HF cache;
  `--kv-cache-memory-bytes 10737418240` → KV 1,164,369 tokens / 2.22× @ 512K
  (matches upstream to the digit — do NOT switch back to GMU sizing).
- First launcher serve of lab died on both ranks at ~54 s (exit 1, logs lost
  to `--rm`); re-run was clean twice. Suspected transient (cold imports +
  simultaneous launch). If it recurs: re-launch without `--rm` and keep logs.
- `sm121 serve` immediately after `sm121 stop` can fail with "cluster has only
  1 nodes" — occupancy race, nodes not yet released; wait ~10 s and retry.
- LibertAIDAI/redhat recipes still run FlashInfer autotune (60-min possible
  boot, `VLLM_ENGINE_READY_TIMEOUT_S=3600` set); the lab recipe disables it via
  `--kernel-config` per upstream's proven boot.
- Chat-template caveat: `reasoning_effort:"low"` still emits a reasoning trace
  (budget ≠ 0); `max_tokens:48` can yield `content: null, finish: length` —
  use ≥256 for smoke tests.

## Follow-ups

- Vendor instanttensor loader into the lab image (~30 s vs 700 s weight load).
- Consider adding `--kernel-config` autotune-off to the LibertAIDAI recipe
  (boot minutes shorter; redhat now has it, see 512K section — verify before
  trusting on LibertAIDAI).
- MTP for redhat: RedHatAI ships MTP as **separate variant repos** (proven
  pattern: `RedHatAI/GLM-5.2-NVFP4-FP8-mtp-FP8` = full quant + `mtp.safetensors`
  alongside main weights). Nothing for 5.3-Flash yet (org search 2026-08-28:
  only NVFP4 + BF16). If/when `GLM-5.3-Flash-NVFP4-mtp-*` lands, the recipe
  change is: swap `model` + add `--speculative-config` (k=3). Cross-grafting
  the lab's MXFP8 MTP as an external `speculative-config.model` draft is
  theoretically possible but untested in vLLM — don't chase before the
  official drop.

## 4p67: --enforce-eager vs CUDA graphs (2026-08-29, same seed/invocation)

Same recipe both arms (MTP k=3 included), only `--enforce-eager` removed;
graphs arm verified to capture (FULL 5/5 + PIECEWISE 11/11), KV identical
(1,164,369 tok pinned).

| metric | eager | graphs |
|---|---|---|
| quality | **87** (153/176) | **88** (154/176) |
| deployability | 67 | 68 |
| tg t/s c1 (d0/d4096/d8192) | 21.1 / 23.2 / **24.4** | **23.6** / 22.6 / 22.9 |
| tg t/s c2 (d0/d4096) | **39.2** / 22.9 | 33.1 / **24.0** |
| TTFT (d0 / d8192) | 1.45 s / 6.17 s | 1.41 s / 6.10 s |
| MTP draft accept | 90.0–92.2 % | 89.0–97.0 % |

Findings:

* **No graphs+MTP quality regression on 4p67** (88 vs 87 = 1 point, run-to-run
  noise; the failure sets differ only by 5 shuffled partials). The lab
  92→89 graphs regression (forum #44) does NOT reproduce on this quant —
  so the bug is build/quant-specific, not universal.
* **Graphs buy nothing on decode here**: every cell within ±5%, and the
  headline c2 number is *worse* with graphs (39.2→33.1). MTP already eats
  the launch-gap that graphs would recover.
* Prefill rows are prefix-cache-polluted (recipe enables prefix caching and
  the perf probe repeats prompts); cold prefill ~1.4–1.5 K t/s both arms.
* **Decision: `--enforce-eager` stays in `glm53-flash-nvfp4-4p67`** — it is
  measurably free here and stays immune to the lab graph bug class.

Reports: `/tmp/runs/2026/08/2026-08-29T09-09-14…--(eager)`,
`…10-27-35.818420Z_181bbd18.md` (graphs).

## sparkrun (community image) A/B (2026-08-30, seed 42)

aidendle94/sparkrun-vllm-glm53-gb10:production-1.0 (same day-0 vLLM dev
base as our lab image, arm64-only) serving the same lab quant: quality
89 (154/176) but decode tg 14.3/23.8 t/s (c1/c2) vs lab-eager 24.4/39.2
and responsiveness 12/100 (median turn 11.7 s) — works, not performant:
- MTP impossible in that image (no modelopt-MTP namespace fix).
- --moe-backend marlin must be pinned (their auto = TileLang JIT storm,
  worker OOM-killed at boot), nvrtc.h bind-mount needed, plain fp8 KV
  (their mla kernel asserts pe_dim==64 for fp8_ds_mla).
Report: /tmp/runs/2026/08/2026-08-30T09-41-57.562243Z_108a798b.md.

## EXL3 lane (Entrpi community image) — boot diagnosis (2026-08-30)

ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2 + brandonmusic
GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. Fixed in-recipe: NGC
entrypoint (needs explicit `vllm serve`), drafter swap to the MXFP8
mirror (incoai raw drafter absent from the NAS hub), GMU 0.86. BLOCKED:
global OOM kills the worker at 75-95% of shard load on BOTH streamed-from-
NAS nodes (repro 2×; dmesg oom-killer 12:22:25 / 13:12:24). Upstream's
"validated NFS topology" does not hold on our fabric — the mirror
topology (per-node local NVMe) is the next step; dgx4 is 98% full
(candidate cleanup targets ~1.3T of re-downloadable zai BF16 repos
there) — awaiting operator call.