|
|
@@ -364,3 +364,70 @@ Boot ladder for `glm53-full-mxfp4-4x-probe` (no DCP, 131K, r35 img):
|
|
|
|
|
|
DCP4 512K pool estimate: 8 GiB/rank ÷ (52.7 KB/token ÷ 4) ≈ 650K tokens
|
|
|
≥ 524288 ✓. Then smoke + dev65 bench `dev65-full-mxfp4-dcp4`, then P2.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## Night shift results (final, 2026-09-14 ~02:45 UTC)
|
|
|
+
|
|
|
+### P0 — full GLM-5.3 (RedHatAI MXFP4, ~740B) on 4× GB10: WORKS (131K lane)
|
|
|
+
|
|
|
+Booted, smoke-passed, benched (label `dev65-full-mxfp4-131k`, report in
|
|
|
+`recipes/glm53-full-mxfp4-4x-probe/reports/`):
|
|
|
+- **Quality 158/176 (90**, their 100-scale "91 Excellent", 1 safety warning
|
|
|
+ TC-51, 87/88 scored).
|
|
|
+- **Perf pp2048/tg128**: d0 c1 **934 pp / 19.0 tg / 2,124 ms**, c2 867/26.5/
|
|
|
+ 3,185; d4096 c1 896/22.6/6,285; d8192 c1 877/22.8/10,406.
|
|
|
+- KV: **52.7 KB/token/rank** (fp8_ds_mla), pool 159,232 tokens per 8 GiB pin,
|
|
|
+ max concurrency 1.21× at 131K. Load ~10 min at ~600 MB/s from NAS.
|
|
|
+- MoE = MarlinExperts (scheme's own pick; `--moe-backend` is inert for the
|
|
|
+ compressed-tensors path), linear = Marlin via `--linear-backend marlin`
|
|
|
+ (the b12x warmup crashes on this checkpoint's shapes; the flashinfer kernel
|
|
|
+ wrongly claims sm_12x support — both documented in the recipe).
|
|
|
+- **512K is BLOCKED on 4×**: DCP4 is the only way 512K fits (26.3 GiB
|
|
|
+ aggregate KV), and the DCP>1 sparse-decode path crashes identically on BOTH
|
|
|
+ r35 and r37: after "DCP sparse indexer currently supports only
|
|
|
+ cp_kv_cache_interleave_size=1 (got 4)" → interleave=1 → graph-capture
|
|
|
+ dummy run dies in deepseek_v32/attention.py:596 with
|
|
|
+ "shape '[16, 16, 512]' is invalid for input of size 524288" (the ×4 DCP
|
|
|
+ factor; the dcp_manager.combine path is gated on use_pcp, which we do not
|
|
|
+ set). Upstream bug to file with the lab. The 512K route for this model is
|
|
|
+ the LMCache RAM/FS tiers (P4a below) or the DCP fix.
|
|
|
+
|
|
|
+### P2 — R37 DS4.1 on 4×: prefill champion, quality regressed
|
|
|
+
|
|
|
+(Report `recipes/dsv41-flash-jovian-r37-4x/reports/`, label
|
|
|
+`dev65-r37-ds41-4x`; image label 9350429b, built on dgx5, 37 min.)
|
|
|
+- **Perf**: d0 c1 **3,830 pp / 46.4 tg / 832 ms** vs prod 1,642/47.0/1,738 →
|
|
|
+ **prefill +133% (2.3×), TTFT 2.1× better, decode flat** (the dispatch
|
|
|
+ removal is a prefill win; the lab measured it on RTX, GB10 matches and
|
|
|
+ beats). c2: 2,688 pp / 71.8 tg.
|
|
|
+- **Quality 150/176 (85), safety-capped** (4 warnings: TC-51/74/82 + one more
|
|
|
+ — the same long-horizon cluster as the r36 K7 run).
|
|
|
+- **Verdict: eugr DS4.1 (nightly-20260910) stays prod of record for agent
|
|
|
+ quality; R37 is the prefill/batch champion** (long-context, low-interactivity
|
|
|
+ workloads, or until the quality regression is fixed).
|
|
|
+
|
|
|
+### P3 skipped (eugr nightly-20260913 A/B)
|
|
|
+The marginal nightly diff is not worth pulling an unverified external image in
|
|
|
+the last hours before teardown; R37 already answers the "newer tree" question
|
|
|
+decisively.
|
|
|
+
|
|
|
+### Teardown
|
|
|
+- dgx5 powered off ~22:33 UTC (operator authorized, at shift start).
|
|
|
+- 4nodes-switch: R37 lane stopped, then `sm121 power off --cluster
|
|
|
+ 4nodes-switch` (operator's instruction for end of shift). No prod restore —
|
|
|
+ the operator re-serves manually in the morning.
|
|
|
+
|
|
|
+### Morning recommendations (priority order)
|
|
|
+1. **File the DCP>1 sparse-decode bug** (repro: full GLM-5.3 MXFP4, DCP4,
|
|
|
+ interleave 1, both r35 and r37 images; exact messages above) — this is the
|
|
|
+ only thing standing between the full model and 512K on 4×.
|
|
|
+2. **512K via LMCache tiers** on the full model (the lab's own pattern:
|
|
|
+ small GPU KV + 64 GiB L1 pinned shm + /cache FS; check host /dev/shm
|
|
|
+ capacity first — Docker --shm-size does not resize host shm).
|
|
|
+3. **R37 as the prefill lane** for batch/long-context; keep eugr for agent
|
|
|
+ traffic. A/B an eugr nightly that inherits the r37 dispatch work if/when
|
|
|
+ the lab merges it.
|
|
|
+4. The b12x-mxfp4-linear warmup rank bug + the flashinfer is_supported
|
|
|
+ sm_12x false-positive (both workarounds are in the recipe; file both).
|
|
|
+5. Push all local commits when git.kilork.org recovers.
|
