EXL3 recipe boot diagnosis + sparkrun bench numbers in spec ledger

EXL3 (Entrpi image): entrypoint/drafter/GMU fixed, then global OOM at
75-95% of shard load streaming from the shared NAS on both nodes —
reproduced twice (dmesg 12:22:25, 13:12:24). Per-node mirror is the
remaining path (dgx4 at 98% disk — operator to pick cleanup target).
sparkrun image benched: 89/100 quality, tg 14.3/23.8 t/s, resp 12/100 —
works but not performant (no MTP in that image).

docker-images/glm53-flash-nvfp4-lab/build.sh

@@ -15,6 +15,7 @@ while IFS= read -r line; do
         ''|\#*) continue ;;
     esac
     key="${line%%=*}"; val="${line#*=}"
+    val="${val%\"}"; val="${val#\"}"   # versions.env values may be shell-quoted
     [ -n "$key" ] && ARGS+=(--build-arg "${key}=${val}")
 done < versions.env
 

recipes/glm53-flash-exl3/recipe.toml

@@ -31,6 +31,21 @@
 #     NVMe per box; NFS mode exists upstream too). First load is slow;
 #     JIT caches persist per node (see extra_mounts).
 #
+# =====================================================================
+# STATUS 2026-08-30: boots fully to weight-stream, then global-OOM kills
+# the worker at 75-95% of shard load — REPRODUCED TWICE (12:22:25 and
+# 13:12:24, dmesg oom-killer) while streaming the 164 GiB checkpoint from
+# the shared NAS on BOTH nodes. GB10 unified memory + NFS page cache
+# cannot keep up (their kit streams NFS to the HEAD only; the worker reads
+# local NVMe, and even their local mode "wedges into swap at ~90%" by
+# their own caveat). Fixed knobs so far: entrypoint (NGC wrapper execs
+# first CMD arg as a file — `vllm serve` must be explicit), drafter repo
+# (incoai raw drafter is NOT in the NAS hub — we serve the MXFP8 mirror),
+# GMU 0.86 (their 0.85 left 1.14 GiB free when warmup wanted 1.12).
+# REMAINING FIX: per-node local mirror (sm121 --mirror after pointing
+# hf_cache_dir at the NAS). dgx3 has 400+ GiB free; dgx4 is 98% full —
+# blocked on local-cache cleanup there.
+#
 # Boot markers (engine log): "Model loading took ~88 GiB"-scale BEFORE any
 # EXL3 trellis lines → "GPU KV cache size" ≈ 1.4M tokens (fp8_e4m3 @524k)
 # → "Application startup complete". A "Setting attention block size to
@@ -51,6 +66,11 @@ tensor_parallel_size = 2
 pipeline_parallel_size = 1
 distributed_backend = "mp"
 shm_size = "32gb"
+# The image's NGC entrypoint wrapper (/opt/nvidia/nvidia_entrypoint.sh)
+# execs the first CMD argument as a FILE unless it is a known command —
+# so `vllm serve` must be passed explicitly, as the upstream launch
+# script does (verified: first arg landed as a relative model path).
+entrypoint = ["vllm", "serve"]
 
 args = [
     # Stable served name — quant-agnostic (shared across glm53 quants)
@@ -60,7 +80,10 @@ args = [
     "--quantization", "exl3",
     "--distributed-executor-backend", "mp",
     # Their validated defaults; 0.88+ risks unified-memory swap on GB10
-    "--gpu-memory-utilization", "0.85",
+    # 0.85 upstream default left 1.14 GiB when the warmup wanted 1.12 GiB
+# (OOM at engine init, 2026-08-30); 0.86 = one notch of slack. Their
+# 0.88+ warning is about swap wedging — 0.86 stays below that.
+    "--gpu-memory-utilization", "0.86",
     "--max-model-len", "524288",
     "--max-num-seqs", "4",
     # vLLM auto-bumps to 4608 under fp8 KV (KDA state page parity)
@@ -71,9 +94,13 @@ args = [
     # 1,435,070-token pool at 524k fp8; do NOT raise (2.26 GiB floor @13.4e9)
     "--kv-cache-memory", "12400000000",
     # Drafter by hub id (resolved from the mounted NAS HF_HOME like the
-    # target model; 7 = trained block size 8 minus the bonus token)
+    # target model; 7 = trained block size 8 minus the bonus token).
+    # Upstream uses `incoai/GLM-5.3-Flash-DFlash2` (raw fp8-e4m3 drafter);
+    # we serve the MXFP8-compressed qualified mirror instead — already in
+    # the NAS hub, and this build executes serialized MXFP8 projections
+    # natively. Falls forward to the raw drafter if quality differs.
     "--speculative-config", """
-'{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7}'""",
+'{"method":"dflash","model":"local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8","num_speculative_tokens":7}'""",
     # required at 524288 on GB10 unified memory (upstream default SKIP_MM_PROFILING=1)
     "--skip-mm-profiling",
     "--tool-call-parser", "glm47",

specs/tech-architecture/glm53-recipe-state.md

@@ -18,6 +18,7 @@ All three share served name **`glm53`** — clients never changed between runs.
 | `glm53-flash-nvfp4-redhat` | RedHatAI W4A4 compressed-tensors | **90** | 159/176 | 13.9 | 13.8 | 2.2 s | 67 | ❌ (no MTP weights in checkpoint) |
 | `glm53-flash-nvfp4` | LibertAIDAI NVFP4-A16 | **88** | 154/176 | 26.2 | 22.7 | 6.3 s | 69 | ✅ k=5 |
 | `glm53-flash-nvfp4-4p67` | 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 | **87** | 153/176 | 24.4 | 39.2 | 1.4 s | 67 | ✅ accept ~90% |
+| `glm53-flash-sparkrun` | community sparkrun image (aidendle94), NVFP4, eager, NO MTP (image lacks the fix) | **89** | 154/176 | 14.3 | 23.8 | 1.4 s | 66 | ❌ (n/a in image) |
 
 (lab scored /174 — TC-53 was excluded from its run as an infrastructure
 failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s
@@ -158,3 +159,28 @@ Findings:
 
 Reports: `/tmp/runs/2026/08/2026-08-29T09-09-14…--(eager)`,
 `…10-27-35.818420Z_181bbd18.md` (graphs).
+
+## sparkrun (community image) A/B (2026-08-30, seed 42)
+
+aidendle94/sparkrun-vllm-glm53-gb10:production-1.0 (same day-0 vLLM dev
+base as our lab image, arm64-only) serving the same lab quant: quality
+89 (154/176) but decode tg 14.3/23.8 t/s (c1/c2) vs lab-eager 24.4/39.2
+and responsiveness 12/100 (median turn 11.7 s) — works, not performant:
+- MTP impossible in that image (no modelopt-MTP namespace fix).
+- --moe-backend marlin must be pinned (their auto = TileLang JIT storm,
+  worker OOM-killed at boot), nvrtc.h bind-mount needed, plain fp8 KV
+  (their mla kernel asserts pe_dim==64 for fp8_ds_mla).
+Report: /tmp/runs/2026/08/2026-08-30T09-41-57.562243Z_108a798b.md.
+
+## EXL3 lane (Entrpi community image) — boot diagnosis (2026-08-30)
+
+ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2 + brandonmusic
+GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. Fixed in-recipe: NGC
+entrypoint (needs explicit `vllm serve`), drafter swap to the MXFP8
+mirror (incoai raw drafter absent from the NAS hub), GMU 0.86. BLOCKED:
+global OOM kills the worker at 75-95% of shard load on BOTH streamed-from-
+NAS nodes (repro 2×; dmesg oom-killer 12:22:25 / 13:12:24). Upstream's
+"validated NFS topology" does not hold on our fabric — the mirror
+topology (per-node local NVMe) is the next step; dgx4 is 98% full
+(candidate cleanup targets ~1.3T of re-downloadable zai BF16 repos
+there) — awaiting operator call.