full GLM-5.3 native lane on 8x: aiden prod-1.2 image, safetensors loader, dspark k8 @64k

history/night-shifts/2026-09-18.md

@@ -212,15 +212,19 @@ Boot D INFEASIBLE on 4x (OOM incident 14:37 UTC, nodes recovered).
 ### P2 — 8x trials (cables on site; nodes provisioned, cluster configured)
 Gate: CLEARED — all 8 nodes up + validated, `8nodes-switch` in config,
 health green. The launcher syncs missing images at serve time.
-STATUS (18:28 UTC): P2.3 first (DS4.1 8x in-memory engram) — up, quality
-gate clean, bench `8x` running.
-1. [running] **8x full GLM-5.3 MXFP4 @512K DCP1** — up 20:12 UTC (boot
+STATUS (22:10 UTC): P2.3 done (DS4.1 8x, 161/176). P2.1 bench killed by
+operator (partial, results clear — no restart). P2.4 IN PROGRESS: full
+native GLM-5.3 + DSpark K8 on 8x (operator's "whole model" pivot).
+1. [done] **8x full GLM-5.3 MXFP4 @512K DCP1** — up 20:12 UTC (boot
    19:19 → 20:12, ~50 min NFS load 376 GB/rank at ~130 MB/s + capture):
    KV pool **972,032 tokens @512K (1.85x)** — 512K works on the full model
    **without DCP** (sidesteps the DCP>1 sparse-decode bug). Quality gate
-   PASSED (RU text clean, identifiers 6/6). Bench `full8x-512k` running
-   vs 4x 131K anchor (158/176, 934 pp / 19.0 tg / 2,124 ms). Note:
-   `ip route get 192.168.88.16` → dev **enP7s7** (a dedicated mgmt NIC,
+   PASSED (RU text clean, identifiers 6/6). Bench `full8x-512k` ran to
+   ~74/88 (69 pass / 5 fail, incl. the TC-51 family quirk + TC-80) and was
+   **killed by the operator mid-TC-82** ("do not restart — results already
+   clear"): no clean run, no report package; the partial score
+   (≥69/74 ≈ 93+ with 14 remaining) sits between the 4x anchors.
+   Note: `ip route get 192.168.88.16` → dev **enP7s7** (a dedicated mgmt NIC,
    not the 200G rails): the NFS model-load (~1 GB/s aggregate) rides the
    mgmt fabric; the RoCE rails stay idle until NCCL collectives start
    (operator saw the mgmt-switch load in the Mikrotik UI with idle
@@ -264,6 +268,45 @@ gate clean, bench `8x` running.
 Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is
 qualified).
 
+4. [running] **8x full GLM-5.3 NATIVE (zai-org) + DSpark K8 @64K** — the
+   operator's "whole model" pivot (21:25 UTC): NOT the RedHatAI MXFP4
+   quant — the original `zai-org/GLM-5.3` checkpoint (753.3 GB FP8 e4m3
+   block-128, 141 shards, on the NAS: blobs 704 GB, refs/main aca966e4 —
+   already downloaded, my "not on NAS" read was wrong: `du` on the
+   snapshot counted symlinks, not `blobs/`). The architecture is unchanged
+   5.2 → 5.3 (operator), so the lab r34 doc (Gilded Gnosis, GLM-5.2 R7:
+   65K model limit, seqs 8, batched 2048, GMU 0.97/0.98 on RTX 6000)
+   transfers except GMU (the GB10 admission check caps at ~0.85-0.89;
+   measured free 108.78 GiB < 0.91 requirement).
+   - **Memory**: 753.3/8 = 94.2 GB/node weights; GMU 0.85 → ~6.8 GB KV →
+     64K request = 3.45 GB/rank (1.9x), 96K = 5.1 GB (1.3x); **512K does
+     not fit** (26.3 GB) — physical limit, hence max-model-len 65536.
+   - **Image**: `aidendle94/sparkrun-vllm-glm53-gb10:production-1.2`
+     (31.2 GB, Aiden's GLM-5.2/5.3 production stack — the operator's
+     "aiden 5.2 image"; verified in-image: DSparkDraftModel speculators
+     loader + glm_moe_dsa arch both present; their DFlash2 drafter runs
+     without the r37 workarounds → skip-layers flag + draft page patch
+     NOT ported). `production-3.75` (the ds4-gb10 DS4 image) is a dead
+     end: no DSpark, no glm_moe_dsa (vLLM 0.11.2 base). Aiden's lil
+     launcher repo (`local-inference-lab/lil`, Go) + `configs/models/
+     _bases.yaml` read: glm family = reasoning glm45 / tool glm47 /
+     B12X attention / VLLM_B12X_MOE_FP4_FORCE_A16=1.
+   - **Loader (operator call)**: plain safetensors, not InstantTensor.
+     `--load-format auto` on the Aiden image = InstantTensor (verified in
+     the first boot log: "Loading safetensors using InstantTensor loader",
+     704 GB/rank stream at ~130 MB/s). First Aiden boot died 137 at 21:32
+     (the launcher's own stale-container cleanup + a hung r37 first boot
+     on the same recipe name; no host OOM, journal clean). With explicit
+     `--load-format safetensors`: the standard per-shard loader runs
+     (141 shards, ~40 s/shard, ~145 MB/s, started 21:54, ETA ~95 min —
+     same per-node NFS volume as InstantTensor with one rank/node, but
+     stable and no staging hang).
+   - Recipe `glm53-full-native-8x-dspark8`: TP8, DCP1, GMU 0.85, 64K,
+     seqs 8, batched 2048, kv fp8_e4m3, moe-backend marlin (the Aiden
+     baseline spellings), block 256, speculator K8 dspark, no workarounds.
+   - After boot: quality gate (RU + identifiers) + bench vs the 4x dspark8
+     anchor (153/176, 892 pp / 33.2 tg / 2,193 ms @131K).
+
 ### P3 — optional, time permitting
 1. **Qwen3.8-Flash-Next-NVFP4** quick trial (1–2 nodes): new family,
    catalog-updated 09-15 (image limit dropped — vision serving);
@@ -298,6 +341,14 @@ qualified).
 - End-state: per operator. Default guardrail unchanged: switch powered
   down at end **unless** the operator leaves an 8x lane serving.
 
+### P5 — r37 rebuild under the latest pins (operator, 21:55 UTC)
+- "r37 пора бы уже пересобрать под последние реалии": re-cut the jovian DS4.1
+  image against the current upstream (eugr b12x master past f1c4e9dd,
+  vLLM dev past c687594b, flashinfer, LMCache) — a new rev (r38?) from a
+  fresh source.lock. Not blocking tonight (the full-native lane runs on
+  the Aiden image; P1's garbling verdict is already in); do it after the
+  night's results, then re-baseline the anchor lane before any adoption.
+
 ## Guardrails
 
 - **Nothing is serving right now** (the 2x eugr lane was stopped 09-18);

recipes/glm53-full-native-8x-dspark8/recipe.toml

@@ -0,0 +1,127 @@
+# GLM-5.3 (full, NATIVE zai-org FP8) + DSpark K8 speculator — 8x @64K.
+#
+# The operator's "whole model" lane (2026-09-18 night): NOT the RedHatAI
+# MXFP4 quant — the original `zai-org/GLM-5.3` checkpoint (753.3 GB, FP8
+# e4m3 block-128, 141 shards, already on the NAS: blobs 704 GB, refs/main
+# aca966e4). Same glm_moe_dsa architecture as the GLM-5.2 lane (the
+# operator's read: the architecture did not change between 5.2 and 5.3),
+# so the Aiden GLM image family carries it.
+#
+# Image: aidendle94/sparkrun-vllm-glm53-gb10:production-1.2 (31.2 GB,
+# Aiden's GLM-5.2/5.3 production stack — the "aiden 5.2" image). Verified
+# in-image (2026-09-18): vLLM dev20051+g487ecf187 has both the
+# DSparkDraftModel speculators loader (config/speculative.py +
+# models/registry.py) and the glm_moe_dsa path
+# (models/deepseek_v32/nvidia/model.py). Their DFlash2 drafter runs on
+# this image WITHOUT the r37 workarounds (no skip-layers flag, no draft
+# page patch) -> the shared-cache-contamination bug is absent from this
+# tree, so the 4x-dspark8 r37 workarounds are NOT ported.
+#
+# Loader (operator call): the standard safetensors loader
+# (`--load-format auto`), NOT InstantTensor. InstantTensor streams the
+# WHOLE checkpoint per rank (~750 GB/rank over the 10G NFS mgmt NIC,
+# ~90-100 min); the standard loader reads each rank's TP slices
+# (page-cache shared) and the dspark38 Aiden precedent already runs on
+# `auto`.
+#
+# Memory math (121.63 GiB/node): weights 753.3/8 = 94.2 GB/node;
+# GMU 0.85 -> 111.0 GB budget; 111.0 - 94.2 - ~10 (process) = ~6.8 GB KV.
+# 64K request = 3.45 GB/rank (52.7 KB/token) -> ~1.9x; 96K = 5.1 GB ->
+# ~1.3x; 512K (26.3 GB) DOES NOT FIT — physical limit, hence
+# max-model-len 65536 (the lab r34 doc's qualified Model limit; their
+# GMU 0.97/0.98 numbers are RTX 6000 (96 GB, smaller baseline reservation)
+# and do not pass the GB10 admission check — measured free 108.78 GiB).
+#
+# Speculator: RedHatAI/GLM-5.3-speculator.dspark (2.3 GB, NAS snapshot
+# 198b3ced) at K8, `method: dspark` — the same drafter as the 4x dspark8
+# lane (that one on the r37 image; here on Aiden's tree, DSpark support
+# verified in-image).
+#
+# Contract mirrors the proven Aiden precedents on this image family:
+# glm53-flash-aiden-2x (same image, DFlash2: kv fp8_e4m3, moe-backend
+# marlin, block 2304, no --attention-backend — the image routes B12X
+# natively) + deepseek-v4-flash-dspark38 (DSpark + CUDA graphs without
+# enforce-eager, --load-format auto, block 256) + the r34 doc's scheduler
+# bounds (seqs 8, batched tokens 2048).
+engine = "vllm"
+name = "GLM-5.3 (full native FP8) + DSpark K8 (aiden prod-1.2 img, zai-org ckpt, 8x @64K)"
+docker_image = "aidendle94/sparkrun-vllm-glm53-gb10:production-1.2"
+entrypoint = []
+model = "zai-org/GLM-5.3"
+parent_model = "zai-org/GLM-5.3"
+required_nodes = 8
+tensor_parallel_size = 8
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+args = [
+    # Stable served name — quant-agnostic (shared across glm53 lanes)
+    "--served-model-name", "glm53",
+    "--trust-remote-code",
+    "--reasoning-parser", "glm45",
+    "--tool-call-parser", "glm47",
+    "--enable-auto-tool-choice",
+    "--default-chat-template-kwargs", """
+'{"enable_thinking": true, "reasoning_effort": "high"}'""",
+    # Aiden-image-proven spellings (glm53-flash-aiden-2x baseline):
+    "--kv-cache-dtype", "fp8_e4m3",
+    "--moe-backend", "marlin",
+    # the plain safetensors loader (operator call 2026-09-18): InstantTensor
+    # streams the WHOLE checkpoint per rank (~704 GB/rank at 130 MB/s = ~90
+    # min, and it hung in the first Aiden boot); `auto` on this image =
+    # InstantTensor (verified in the boot log), so the format is spelled
+    # explicitly. The standard loader reads each rank's TP slices with mmap
+    # (page-cache shared) — per-node NFS traffic ≈ the model, not 8x it.
+    "--load-format", "safetensors",
+    # block 256 (dspark38 precedent; the baseline's 2304 is an NVFP4-Flash
+    # knob)
+    "--block-size", "256",
+    # No KV pin: the pool is GMU-driven (~6.8 GB, see header).
+    "--max-model-len", "65536",
+    "--max-num-seqs", "8",
+    "--max-num-batched-tokens", "2048",
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--gpu-memory-utilization", "0.85",
+    # DSpark K8 speculator (RedHatAI/GLM-5.3-speculator.dspark, 2.3 GB, in
+    # the shared HF cache; HF_HUB_OFFLINE=1 resolves it there).
+    "--speculative-config", """
+'{"model":"RedHatAI/GLM-5.3-speculator.dspark","num_speculative_tokens":8,"method":"dspark"}'""",
+    "--language-model-only",
+]
+
+[env]
+
+HF_HUB_OFFLINE = "1"
+TRANSFORMERS_OFFLINE = "1"
+TORCH_USE_RTLD_GLOBAL = "1"
+TORCHINDUCTOR_COMPILE_THREADS = "1"
+CUTE_DSL_ARCH = "sm_121a"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+VLLM_ENGINE_READY_TIMEOUT_S = "3600"
+VLLM_NO_USAGE_STATS = "1"
+OMP_NUM_THREADS = "2"
+# Compile-OOM guard (cold /root/.cache compiles behind 94 GB of resident
+# weights; the JIT mounts below keep it warm after the first boot).
+MAX_JOBS = "2"
+
+# JIT caches persist per-node across serves (cold /root/.cache -> ~20 min
+# recompile every serve). The host dirs are shared between lanes of the
+# same arch — each cache self-namespaces by its engine version internally.
+[[extra_mounts]]
+host = "~/sm121-cache/glm53-jit-121f/b12x"
+container = "/root/.cache/b12x"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/flashinfer"
+container = "/root/.cache/flashinfer"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/torchinductor"
+container = "/root/.cache/torchinductor"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/vllm"
+container = "/root/.cache/vllm"