EXL3 recipe: verified serving via instanttensor loader overlay

Root-caused the three OOM kills: the stock loader balloons ~80 GiB of
CPU-side anon while streaming the 164 GiB EXL3 checkpoint from the NAS;
unified memory (GPU pool + loader buffers) hits GB10's 121.6 GiB wall.
Overlay image (docker-images/glm53-flash-exl3): digest-pinned Entrpi
upstream + instanttensor==0.1.9 (--no-deps). Recipe serves with
--load-format instanttensor + the upstream incoai drafter (downloaded to
the NAS hub). Verified on ds4: stream @ ~175 MB/s, no OOM, ~22.8 tok/s
with DFlash2 k=7 (accept 48-81%), math smoke exact.

docker-images/glm53-flash-exl3/Dockerfile

@@ -0,0 +1,8 @@
+# syntax=docker/dockerfile:1.7
+# GLM-5.3-Flash EXL3 SM121: upstream Entrpi image + instanttensor loader.
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
+
+RUN set -eux; \
+    pip install --no-deps --no-build-isolation instanttensor==0.1.9; \
+    python3 -c "import instanttensor; print('instanttensor ok')"

docker-images/glm53-flash-exl3/build.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Build the SM121 GLM-5.3-Flash NVFP4 (lab quant) image: day-0 base +
+# NoPE-MLA mod + FujitsuPolycom lab-checkpoint patches.
+#
+# Generic driver: forwards every KEY=VAL pin in versions.env as a --build-arg
+# to the Dockerfile. TAG is set by the sm121 launcher to the recipe's
+# docker_image (manual build: TAG=glm53-flash-nvfp4-lab:local bash build.sh).
+set -euo pipefail
+cd "$(dirname "$0")"
+TAG="${TAG:-glm53-flash-exl3:local}"
+
+ARGS=()
+while IFS= read -r line; do
+    case "$line" in
+        ''|\#*) continue ;;
+    esac
+    key="${line%%=*}"; val="${line#*=}"
+    val="${val%\"}"; val="${val#\"}"   # versions.env values may be shell-quoted
+    [ -n "$key" ] && ARGS+=(--build-arg "${key}=${val}")
+done < versions.env
+
+echo "=== Building ${TAG} ==="
+DOCKER_BUILDKIT=1 docker build "${ARGS[@]}" --progress=plain -t "${TAG}" -f Dockerfile . 2>&1
+echo "=== Done: ${TAG} ==="

docker-images/glm53-flash-exl3/versions.env

@@ -0,0 +1,4 @@
+# Authoritative build inputs for the GLM-5.3-Flash EXL3 SM121 image:
+# the Entrpi community GB10 image (digest-pinned) + the instanttensor
+# loader (pip, no-deps so NCCL wheel resolution is untouched).
+BASE_IMAGE="ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2@sha256:284142c5833cbfd540ad42bb8f32cb340451db05a3b84029eaebae54579e9135"

recipes/glm53-flash-exl3/recipe.toml

@@ -32,7 +32,18 @@
 #     JIT caches persist per node (see extra_mounts).
 #
 # =====================================================================
-# STATUS 2026-08-30: boots fully to weight-stream, then global-OOM kills
+# STATUS 2026-08-30 (2nd entry): FIXED and verified serving on ds4.
+# ROOT CAUSE of the three OOM kills: the stock loader balloons ~80 GiB of
+# CPU-side anon buffers while streaming the 164 GiB checkpoint from the
+# NAS — unified memory (GPU pool + loader anon) hits the 121.6 GiB wall.
+# FIX: --load-format instanttensor (overlay image, this build context adds
+# instanttensor==0.1.9 --no-deps to the pinned Entrpi image): streams at
+# ~175 MB/s with tiny buffers; boot ~20 min end-to-end, no OOM (third+
+# boot clean at 16:40-17:00). Serving: ~22.8 tok/s with DFlash2 k=7
+# (accept 48-81% by content), math smoke exact. Recipe now builds its own
+# glm53-flash-exl3:local FROM the digest-pinned upstream image.
+# ---- first-entry diagnosis (kept for context) ----
+# boots fully to weight-stream, then global-OOM kills
 # the worker at 75-95% of shard load — REPRODUCED TWICE (12:22:25 and
 # 13:12:24, dmesg oom-killer) while streaming the 164 GiB checkpoint from
 # the shared NAS on BOTH nodes. GB10 unified memory + NFS page cache
@@ -59,7 +70,8 @@
 
 engine = "vllm"
 name = "GLM-5.3-Flash EXL3-4bpw + DFlash2 (Entrpi image, 2x Spark, TP=2)"
-docker_image = "ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2"
+docker_image = "glm53-flash-exl3:local"
+docker_image_dir = "glm53-flash-exl3"
 model = "brandonmusic/GLM-5.3-Flash-tr3-4bpw"
 required_nodes = 2
 tensor_parallel_size = 2
@@ -78,6 +90,11 @@ args = [
     "--trust-remote-code",
     # EXL3 fused trellis MoE (fork-baked kernels)
     "--quantization", "exl3",
+    # instanttensor loader (installed by the overlay): streams shards with
+    # tiny CPU buffers — the stock loader OOM-kills this box by ballooning
+    # ~80 GiB anon while streaming 164 GiB from the NAS (3 kernel kills,
+    # 2026-08-30). Also drops load time from ~13 min to ~1.
+    "--load-format", "instanttensor",
     "--distributed-executor-backend", "mp",
     # Their validated defaults; 0.88+ risks unified-memory swap on GB10
     # 0.85 upstream default left 1.14 GiB when the warmup wanted 1.12 GiB
@@ -95,12 +112,11 @@ args = [
     "--kv-cache-memory", "12400000000",
     # Drafter by hub id (resolved from the mounted NAS HF_HOME like the
     # target model; 7 = trained block size 8 minus the bonus token).
-    # Upstream uses `incoai/GLM-5.3-Flash-DFlash2` (raw fp8-e4m3 drafter);
-    # we serve the MXFP8-compressed qualified mirror instead — already in
-    # the NAS hub, and this build executes serialized MXFP8 projections
-    # natively. Falls forward to the raw drafter if quality differs.
+    # incoai/GLM-5.3-Flash-DFlash2 = the exact upstream drafter (raw
+    # fp8-e4m3, CC BY-NC-ND 4.0), downloaded to the NAS hub 2026-08-30 at
+    # the operator's request; the MXFP8 mirror is the fallback.
     "--speculative-config", """
-'{"method":"dflash","model":"local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8","num_speculative_tokens":7}'""",
+'{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7}'""",
     # required at 524288 on GB10 unified memory (upstream default SKIP_MM_PROFILING=1)
     "--skip-mm-profiling",
     "--tool-call-parser", "glm47",

specs/tech-architecture/glm53-recipe-state.md

@@ -175,7 +175,14 @@ Report: /tmp/runs/2026/08/2026-08-30T09-41-57.562243Z_108a798b.md.
 ## EXL3 lane (Entrpi community image) — boot diagnosis (2026-08-30)
 
 ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2 + brandonmusic
-GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. Fixed in-recipe: NGC
+GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. VERIFIED SERVING
+2026-08-30 evening: the loader-side OOM chain (`--load-format instanttensor`
+overlay) resolved — stock loader balloons ~80 GiB CPU-anon streaming the
+checkpoint; instanttensor streams ~175 MB/s with tiny buffers. ~22.8 tok/s
+with DFlash2 k=7 (accept 48-81% by content); math smoke exact. DFlash2
+drafter switched to the upstream `incoai/GLM-5.3-Flash-DFlash2` (downloaded
+to NAS at operator request); MTP3-language-only + v84 image lanes remain
+SM120-only upstream. Fixed in-recipe: NGC
 entrypoint (needs explicit `vllm serve`), drafter swap to the MXFP8
 mirror (incoai raw drafter absent from the NAS hub), GMU 0.86. BLOCKED:
 global OOM kills the worker at 75-95% of shard load on BOTH streamed-from-