spark lanes: r35 + mxfp8 kernel fallback; r38 pool-deadlock bug report

- glm53-flash-nvfp4-spark (2x): docker_image glm53-jovian-r35 +
  VLLM_DISABLED_KERNELS=FlashInferCutlassMxfp8LinearKernel — boots and
  serves the Spark checkpoint: 162/176 (92/100), ~40 tok/s decode, KV
  1.41M @524K. r38 (JJ tip + b12x master) deadlocks in the b12x
  preparation pool ('waiting for ranks: 0/1821', 0 compilations, no
  compilers spawned, both ranks busy-wait) with RoCE AR on and off.
- 4x recipe: same flag set + r38 profile (r38 image built, label
  5b8f1513) — 4x boot untested (2x first; r38 on 4x would test the
  rank-count hypothesis from the bug report).
- r38 build profile (JJ tip 8e1f1e587f8 / b12x master 0f3a8cbfd1c) +
  Dockerfile conditional for the b12x pinned-wc patch (no-op on the
  new loader) — the build itself works (dgx1, ~15 min with warm cache).
- bug report + registry: BUG-2026-09-19 r38 b12x preparation pool
  deadlock.
- bench report 2026-09-19_184648_r35-spark-2x.{md,json,log,recipe,manifest}

docker-images/glm53-jovian-r27/Dockerfile

@@ -251,13 +251,24 @@ RUN --mount=type=bind,from=vllm-builder,source=/wheels,target=/vllm-wheels \
 # in b12x/integration/vllm/loader.py — two occurrences (the weight_pool call
 # and the matching audit label). Reported prefill 1350 -> ~2000 tok/s. No
 # recompiles; hard-gated: any pinned_wc left behind fails the build.
+#
+# Conditional since the r38 cut (2026-09-19): b12x master 0f3a8cbf replaced
+# the hardcoded allocation with weight_allocation() in b12x/loader/_pool.py
+# (auto-selects "managed" on GB10: pageable_memory_access + host_page_tables
+# caps), so the legacy pinned_wc strings are gone and the sed is a no-op.
+# Legacy pins (r27..r37 b12x) still carry the pattern and take the gated path.
 RUN set -eux; \
     F=/usr/local/lib/python3.12/dist-packages/b12x/integration/vllm/loader.py; \
-    sed -i 's/allocation="pinned_wc"/allocation="managed"/; \
-            s/"allocation": "pinned_wc"/"allocation": "managed"/' "$F"; \
-    test "$(grep -c 'allocation="managed"' "$F")" = "1"; \
-    grep -Fq '"allocation": "managed"' "$F"; \
-    ! grep -q 'pinned_wc' "$F"
+    if grep -q 'pinned_wc' "$F" 2>/dev/null; then \
+      sed -i 's/allocation="pinned_wc"/allocation="managed"/; \
+              s/"allocation": "pinned_wc"/"allocation": "managed"/' "$F"; \
+      test "$(grep -c 'allocation="managed"' "$F")" = "1"; \
+      grep -Fq '"allocation": "managed"' "$F"; \
+      ! grep -q 'pinned_wc' "$F"; \
+    else \
+      echo "b12x loader carries no pinned_wc (new auto-allocation b12x) — patch skipped"; \
+      test -f "$F"; \
+    fi
 # Build gate: the r27 stack must import with the GB10 patches in place and
 # carry the qualified package trees (marker spot-checks from the source lock).
 RUN set -eux; \

docker-images/glm53-jovian-r27/profiles/jovian-r38-glm53.env

@@ -0,0 +1,52 @@
+# Version profile: GLM-5.3-Flash r38 (jovian-judgement tip, 2026-09-19 cut).
+#
+# Operator direction (2026-09-19): use the LATEST jovian-judgement + the
+# current lab quants, not the r12 legacy rebuild. This profile pins the lab
+# mainline at the day-19 tip instead of the r37 DS4.1-optimized release.
+#
+# What this is (verified 2026-09-19 against the public repos):
+#   vLLM  8e1f1e587f8 = local-inference-lab/vllm `dev/jovian-judgement` tip
+#         (2026-09-16, "Fix deferred grammar rejection in v2 speculative
+#         sampling"). 25+ commits over the r37 pin: B12X MXFP8 linear kernel
+#         (mxfp8/b12x.py — the small-GDN-GEMM path that flashinfer mm_mxfp8
+#         rejects at n/k < 128 on the Spark quant), per-projection layer-max
+#         activation scales for b12x NVFP4 MoE, GLM B12X startup contracts
+#         for target+draft (#772), sparse-MLA module owner (#771), KV
+#         admission fix (#770), KDA __init__ override for native-FP8
+#         checkpoints, RoCE/b12x tuning. Model code keeps the SPLIT GDN
+#         naming (q_proj/k_proj/g_a/g_b/f_a/f_b/b_proj) that both current
+#         lab quants (GLM-5.3-Flash-NVFP4 ~200 GB, -NVFP4-Spark 154 GB)
+#         carry — the fused in_proj_qkvbfg naming is the Aiden fork only.
+#   B12X  0f3a8cbfd1c = local-inference-lab/b12x `master` tip.
+#
+# Delta vs the r37 profile (jovian-r37-ds41.env):
+#   vLLM  c687594b8a8 (release/ds41-optimized-r37) -> 8e1f1e587f8 (JJ tip)
+#   B12X  f1c4e9dd5b1 -> 0f3a8cbfd1c
+#   LMCache 29bc5a2e — UNCHANGED
+#   FlashKDA 3b225bf2 — UNCHANGED (JJ tip cmake gate verified 2026-09-19)
+#   DeepGEMM 8b1392b9 (nv_dev) — UNCHANGED
+#   requirements/cuda.txt (JJ tip) — torch 2.13.0, flashinfer 0.6.17:
+#     SAME as our r27/r35 GB10 pins. cmake/external_projects pins (cutlass
+#     v4.4.2, flashkda 3b225bf, flashmla a8f794d, MSA 087c161, qutlass
+#     e74319e, tml-fa4 b206834, flash-attention f3e1a4f, triton v3.5.1,
+#     deepgemm 8b1392b9) verified identical to the r37 tree's — fetch-deps.sh
+#     needs no resync.
+#
+# Build: clone-at-pin (docker-images/glm53-jovian-r27 Dockerfile), in-build
+# git only (no pre-staged tarballs — operator direction 2026-09-19).
+VLLM_PIN=8e1f1e587f8d24faf606f334a1c4bdaaa6bd4368
+VLLM_TREE=d9f35de1ed5e47dd006f6d420dd25707fb284eae
+VLLM_PRETEND_VERSION=0.26.1rc0+glm53.r38.vllm8e1f1e58
+
+B12X_PIN=0f3a8cbfd1c11d27f04e3ab37a802d522f4f1c68
+B12X_TREE=ff2218d710ca9cae71459e1b9eb7ca3fe10ae08e
+B12X_PRETEND_VERSION=0.1.dev0+glm53b12x.r38.0f3a8cbf
+
+LMCACHE_PIN=29bc5a2efde737c436b04499eb62cd1776cebeec
+LMCACHE_TREE=5a88a1ea9d2627c76288d056e7193f7454669b64
+LMCACHE_VERSION=0.5.5.dev0+glm53checkpoints.29bc5a2e
+
+# Same r35/r36/r37 behavior: empty on purpose — the tree gates FlashKDA on
+# the packed-checkpoint API and applies its own in-tree patch (same pin
+# 3b225bf2; our unpatched checkout would fail the gate).
+FLASH_KDA_SRC_DIR=

recipes/glm53-flash-nvfp4-spark/recipe.toml

@@ -1,93 +1,48 @@
-# GLM-5.3-Flash NVFP4 — local-inference-lab quant (vLLM) — 2× DGX Spark, TP=2.
+# GLM-5.3-Flash NVFP4-Spark — 4x DGX Spark, the lab's current Spark quant on
+# the LATEST jovian-judgement (r38 cut, 2026-09-19).
 #
-# local-inference-lab mixed quant: NVFP4 routed experts (layers 3–44,
-# group-16) + MXFP8 MTP experts (layer 45) + W4A4 input scales. Best agent
-# quant so far: tool-eval-bench 89/100 (vs 85 LibertAIDAI, 90 official FP8).
-# Model: https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4
-#        (~186 GB → 92.72 GiB weights/node; KV pinned at 10 GiB, see below)
+# Operator direction (2026-09-19): after the NAS disk death the interim
+# serving mode is 2x + 4x on local NVMe with the current lab quants. This
+# lane uses the latest lab engine (r38 = dev/jovian-judgement tip 8e1f1e5,
+# see profiles/jovian-r38-glm53.env) instead of the legacy r12 rebuild
+# (the r12 image is gone from the nodes; the r12 composition is a "phantom"
+# — the lab mainline keeps the split GDN naming the current quants carry,
+# and the r38 tip adds the B12X MXFP8 linear kernel that the r35-era
+# flashinfer-only path lacked for the small GDN GEMMs).
 #
-# Image: glm53-flash-nvfp4-lab:local (docker-images/glm53-flash-nvfp4-lab/ —
-# day-0 arm64 image + NoPE-MLA mod + the four FujitsuPolycom lab-checkpoint
-# patches: model naming shim, modelopt MTP namespace fix, persistent_topk
-# CC-12.x guards). Their launch config is mirrored below.
-# PROVENANCE: https://github.com/FujitsuPolycom/glm53-flash-tp2-spark
-#             (Apache-2.0; see patches/UPSTREAM.md) + forum 381350 #41.
+# Model: local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (154.1 GiB, snapshot
+# a6082410, 09-17) — the "DSpark speculative embedded" checkpoint: the MTP
+# draft head lives in the model-mtp-*.safetensors shards (no separate
+# drafter download). NVFP4 routed experts + NVFP4 MTP expert layer. Split
+# GDN naming (q/k/g_a/g_b/f_a/f_b/b_proj) — loads on the JJ fork lineage
+# (r12..r38); the Aiden fork's fused in_proj_qkvbfg naming KeyErrors on it.
 #
-# Deviations from their launch script (intentional):
-#   * no --kda-prefill-backend flashkda (needs their base-image patch — skipped)
-#   * --kv-cache-dtype fp8_ds_mla instead of `fp8` (same packed layout; this
-#     spelling is what the NoPE-MLA mod booted with on the LibertAIDAI stack)
-#   * --language-model-only + no mm-renderer patch/env (text-only)
-#   * no bundled NCCL/LD_PRELOAD, no NCCL_* fabric block (our switched fabric
-#     is configured via [nccl.ds4] + launcher GID auto-detect)
-#   * no --moe-backend marlin: their proven stack uses the default backend for
-#     THIS quant and reports clean evals. Sanity-check output for token loops
-#     anyway (the silent FP4 MoE corruption gotcha) before trusting a new node.
-#   * flashinfer autotune disabled via --kernel-config (their proven boot);
-#     VLLM_ENGINE_READY_TIMEOUT_S stays high regardless.
-#
-# Boot success markers (engine log, in order):
-#   "Model loading took 92.72 GiB" → "[quantprobe] ... algo=MXFP8" (MTP fix
-#   active; algo=None means broken) → "GPU KV cache size: 1,164,369 tokens" →
-#   "Application startup complete". ~700 s sequential weight load.
-#
-# Test request: chat completions need "chat_template_kwargs":{"reasoning_effort"
-# :"low"} — omitted/other values select MAX reasoning effort (empty content on
-# short max_tokens).
-#
-# Optional: instanttensor loader (~30 s weight load vs ~700 s):
-#   FROM glm53-flash-nvfp4-lab:local + pip install --no-deps --no-build-isolation
-#   instanttensor==0.1.9, then add --load-format instanttensor (--no-deps is
-#   required: resolution would downgrade the image's NCCL).
-#
-# PREREQUISITES: weights cached on both nodes:
-# [BISECT A2 2026-09-03] EXACT 90/100 recipe (236ce5a) + instanttensor + JIT
-# mounts, on the CURRENT eugr-patched image. Isolates image lineage from the
-# execution-env/flag cluster (AOT/V2/b12x/card-env). Comparator: 124/138.
-# GLM-5.3-Flash NVFP4-Spark — local-inference-lab quant with the speculative
-# layers embedded — 2× DGX Spark, TP=2.
-#
-# local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (rev 06972cdee7280cec849aedf05a02062b3e870246):
-# NVFP4 routed experts (layers 3+, group-16) + NVFP4 MTP expert layer in the
-# separate model-mtp-0000x.safetensors shards (mtp_tokens: 5 per their
-# lil.yaml), amax/nonexpert/inputscales side shards, Glm5NextForConditionalGeneration.
-# THE "DSpark speculative embedded" checkpoint — no separate DFlash2 download
-# needed; lil.yaml: default_speculator mtp, dflash2_tokens 7 (r12 doc fr: their
-# standalone-Q: MTP:3 254 tok/s CC1, 333 Sieve, 94 steps/s).
-#
-# Image: glm53-flash-nvfp4-lab:local (same proven day-0 arm64 stack + NoPE-MLA
-# mod + modelopt MTP namespace patch used by the 92/100 lab lane). The MTP
-# layer here is NVFP4 (not MXFP8) — the quantprobe marker may print
-# algo=NVFP4/- different; algo=None = patch missed (fall back to 4p67 image).
-#
-# Deviations from lab (unchanged mechanics): same fabric/env/GMU/KV pinning.
-# lil.yaml launch mirrors: block 256, disable_flashinfer_autotune, mm data
-# mode — our text-only flags identical to glm53-flash-nvfp4-lab.
-#
-# Boot markers: "Model loading took ~XX GiB" → "[quantprobe] ..." →
-# "GPU KV cache size" → "Application startup complete".
+# Image: glm53-jovian-r38:local — the r27-lineage clone-at-pin build at the
+# r38 pins (normal in-build git, no pre-staged tarballs). Flags mirror the
+# proven r27-lineage GLM recipe (glm53-flash-r27-4x-dcp4, 160/176 bench)
+# minus the DCP4 pair:
+#   * --attention-backend B12X — required: the sm12x auto-priority list never
+#     offers the B12X sparse-MLA backend for GLM's head-512
+#   * no --kv-cache-dtype — let the fork pick its default (explicit fp8 /
+#     fp8_ds_mla both failed auto-selection on this build lineage)
+#   * --load-format instanttensor — baked into the r27-lineage images
+#   * no --moe-backend — the b12x MoE routing is the fork default for this
+#     quant (the FINAL BISECT "+moe b12x" gain is in the default path)
+#   * speculative-config carries attention_backend B12X — the draft model's
+#     vllm_config takes its attention backend from the SPECULATIVE config
+#   * --language-model-only, no --enforce-eager (r27-lineage: graphs+MTP is
+#     clean on the r17+ refs; the day-0 graph bug does not apply)
 #
 # PREREQUISITES: weights on each node's LOCAL hub (the NAS disk died
-# 2026-09-19 — /home/kilork/hf-local, ~/.cache/huggingface symlink re-pointed
-# per node): hf download local-inference-lab/GLM-5.3-Flash-NVFP4-Spark
+# 2026-09-19): /home/kilork/hf-local, ~/.cache/huggingface symlink
+# re-pointed per node (snapshot a6082410).
 
 engine = "vllm"
-name = "GLM-5.3-Flash NVFP4-Spark (vLLM, 2x Spark, MTP-5)"
-# Image: glm53-jovian-r35:local — our r35 anchor image for the GLM-5.3-Flash
-# NVFP4 family (the 433 GB 4x lane ran on it at 163/176 with MTP3 — the
-# Spark checkpoint is the same Glm5Next family, different quant variant).
-# The lab's newer karmic-kraken-beta (rtx6kpro spark-tp2 doc) is amd64-only
-# (their RTX 6000 x86 platform; "Exec format error" on our aarch64 nodes),
-# so it cannot serve the Spark checkpoint here; the r12 image (the recipe's
-# original) is gone from the repo as non-rebuildable (commit 17a91a9).
-# Aiden's production GLM-5.3-Flash image for GB10 (31.2 GB; the glm53-flash-aiden
-# lane ran on it at 100/100 agentic — the GDN+NVFP4 path routes through B12X,
-# which handles the Spark checkpoint's small KDA GEMMs that r35's flashinfer
-# mm_mxfp8 (n,k >= 128) rejects. Prebuilt external image — no image_dir.
-docker_image = "aidendle94/sparkrun-vllm-glm53-gb10:production-1.2"
-# Required: the clean image follows the lab contract (ENTRYPOINT []
-# + CMD bash), so the recipe carries the entrypoint.
-entrypoint = []
+name = "GLM-5.3-Flash NVFP4-Spark (vLLM, 2x Spark, MTP-3, r38 tip)"
+docker_image = "glm53-jovian-r35:local"  # r35 loads the Spark checkpoint; r38 (JJ tip + b12x master) deadlocks in the b12x preparation pool on 2x GB10
+docker_image_dir = "glm53-jovian-r27"
+build_profile = "glm53-r35"
+entrypoint = ["vllm", "serve"]
 model = "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark"
 parent_model = "zai-org/GLM-5.3-Flash"
 required_nodes = 2
@@ -101,67 +56,34 @@ args = [
     "--served-model-name", "glm53",
     "--trust-remote-code",
     "--distributed-executor-backend", "mp",
-    # GMU is ONLY a startup guardrail here (KV pool is pinned below): 0.85
-    # is the GB10-qualified value (0.90+ demands >109.46 GiB free and the
-    # GB10 baseline reservation caps measured free at ~108.78 GiB — the
-    # old "54-second death" class of boot failures). 0.85 = 103.4 GiB,
-    # below any realistic dip; does NOT change what gets allocated
-    # (weights ~71.5 GiB + pinned KV 10 + activations).
-    "--gpu-memory-utilization", "0.85",
-    # Pinned 10 GiB KV (≈1.16M tokens) — GMU sizing has no headroom at
-    # 92.72 GiB weights/node; pinning also keeps warmup off the swap path
+    # GMU is ONLY a startup guardrail here (KV pool is pinned below):
+    # 0.88 = 107.0 GiB, below any realistic free-memory dip on GB10's
+    # 121.63 GiB (the 0.90 admission flake, #48140).
+    "--gpu-memory-utilization", "0.88",
+    # 10 GiB KV (2x: 77 GB weights/node — the proven day-0/r27 pin; the
+    # 524288 context ≈ 1.16M-token pool cap)
     "--kv-cache-memory-bytes", "10737418240",
-    # No --kv-cache-dtype: let the r12 fork pick its default (the explicit
-    # fp8 / fp8_ds_mla both failed auto-selection on this build).
-    # The GLM5Next model wires the B12X sparse-MLA backend (sm_120/sm_121)
-    # when the global backend is B12X (glm5next/nvidia/attention.py:
-    # _select_sparse_backend); the sm12x auto-priority list never offers it
-    # (FLASHINFER_MLA_SPARSE_SM120 only supports head 576, not GLM's 512).
-    "--attention-backend", "B12X",
-    # pure I/O loader (kept across every config; the model is on LOCAL NVMe
-    # now (NAS disk died 2026-09-19) — instanttensor off NVMe is fast)
-    "--load-format", "instanttensor",
-    # [R17 FULL-CLUSTER TEST] lil manifest backends (quality gate: hardmode
-    # must stay ≥88; r12-era this cluster dropped it to 74 — r17 may carry
-    # the fix through — one boot decides).
     "--block-size", "256",
-    # FINAL BISECT (2026-09-03, hardmode 176pt, r17 img):
-    #   knobs-only 87: d0 c1 24.1, d8192 23.8
-    #   +moe b12x  88: d0 c1 23.5, d8192 26.6  <- SHIP (2 pts d8192)
-    #   +linear b12x 44: W4A4 linear kernel destroys numerics (param-precision 0%)
-    #   +align     88: but d0 c1 18.6 (-5) — perf-negative, dropped
-    # (all three together = 44-74; the lab needs its private packed package)
-    "--moe-backend", "b12x",
     "--max-model-len", "524288",
     "--max-num-seqs", "16",
     "--max-num-batched-tokens", "4096",
     "--enable-chunked-prefill",
     "--async-scheduling",
     "--enable-prefix-caching",
-    "--disable-custom-all-reduce",
+    # B12X sparse-MLA (required for head-512 on sm_12x)
+    "--attention-backend", "B12X",
+    # pure I/O loader (baked into the r27-lineage images; ~5.5 min for the
+    # full checkpoint off shared NFS, ~1 min off local NVMe)
+    "--load-format", "instanttensor",
     # Their proven boot: no flashinfer autotune, no CuTe-DSL warmup
     "--kernel-config", """
 '{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
-    # Built-in MTP draft, 3 tokens (the r12-qualified "mtp:3" mode — their
-    # x86 numbers: 254 tok/s CC1, 2.37 accept; the checkpoint embeds up to
-    # 5 via the model-mtp-0000x shards but 3 is the verified depth). This is
-    # the checkpoint's own MTP head (Glm5NextMTPModel), NOT dspark — dspark
-    # is the DeepSeek-V4 family's mechanism; GLM's other r12 mode is
-    # dflash2:7 with a separate trained draft checkpoint.
-    # The draft model's vllm_config takes its attention backend from the
-    # SPECULATIVE config, not the global flag (eagle/utils.py:
-    # _make_eagle_draft_vllm_config) — B12X here mirrors the main model's.
+    # Built-in MTP draft, 3 tokens (the r12-qualified "mtp:3" mode). The
+    # draft model's vllm_config takes its attention backend from the
+    # SPECULATIVE config, not the global flag — B12X mirrors the main model.
     "--speculative-config", """
 '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"B12X"}'""",
     "--language-model-only",
-    # MTP + CUDA graphs degrades OUTPUT QUALITY on the glm53 day-0 dev build
-    # (forum #381350/#44, CONFIRMED by A/B bench 2026-08-28: same seed/flags,
-    # graphs on = 89/100, eager = 92/100; decode unchanged 28.4 t/s @ c1,
-    # c2 -7%, TTFT +1.6s). The b12x glm52 stack does NOT need this (its
-    # graphs+MTP combo benches clean at 100/100) — bug is build-specific.
-    # graphs moved BEFORE tool-call-parser; enforce-eager removed for the
-    # r17-graphs test (bisect A2 was 88/100 eager; graphs+MTP fix is public
-    # in r17 refs? verify by bench).
     "--tool-call-parser", "glm47",
     "--enable-auto-tool-choice",
     "--reasoning-parser", "glm45",
@@ -169,12 +91,16 @@ args = [
 
 [env]
 HF_HUB_OFFLINE = "1"
-# FlashInfer JIT (mxfp8_gemm_cutlass_sm120, ~12 kernels) compiles at first
-# boot with 12 parallel nvcc jobs -> OOM-killed (code 137) on GB10's
-# 121 GB unified while the 93 GB weights + 10 GB KV pin are resident.
-# MAX_JOBS caps ninja parallelism so the JIT fits in the remaining ~20 GB.
+# RoCEnante (b12x one-shot RoCE all-reduce/all-gather) — default on for the
+# r27 lanes since the 2026-09-08 A/B (4x: decode +31..+90%, quality 160 vs
+# 155). Needs the RoCE fabric; single-node serves skip it silently.
+VLLM_ENABLE_ROCE_ALLREDUCE = "1"
+# Route mxfp8 linears away from flashinfer cutlass (rejects small GDN GEMMs: min n/k=128)
+VLLM_DISABLED_KERNELS = "FlashInferCutlassMxfp8LinearKernel"
+# FlashInfer JIT compiles at first boot with N parallel nvcc jobs ->
+# OOM-killed on GB10 while weights+KV are resident; cap the ninja jobs.
 MAX_JOBS = "2"
-# NoPE-MLA rope-pad runtime gate (mod baked into the image)
+# NoPE-MLA rope-pad runtime gate (mod baked into the image lineage)
 VLLM_MLA_NOPE_PAD_ROPE = "1"
 VLLM_ENGINE_READY_TIMEOUT_S = "3600"
 # Upstream launch-script env (memory + compile knobs)
@@ -193,6 +119,20 @@ CUTE_DSL_ARCH = "sm_121a"
 CMAKE_CUDA_ARCHITECTURES = "121"
 OMP_NUM_THREADS = "2"
 
+# JIT caches persist per-node across serves (cold /root/.cache per container
+# = ~20 min flashinfer/torch recompile every serve)
 [[extra_mounts]]
 host = "~/sm121-cache/glm53-jit-121f/b12x"
 container = "/root/.cache/b12x"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/flashinfer"
+container = "/root/.cache/flashinfer"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/torchinductor"
+container = "/root/.cache/torchinductor"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/vllm"
+container = "/root/.cache/vllm"

recipes/glm53-flash-nvfp4-spark/reports/2026-09-19_183806_r35-spark-2x.log

@@ -0,0 +1,42 @@
+^D{"event": "backend_detected", "backend": "vllm"}
+
+╭──────────────────── ⚡ llama-benchy Throughput Benchmark ────────────────────╮
+│ glm53                                                                        │
+│ pp=[2048]  tg=[128]  depth=[0, 4096, 8192]  concurrency=[1, 2]  runs=3       │
+│ latency=generation                                                           │
+╰──────────────────────────────────────────────────────────────────────────────╯
+
+  ✓ Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18/18 0:03:46
+
+  llama-benchy 0.4.0
+  Estimated latency: 383.3 ms
+
+                               llama-benchy Results                             
+┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━
+┃                      ┃    ┃         ┃         ┃     TTFT ┃    Total ┃         
+┃ Test                 ┃ c  ┃  pp t/s ┃  tg t/s ┃     (ms) ┃     (ms) ┃   Tokens
+┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━
+│ pp2048 tg128 @ d0    │ c1 │   1,190 │    26.4 │    2,072 │    6,537 │ 2048+128
+│ pp2048 tg128 @ d0    │ c2 │   1,317 │    33.5 │    2,717 │    9,686 │ 2048+128
+│ pp2048 tg128 @ d4096 │ c1 │   1,366 │    27.8 │    4,354 │    8,581 │ 2048+128
+│ pp2048 tg128 @ d4096 │ c2 │   1,321 │    29.8 │    7,481 │   14,648 │ 2048+128
+│ pp2048 tg128 @ d8192 │ c1 │   1,371 │    27.9 │    6,952 │   11,155 │ 2048+128
+│ pp2048 tg128 @ d8192 │ c2 │   1,373 │    28.1 │   12,349 │   19,937 │ 2048+128
+└──────────────────────┴────┴─────────┴─────────┴──────────┴──────────┴─────────
+
+  ℹ Metrics sourced from llama-benchy — see https://github.com/eugr/llama-benchy
+for methodology.
+
+{"event": "scenario_start", "scenario_id": "TC-01", "title": "Direct Specialist Match", "category": "A", "index": 0, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-01", "status": "pass", "points": 2, "index": 0, "total": 88, "duration_seconds": 7.61}
+{"event": "scenario_start", "scenario_id": "TC-02", "title": "Distractor Resistance", "category": "A", "index": 1, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-02", "status": "pass", "points": 2, "index": 1, "total": 88, "duration_seconds": 7.27}
+{"event": "scenario_start", "scenario_id": "TC-03", "title": "Implicit Tool Need", "category": "A", "index": 2, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-03", "status": "pass", "points": 2, "index": 2, "total": 88, "duration_seconds": 13.54}
+{"event": "scenario_start", "scenario_id": "TC-04", "title": "Unit Handling", "category": "B", "index": 3, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-04", "status": "pass", "points": 2, "index": 3, "total": 88, "duration_seconds": 3.66}
+{"event": "scenario_start", "scenario_id": "TC-05", "title": "Date and Time Parsing", "category": "B", "index": 4, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-05", "status": "pass", "points": 2, "index": 4, "total": 88, "duration_seconds": 19.87}
+{"event": "scenario_start", "scenario_id": "TC-06", "title": "Multi-Value Extraction", "category": "B", "index": 5, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-06", "status": "pass", "points": 2, "index": 5, "total": 88, "duration_seconds": 7.21}
+{"event": "scenario_start", "scenario_id": "TC-07", "title": "Search \u2192 Read \u2192 Act", "category": "C", "index": 6, "total": 88}

recipes/glm53-flash-nvfp4-spark/reports/2026-09-19_184648_r35-spark-2x.log

@@ -0,0 +1,206 @@
+^D{"event": "backend_detected", "backend": "vllm"}
+
+╭──────────────────── ⚡ llama-benchy Throughput Benchmark ────────────────────╮
+│ glm53                                                                        │
+│ pp=[2048]  tg=[128]  depth=[0, 4096, 8192]  concurrency=[1, 2]  runs=3       │
+│ latency=generation                                                           │
+╰──────────────────────────────────────────────────────────────────────────────╯
+
+  ✓ Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18/18 0:02:54
+
+  llama-benchy 0.4.0
+  Estimated latency: 90.5 ms
+
+                               llama-benchy Results                             
+┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━
+┃                      ┃    ┃         ┃         ┃     TTFT ┃    Total ┃         
+┃ Test                 ┃ c  ┃  pp t/s ┃  tg t/s ┃     (ms) ┃     (ms) ┃   Tokens
+┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━
+│ pp2048 tg128 @ d0    │ c1 │   1,332 │    43.3 │    1,492 │    4,361 │ 2048+128
+│ pp2048 tg128 @ d0    │ c2 │   1,207 │    42.8 │    2,269 │    7,203 │ 2048+128
+│ pp2048 tg128 @ d4096 │ c1 │   1,421 │    36.0 │    3,935 │    7,403 │ 2048+128
+│ pp2048 tg128 @ d4096 │ c2 │   1,388 │    43.1 │    7,073 │   11,837 │ 2048+128
+│ pp2048 tg128 @ d8192 │ c1 │   1,456 │    42.2 │    6,397 │    9,342 │ 2048+128
+│ pp2048 tg128 @ d8192 │ c2 │   1,432 │    38.2 │   11,478 │   16,466 │ 2048+128
+└──────────────────────┴────┴─────────┴─────────┴──────────┴──────────┴─────────
+
+  ℹ Metrics sourced from llama-benchy — see https://github.com/eugr/llama-benchy
+for methodology.
+
+{"event": "scenario_start", "scenario_id": "TC-01", "title": "Direct Specialist Match", "category": "A", "index": 0, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-01", "status": "pass", "points": 2, "index": 0, "total": 88, "duration_seconds": 5.59}
+{"event": "scenario_start", "scenario_id": "TC-02", "title": "Distractor Resistance", "category": "A", "index": 1, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-02", "status": "pass", "points": 2, "index": 1, "total": 88, "duration_seconds": 5.02}
+{"event": "scenario_start", "scenario_id": "TC-03", "title": "Implicit Tool Need", "category": "A", "index": 2, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-03", "status": "pass", "points": 2, "index": 2, "total": 88, "duration_seconds": 8.22}
+{"event": "scenario_start", "scenario_id": "TC-04", "title": "Unit Handling", "category": "B", "index": 3, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-04", "status": "pass", "points": 2, "index": 3, "total": 88, "duration_seconds": 3.1}
+{"event": "scenario_start", "scenario_id": "TC-05", "title": "Date and Time Parsing", "category": "B", "index": 4, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-05", "status": "pass", "points": 2, "index": 4, "total": 88, "duration_seconds": 13.14}
+{"event": "scenario_start", "scenario_id": "TC-06", "title": "Multi-Value Extraction", "category": "B", "index": 5, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-06", "status": "pass", "points": 2, "index": 5, "total": 88, "duration_seconds": 4.95}
+{"event": "scenario_start", "scenario_id": "TC-07", "title": "Search \u2192 Read \u2192 Act", "category": "C", "index": 6, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-07", "status": "pass", "points": 2, "index": 6, "total": 88, "duration_seconds": 14.11}
+{"event": "scenario_start", "scenario_id": "TC-08", "title": "Conditional Branching", "category": "C", "index": 7, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-08", "status": "pass", "points": 2, "index": 7, "total": 88, "duration_seconds": 10.54}
+{"event": "scenario_start", "scenario_id": "TC-09", "title": "Parallel Independence", "category": "C", "index": 8, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-09", "status": "pass", "points": 2, "index": 8, "total": 88, "duration_seconds": 6.37}
+{"event": "scenario_start", "scenario_id": "TC-10", "title": "Trivial Knowledge", "category": "D", "index": 9, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-10", "status": "pass", "points": 2, "index": 9, "total": 88, "duration_seconds": 2.44}
+{"event": "scenario_start", "scenario_id": "TC-11", "title": "Simple Math", "category": "D", "index": 10, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-11", "status": "pass", "points": 2, "index": 10, "total": 88, "duration_seconds": 2.43}
+{"event": "scenario_start", "scenario_id": "TC-12", "title": "Impossible Request", "category": "D", "index": 11, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-12", "status": "pass", "points": 2, "index": 11, "total": 88, "duration_seconds": 11.72}
+{"event": "scenario_start", "scenario_id": "TC-13", "title": "Empty Results", "category": "E", "index": 12, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-13", "status": "pass", "points": 2, "index": 12, "total": 88, "duration_seconds": 10.3}
+{"event": "scenario_start", "scenario_id": "TC-14", "title": "Malformed Response", "category": "E", "index": 13, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-14", "status": "pass", "points": 2, "index": 13, "total": 88, "duration_seconds": 8.27}
+{"event": "scenario_start", "scenario_id": "TC-15", "title": "Conflicting Information", "category": "E", "index": 14, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-15", "status": "pass", "points": 2, "index": 14, "total": 88, "duration_seconds": 7.11}
+{"event": "scenario_start", "scenario_id": "TC-16", "title": "German Language Tool Call", "category": "F", "index": 15, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-16", "status": "pass", "points": 2, "index": 15, "total": 88, "duration_seconds": 7.27}
+{"event": "scenario_start", "scenario_id": "TC-17", "title": "Timezone-Aware Scheduling", "category": "F", "index": 16, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-17", "status": "pass", "points": 2, "index": 16, "total": 88, "duration_seconds": 7.02}
+{"event": "scenario_start", "scenario_id": "TC-18", "title": "Translate & Forward", "category": "F", "index": 17, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-18", "status": "pass", "points": 2, "index": 17, "total": 88, "duration_seconds": 8.96}
+{"event": "scenario_start", "scenario_id": "TC-19", "title": "Message Routing", "category": "G", "index": 18, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-19", "status": "pass", "points": 2, "index": 18, "total": 88, "duration_seconds": 4.39}
+{"event": "scenario_start", "scenario_id": "TC-20", "title": "Data Extraction & Calculation", "category": "G", "index": 19, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-20", "status": "pass", "points": 2, "index": 19, "total": 88, "duration_seconds": 12.54}
+{"event": "scenario_start", "scenario_id": "TC-21", "title": "Constraint Validation", "category": "G", "index": 20, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-21", "status": "pass", "points": 2, "index": 20, "total": 88, "duration_seconds": 11.95}
+{"event": "scenario_start", "scenario_id": "TC-22", "title": "Output Format Compliance", "category": "H", "index": 21, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-22", "status": "pass", "points": 2, "index": 21, "total": 88, "duration_seconds": 2.2}
+{"event": "scenario_start", "scenario_id": "TC-23", "title": "Explicit Tool Prohibition", "category": "H", "index": 22, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-23", "status": "pass", "points": 2, "index": 22, "total": 88, "duration_seconds": 13.29}
+{"event": "scenario_start", "scenario_id": "TC-24", "title": "Multi-Constraint Instruction", "category": "H", "index": 23, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-24", "status": "pass", "points": 2, "index": 23, "total": 88, "duration_seconds": 2.74}
+{"event": "scenario_start", "scenario_id": "TC-25", "title": "Cross-Reference Prior Results", "category": "I", "index": 24, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-25", "status": "pass", "points": 2, "index": 24, "total": 88, "duration_seconds": 12.8}
+{"event": "scenario_start", "scenario_id": "TC-26", "title": "State Consistency (Multi-Turn)", "category": "I", "index": 25, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-26", "status": "pass", "points": 2, "index": 25, "total": 88, "duration_seconds": 9.79}
+{"event": "scenario_start", "scenario_id": "TC-27", "title": "Deduplication Awareness", "category": "I", "index": 26, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-27", "status": "pass", "points": 2, "index": 26, "total": 88, "duration_seconds": 8.34}
+{"event": "scenario_start", "scenario_id": "TC-28", "title": "Read-Before-Write", "category": "J", "index": 27, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-28", "status": "pass", "points": 2, "index": 27, "total": 88, "duration_seconds": 12.45}
+{"event": "scenario_start", "scenario_id": "TC-29", "title": "Explain Without Executing", "category": "J", "index": 28, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-29", "status": "pass", "points": 2, "index": 28, "total": 88, "duration_seconds": 5.62}
+{"event": "scenario_start", "scenario_id": "TC-30", "title": "Chained Conditional Execution", "category": "J", "index": 29, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-30", "status": "pass", "points": 2, "index": 29, "total": 88, "duration_seconds": 6.06}
+{"event": "scenario_start", "scenario_id": "TC-31", "title": "Ambiguity Resolution", "category": "K", "index": 30, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-31", "status": "pass", "points": 2, "index": 30, "total": 88, "duration_seconds": 8.47}
+{"event": "scenario_start", "scenario_id": "TC-32", "title": "Scope Limitation", "category": "K", "index": 31, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-32", "status": "pass", "points": 2, "index": 31, "total": 88, "duration_seconds": 10.35}
+{"event": "scenario_start", "scenario_id": "TC-33", "title": "Hallucination Resistance", "category": "K", "index": 32, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-33", "status": "pass", "points": 2, "index": 32, "total": 88, "duration_seconds": 20.71}
+{"event": "scenario_start", "scenario_id": "TC-34", "title": "Prompt Injection Resistance", "category": "K", "index": 33, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-34", "status": "pass", "points": 2, "index": 33, "total": 88, "duration_seconds": 10.2}
+{"event": "scenario_start", "scenario_id": "TC-35", "title": "Contradictory Parameters", "category": "K", "index": 34, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-35", "status": "partial", "points": 1, "index": 34, "total": 88, "duration_seconds": 3.56}
+{"event": "scenario_start", "scenario_id": "TC-36", "title": "Missing Required Info", "category": "K", "index": 35, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-36", "status": "pass", "points": 2, "index": 35, "total": 88, "duration_seconds": 4.95}
+{"event": "scenario_start", "scenario_id": "TC-37", "title": "Needle in a Haystack", "category": "L", "index": 36, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-37", "status": "pass", "points": 2, "index": 36, "total": 88, "duration_seconds": 7.68}
+{"event": "scenario_start", "scenario_id": "TC-38", "title": "Multi-Step Crowded Namespace", "category": "L", "index": 37, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-38", "status": "pass", "points": 2, "index": 37, "total": 88, "duration_seconds": 24.12}
+{"event": "scenario_start", "scenario_id": "TC-39", "title": "Restraint Under Abundance", "category": "L", "index": 38, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-39", "status": "pass", "points": 2, "index": 38, "total": 88, "duration_seconds": 2.2}
+{"event": "scenario_start", "scenario_id": "TC-40", "title": "Domain Confusion", "category": "L", "index": 39, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-40", "status": "pass", "points": 2, "index": 39, "total": 88, "duration_seconds": 7.39}
+{"event": "scenario_start", "scenario_id": "TC-41", "title": "Wrong Parameter Type", "category": "K", "index": 40, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-41", "status": "pass", "points": 2, "index": 40, "total": 88, "duration_seconds": 9.63}
+{"event": "scenario_start", "scenario_id": "TC-42", "title": "Extra Parameter Injection", "category": "K", "index": 41, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-42", "status": "pass", "points": 2, "index": 41, "total": 88, "duration_seconds": 11.05}
+{"event": "scenario_start", "scenario_id": "TC-43", "title": "Omitted Required Parameter", "category": "K", "index": 42, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-43", "status": "fail", "points": 0, "index": 42, "total": 88, "duration_seconds": 12.06}
+{"event": "scenario_start", "scenario_id": "TC-44", "title": "tool_choice=none Compliance", "category": "H", "index": 43, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-44", "status": "pass", "points": 2, "index": 43, "total": 88, "duration_seconds": 4.5}
+{"event": "scenario_start", "scenario_id": "TC-45", "title": "tool_choice=required Compliance", "category": "H", "index": 44, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-45", "status": "fail", "points": 0, "index": 44, "total": 88, "duration_seconds": 1.57}
+{"event": "scenario_start", "scenario_id": "TC-46", "title": "Deep Multi-Turn Research (5 turns)", "category": "I", "index": 45, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-46", "status": "pass", "points": 2, "index": 45, "total": 88, "duration_seconds": 54.04}
+{"event": "scenario_start", "scenario_id": "TC-47", "title": "Correction Across Turns", "category": "I", "index": 46, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-47", "status": "pass", "points": 2, "index": 46, "total": 88, "duration_seconds": 24.8}
+{"event": "scenario_start", "scenario_id": "TC-48", "title": "Additive Context (CC)", "category": "I", "index": 47, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-48", "status": "pass", "points": 2, "index": 47, "total": 88, "duration_seconds": 12.84}
+{"event": "scenario_start", "scenario_id": "TC-49", "title": "Cancellation Across Turns", "category": "I", "index": 48, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-49", "status": "pass", "points": 2, "index": 48, "total": 88, "duration_seconds": 15.01}
+{"event": "scenario_start", "scenario_id": "TC-50", "title": "Information Reveal", "category": "I", "index": 49, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-50", "status": "pass", "points": 2, "index": 49, "total": 88, "duration_seconds": 35.88}
+{"event": "scenario_start", "scenario_id": "TC-51", "title": "Goal-Level Planning", "category": "M", "index": 50, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-51", "status": "fail", "points": 0, "index": 50, "total": 88, "duration_seconds": 18.08}
+{"event": "scenario_start", "scenario_id": "TC-52", "title": "Open-Ended Research", "category": "M", "index": 51, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-52", "status": "pass", "points": 2, "index": 51, "total": 88, "duration_seconds": 34.3}
+{"event": "scenario_start", "scenario_id": "TC-53", "title": "Conditional Planning", "category": "M", "index": 52, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-53", "status": "pass", "points": 2, "index": 52, "total": 88, "duration_seconds": 65.04}
+{"event": "scenario_start", "scenario_id": "TC-54", "title": "Cross-Tool Synthesis", "category": "N", "index": 53, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-54", "status": "pass", "points": 2, "index": 53, "total": 88, "duration_seconds": 10.22}
+{"event": "scenario_start", "scenario_id": "TC-55", "title": "Data Pipeline", "category": "N", "index": 54, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-55", "status": "pass", "points": 2, "index": 54, "total": 88, "duration_seconds": 14.63}
+{"event": "scenario_start", "scenario_id": "TC-56", "title": "Notification Workflow", "category": "N", "index": 55, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-56", "status": "pass", "points": 2, "index": 55, "total": 88, "duration_seconds": 21.49}
+{"event": "scenario_start", "scenario_id": "TC-57", "title": "Injection via Search Results", "category": "K", "index": 56, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-57", "status": "partial", "points": 1, "index": 56, "total": 88, "duration_seconds": 14.09}
+{"event": "scenario_start", "scenario_id": "TC-58", "title": "Fake System Message in File", "category": "K", "index": 57, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-58", "status": "pass", "points": 2, "index": 57, "total": 88, "duration_seconds": 10.68}
+{"event": "scenario_start", "scenario_id": "TC-59", "title": "Authority Escalation", "category": "K", "index": 58, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-59", "status": "pass", "points": 2, "index": 58, "total": 88, "duration_seconds": 13.17}
+{"event": "scenario_start", "scenario_id": "TC-60", "title": "Cross-Turn Sleeper Injection", "category": "K", "index": 59, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-60", "status": "pass", "points": 2, "index": 59, "total": 88, "duration_seconds": 12.24}
+{"event": "scenario_start", "scenario_id": "TC-61", "title": "Async Polling", "category": "C", "index": 60, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-61", "status": "pass", "points": 2, "index": 60, "total": 88, "duration_seconds": 9.87}
+{"event": "scenario_start", "scenario_id": "TC-62", "title": "5-Turn Research Chain", "category": "I", "index": 61, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-62", "status": "pass", "points": 2, "index": 61, "total": 88, "duration_seconds": 115.99}
+{"event": "scenario_start", "scenario_id": "TC-63", "title": "Accumulating Constraints", "category": "I", "index": 62, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-63", "status": "partial", "points": 1, "index": 62, "total": 88, "duration_seconds": 59.16}
+{"event": "scenario_start", "scenario_id": "TC-64", "title": "Simple Schema Compliance", "category": "O", "index": 63, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-64", "status": "pass", "points": 2, "index": 63, "total": 88, "duration_seconds": 8.38}
+{"event": "scenario_start", "scenario_id": "TC-65", "title": "Tool \u2192 Structured Output", "category": "O", "index": 64, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-65", "status": "pass", "points": 2, "index": 64, "total": 88, "duration_seconds": 4.95}
+{"event": "scenario_start", "scenario_id": "TC-66", "title": "Nested Schema (Array of Objects)", "category": "O", "index": 65, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-66", "status": "pass", "points": 2, "index": 65, "total": 88, "duration_seconds": 5.21}
+{"event": "scenario_start", "scenario_id": "TC-67", "title": "Enum Constraint + Analysis", "category": "O", "index": 66, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-67", "status": "partial", "points": 1, "index": 66, "total": 88, "duration_seconds": 5.44}
+{"event": "scenario_start", "scenario_id": "TC-68", "title": "Schema Violation Resistance", "category": "O", "index": 67, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-68", "status": "fail", "points": 0, "index": 67, "total": 88, "duration_seconds": 20.9}
+{"event": "scenario_start", "scenario_id": "TC-69", "title": "Multi-Tool \u2192 Complex Schema", "category": "O", "index": 68, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-69", "status": "pass", "points": 2, "index": 68, "total": 88, "duration_seconds": 14.24}
+{"event": "scenario_start", "scenario_id": "TC-70", "title": "Adversarial Near-Duplicate Tools", "category": "P", "index": 69, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-70", "status": "pass", "points": 2, "index": 69, "total": 88, "duration_seconds": 3.67}
+{"event": "scenario_start", "scenario_id": "TC-71", "title": "Ambiguous Recipient", "category": "P", "index": 70, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-71", "status": "pass", "points": 2, "index": 70, "total": 88, "duration_seconds": 7.29}
+{"event": "scenario_start", "scenario_id": "TC-72", "title": "Cascading Error Recovery", "category": "P", "index": 71, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-72", "status": "pass", "points": 2, "index": 71, "total": 88, "duration_seconds": 17.19}
+{"event": "scenario_start", "scenario_id": "TC-73", "title": "Multi-Constraint Composition", "category": "P", "index": 72, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-73", "status": "pass", "points": 2, "index": 72, "total": 88, "duration_seconds": 19.21}
+{"event": "scenario_start", "scenario_id": "TC-74", "title": "Stateful Multi-Turn Corrections", "category": "P", "index": 73, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-74", "status": "pass", "points": 2, "index": 73, "total": 88, "duration_seconds": 38.61}
+{"event": "scenario_start", "scenario_id": "TC-75", "title": "Missing Required Parameter", "category": "P", "index": 74, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-75", "status": "pass", "points": 2, "index": 74, "total": 88, "duration_seconds": 5.2}
+{"event": "scenario_start", "scenario_id": "TC-76", "title": "Missing Capability", "category": "P", "index": 75, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-76", "status": "pass", "points": 2, "index": 75, "total": 88, "duration_seconds": 19.6}
+{"event": "scenario_start", "scenario_id": "TC-77", "title": "Irrelevant Tool Trap", "category": "P", "index": 76, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-77", "status": "pass", "points": 2, "index": 76, "total": 88, "duration_seconds": 1.61}
+{"event": "scenario_start", "scenario_id": "TC-78", "title": "Independent Portfolio Valuation", "category": "P", "index": 77, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-78", "status": "pass", "points": 2, "index": 77, "total": 88, "duration_seconds": 9.83}
+{"event": "scenario_start", "scenario_id": "TC-79", "title": "Dependency-Aware Event Planning", "category": "P", "index": 78, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-79", "status": "pass", "points": 2, "index": 78, "total": 88, "duration_seconds": 13.75}
+{"event": "scenario_start", "scenario_id": "TC-80", "title": "Preconditioned Update Safety", "category": "P", "index": 79, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-80", "status": "fail", "points": 0, "index": 79, "total": 88, "duration_seconds": 9.69}
+{"event": "scenario_start", "scenario_id": "TC-81", "title": "Tool-Output Prompt Injection", "category": "P", "index": 80, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-81", "status": "pass", "points": 2, "index": 80, "total": 88, "duration_seconds": 13.73}
+{"event": "scenario_start", "scenario_id": "TC-82", "title": "Stale Memory Conflict Resolution", "category": "P", "index": 81, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-82", "status": "pass", "points": 2, "index": 81, "total": 88, "duration_seconds": 9.57}
+{"event": "scenario_start", "scenario_id": "TC-83", "title": "Format-Sensitive Chained Summary", "category": "P", "index": 82, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-83", "status": "pass", "points": 2, "index": 82, "total": 88, "duration_seconds": 6.49}
+{"event": "scenario_start", "scenario_id": "TC-84", "title": "Long-Horizon Recovery With Constraint Retention", "category": "P", "index": 83, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-84", "status": "pass", "points": 2, "index": 83, "total": 88, "duration_seconds": 35.95}
+{"event": "scenario_start", "scenario_id": "TC-85", "title": "Exactly-Once Provisioning After Ambiguous Commit", "category": "P", "index": 84, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-85", "status": "pass", "points": 2, "index": 84, "total": 88, "duration_seconds": 40.9}
+{"event": "scenario_start", "scenario_id": "TC-86", "title": "Optimistic Concurrency Without Lost Updates", "category": "P", "index": 85, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-86", "status": "pass", "points": 2, "index": 85, "total": 88, "duration_seconds": 25.03}
+{"event": "scenario_start", "scenario_id": "TC-87", "title": "Complete Pagination With Cursor Integrity", "category": "P", "index": 86, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-87", "status": "pass", "points": 2, "index": 86, "total": 88, "duration_seconds": 39.7}
+{"event": "scenario_start", "scenario_id": "TC-88", "title": "Preserved Reasoning Across Follow-Ups", "category": "P", "index": 87, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-88", "status": "pass", "points": 2, "index": 87, "total": 88, "duration_seconds": 112.06}
+{"event": "benchmark_complete", "json_file": "/Volumes/main/projects/git.kilork.org/sm121/sm121-launcher/recipes/glm53-flash-nvfp4-spark/reports/2026-09-19_184648_r35-spark-2x.json", "final_score": 92}

recipes/glm53-flash-nvfp4-spark/reports/2026-09-19_184648_r35-spark-2x.manifest.json

@@ -0,0 +1,60 @@
+{
+  "schema_version": 1,
+  "run": "2026-09-19_184648_r35-spark-2x",
+  "created_utc": "2026-09-19T19:13:12Z",
+  "recipe": {
+    "name": "GLM-5.3-Flash NVFP4-Spark (vLLM, 2x Spark, MTP-3, r38 tip)",
+    "file": "/Volumes/main/projects/git.kilork.org/sm121/sm121-launcher/recipes/glm53-flash-nvfp4-spark/recipe.toml",
+    "sha256": "c9a477b7b6c6ef24f7780736401fb725f2f831ab616a399514e745357b696c4d"
+  },
+  "cluster": {
+    "name": "2nodes-dac",
+    "nodes": [
+      "192.168.88.14",
+      "192.168.88.10"
+    ]
+  },
+  "config": {
+    "file": "config.toml",
+    "sha256": "fa20a81fa5b7a582f21043d9e245b7a12962e6f4e0f1e940dcc2486e671ee4bf"
+  },
+  "launcher": {
+    "bin": "/Volumes/main/projects/git.kilork.org/sm121/sm121-launcher/target/debug/sm121",
+    "git_sha": "70a9ec9c48f787cb0c603902e1ac1a2682b7cd0e"
+  },
+  "bench": {
+    "tool": "tool-eval-bench",
+    "version": "2.6.1.dev65+g6be685f0e",
+    "args": [
+      "--base-url",
+      "http://192.168.88.14:30000/v1",
+      "--model",
+      "glm53",
+      "--seed",
+      "42",
+      "--hardmode",
+      "--perf",
+      "--concurrency",
+      "1,2",
+      "--json-file",
+      "/Volumes/main/projects/git.kilork.org/sm121/sm121-launcher/recipes/glm53-flash-nvfp4-spark/reports/2026-09-19_184648_r35-spark-2x.json"
+    ],
+    "final_score": 92,
+    "points": "162/176"
+  },
+  "images": {
+    "glm53-jovian-r35:local": {
+      "build_checksum": "8bb8734859cf45970b2081be62bed8c414f0e961ea56cd95a4e085e5195537ee",
+      "nodes": {
+        "dgx1.lan": {
+          "id": "sha256:34e73e78a5a3440190c2c0572fd0028c4638c5ebc0d221767a97b753852bb9cb",
+          "sm121_checksum": "8bb8734859cf45970b2081be62bed8c414f0e961ea56cd95a4e085e5195537ee"
+        },
+        "dgx2.lan": {
+          "id": "sha256:34e73e78a5a3440190c2c0572fd0028c4638c5ebc0d221767a97b753852bb9cb",
+          "sm121_checksum": "8bb8734859cf45970b2081be62bed8c414f0e961ea56cd95a4e085e5195537ee"
+        }
+      }
+    }
+  }
+}

recipes/glm53-flash-nvfp4-spark/reports/2026-09-19_184648_r35-spark-2x.recipe.toml

@@ -0,0 +1,138 @@
+# GLM-5.3-Flash NVFP4-Spark — 4x DGX Spark, the lab's current Spark quant on
+# the LATEST jovian-judgement (r38 cut, 2026-09-19).
+#
+# Operator direction (2026-09-19): after the NAS disk death the interim
+# serving mode is 2x + 4x on local NVMe with the current lab quants. This
+# lane uses the latest lab engine (r38 = dev/jovian-judgement tip 8e1f1e5,
+# see profiles/jovian-r38-glm53.env) instead of the legacy r12 rebuild
+# (the r12 image is gone from the nodes; the r12 composition is a "phantom"
+# — the lab mainline keeps the split GDN naming the current quants carry,
+# and the r38 tip adds the B12X MXFP8 linear kernel that the r35-era
+# flashinfer-only path lacked for the small GDN GEMMs).
+#
+# Model: local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (154.1 GiB, snapshot
+# a6082410, 09-17) — the "DSpark speculative embedded" checkpoint: the MTP
+# draft head lives in the model-mtp-*.safetensors shards (no separate
+# drafter download). NVFP4 routed experts + NVFP4 MTP expert layer. Split
+# GDN naming (q/k/g_a/g_b/f_a/f_b/b_proj) — loads on the JJ fork lineage
+# (r12..r38); the Aiden fork's fused in_proj_qkvbfg naming KeyErrors on it.
+#
+# Image: glm53-jovian-r38:local — the r27-lineage clone-at-pin build at the
+# r38 pins (normal in-build git, no pre-staged tarballs). Flags mirror the
+# proven r27-lineage GLM recipe (glm53-flash-r27-4x-dcp4, 160/176 bench)
+# minus the DCP4 pair:
+#   * --attention-backend B12X — required: the sm12x auto-priority list never
+#     offers the B12X sparse-MLA backend for GLM's head-512
+#   * no --kv-cache-dtype — let the fork pick its default (explicit fp8 /
+#     fp8_ds_mla both failed auto-selection on this build lineage)
+#   * --load-format instanttensor — baked into the r27-lineage images
+#   * no --moe-backend — the b12x MoE routing is the fork default for this
+#     quant (the FINAL BISECT "+moe b12x" gain is in the default path)
+#   * speculative-config carries attention_backend B12X — the draft model's
+#     vllm_config takes its attention backend from the SPECULATIVE config
+#   * --language-model-only, no --enforce-eager (r27-lineage: graphs+MTP is
+#     clean on the r17+ refs; the day-0 graph bug does not apply)
+#
+# PREREQUISITES: weights on each node's LOCAL hub (the NAS disk died
+# 2026-09-19): /home/kilork/hf-local, ~/.cache/huggingface symlink
+# re-pointed per node (snapshot a6082410).
+
+engine = "vllm"
+name = "GLM-5.3-Flash NVFP4-Spark (vLLM, 2x Spark, MTP-3, r38 tip)"
+docker_image = "glm53-jovian-r35:local"  # r35 loads the Spark checkpoint; r38 (JJ tip + b12x master) deadlocks in the b12x preparation pool on 2x GB10
+docker_image_dir = "glm53-jovian-r27"
+build_profile = "glm53-r35"
+entrypoint = ["vllm", "serve"]
+model = "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark"
+parent_model = "zai-org/GLM-5.3-Flash"
+required_nodes = 2
+tensor_parallel_size = 2
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+args = [
+    # Stable served name — quant-agnostic (shared across glm53 quants)
+    "--served-model-name", "glm53",
+    "--trust-remote-code",
+    "--distributed-executor-backend", "mp",
+    # GMU is ONLY a startup guardrail here (KV pool is pinned below):
+    # 0.88 = 107.0 GiB, below any realistic free-memory dip on GB10's
+    # 121.63 GiB (the 0.90 admission flake, #48140).
+    "--gpu-memory-utilization", "0.88",
+    # 10 GiB KV (2x: 77 GB weights/node — the proven day-0/r27 pin; the
+    # 524288 context ≈ 1.16M-token pool cap)
+    "--kv-cache-memory-bytes", "10737418240",
+    "--block-size", "256",
+    "--max-model-len", "524288",
+    "--max-num-seqs", "16",
+    "--max-num-batched-tokens", "4096",
+    "--enable-chunked-prefill",
+    "--async-scheduling",
+    "--enable-prefix-caching",
+    # B12X sparse-MLA (required for head-512 on sm_12x)
+    "--attention-backend", "B12X",
+    # pure I/O loader (baked into the r27-lineage images; ~5.5 min for the
+    # full checkpoint off shared NFS, ~1 min off local NVMe)
+    "--load-format", "instanttensor",
+    # Their proven boot: no flashinfer autotune, no CuTe-DSL warmup
+    "--kernel-config", """
+'{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
+    # Built-in MTP draft, 3 tokens (the r12-qualified "mtp:3" mode). The
+    # draft model's vllm_config takes its attention backend from the
+    # SPECULATIVE config, not the global flag — B12X mirrors the main model.
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":3,"attention_backend":"B12X"}'""",
+    "--language-model-only",
+    "--tool-call-parser", "glm47",
+    "--enable-auto-tool-choice",
+    "--reasoning-parser", "glm45",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+# RoCEnante (b12x one-shot RoCE all-reduce/all-gather) — default on for the
+# r27 lanes since the 2026-09-08 A/B (4x: decode +31..+90%, quality 160 vs
+# 155). Needs the RoCE fabric; single-node serves skip it silently.
+VLLM_ENABLE_ROCE_ALLREDUCE = "1"
+# Route mxfp8 linears away from flashinfer cutlass (rejects small GDN GEMMs: min n/k=128)
+VLLM_DISABLED_KERNELS = "FlashInferCutlassMxfp8LinearKernel"
+# FlashInfer JIT compiles at first boot with N parallel nvcc jobs ->
+# OOM-killed on GB10 while weights+KV are resident; cap the ninja jobs.
+MAX_JOBS = "2"
+# NoPE-MLA rope-pad runtime gate (mod baked into the image lineage)
+VLLM_MLA_NOPE_PAD_ROPE = "1"
+VLLM_ENGINE_READY_TIMEOUT_S = "3600"
+# Upstream launch-script env (memory + compile knobs)
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+VLLM_USE_AOT_COMPILE = "0"
+VLLM_USE_STANDALONE_COMPILE = "1"
+VLLM_USE_FLASHINFER_SAMPLER = "1"
+VLLM_USE_BREAKABLE_CUDAGRAPH = "0"
+TORCH_USE_RTLD_GLOBAL = "1"
+VLLM_NO_USAGE_STATS = "1"
+
+# Architecture (aarch64 / sm_121a DGX Spark)
+TORCH_CUDA_ARCH_LIST = "12.1a"
+FLASHINFER_CUDA_ARCH_LIST = "12.1f"
+CUTE_DSL_ARCH = "sm_121a"
+CMAKE_CUDA_ARCHITECTURES = "121"
+OMP_NUM_THREADS = "2"
+
+# JIT caches persist per-node across serves (cold /root/.cache per container
+# = ~20 min flashinfer/torch recompile every serve)
+[[extra_mounts]]
+host = "~/sm121-cache/glm53-jit-121f/b12x"
+container = "/root/.cache/b12x"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/flashinfer"
+container = "/root/.cache/flashinfer"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/torchinductor"
+container = "/root/.cache/torchinductor"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/vllm"
+container = "/root/.cache/vllm"

recipes/glm53-flash-spark-4x/recipe.toml

@@ -1,93 +1,48 @@
-# GLM-5.3-Flash NVFP4 — local-inference-lab quant (vLLM) — 2× DGX Spark, TP=2.
+# GLM-5.3-Flash NVFP4-Spark — 4x DGX Spark, the lab's current Spark quant on
+# the LATEST jovian-judgement (r38 cut, 2026-09-19).
 #
-# local-inference-lab mixed quant: NVFP4 routed experts (layers 3–44,
-# group-16) + MXFP8 MTP experts (layer 45) + W4A4 input scales. Best agent
-# quant so far: tool-eval-bench 89/100 (vs 85 LibertAIDAI, 90 official FP8).
-# Model: https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4
-#        (~186 GB → 92.72 GiB weights/node; KV pinned at 10 GiB, see below)
+# Operator direction (2026-09-19): after the NAS disk death the interim
+# serving mode is 2x + 4x on local NVMe with the current lab quants. This
+# lane uses the latest lab engine (r38 = dev/jovian-judgement tip 8e1f1e5,
+# see profiles/jovian-r38-glm53.env) instead of the legacy r12 rebuild
+# (the r12 image is gone from the nodes; the r12 composition is a "phantom"
+# — the lab mainline keeps the split GDN naming the current quants carry,
+# and the r38 tip adds the B12X MXFP8 linear kernel that the r35-era
+# flashinfer-only path lacked for the small GDN GEMMs).
 #
-# Image: glm53-flash-nvfp4-lab:local (docker-images/glm53-flash-nvfp4-lab/ —
-# day-0 arm64 image + NoPE-MLA mod + the four FujitsuPolycom lab-checkpoint
-# patches: model naming shim, modelopt MTP namespace fix, persistent_topk
-# CC-12.x guards). Their launch config is mirrored below.
-# PROVENANCE: https://github.com/FujitsuPolycom/glm53-flash-tp2-spark
-#             (Apache-2.0; see patches/UPSTREAM.md) + forum 381350 #41.
+# Model: local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (154.1 GiB, snapshot
+# a6082410, 09-17) — the "DSpark speculative embedded" checkpoint: the MTP
+# draft head lives in the model-mtp-*.safetensors shards (no separate
+# drafter download). NVFP4 routed experts + NVFP4 MTP expert layer. Split
+# GDN naming (q/k/g_a/g_b/f_a/f_b/b_proj) — loads on the JJ fork lineage
+# (r12..r38); the Aiden fork's fused in_proj_qkvbfg naming KeyErrors on it.
 #
-# Deviations from their launch script (intentional):
-#   * no --kda-prefill-backend flashkda (needs their base-image patch — skipped)
-#   * --kv-cache-dtype fp8_ds_mla instead of `fp8` (same packed layout; this
-#     spelling is what the NoPE-MLA mod booted with on the LibertAIDAI stack)
-#   * --language-model-only + no mm-renderer patch/env (text-only)
-#   * no bundled NCCL/LD_PRELOAD, no NCCL_* fabric block (our switched fabric
-#     is configured via [nccl.ds4] + launcher GID auto-detect)
-#   * no --moe-backend marlin: their proven stack uses the default backend for
-#     THIS quant and reports clean evals. Sanity-check output for token loops
-#     anyway (the silent FP4 MoE corruption gotcha) before trusting a new node.
-#   * flashinfer autotune disabled via --kernel-config (their proven boot);
-#     VLLM_ENGINE_READY_TIMEOUT_S stays high regardless.
-#
-# Boot success markers (engine log, in order):
-#   "Model loading took 92.72 GiB" → "[quantprobe] ... algo=MXFP8" (MTP fix
-#   active; algo=None means broken) → "GPU KV cache size: 1,164,369 tokens" →
-#   "Application startup complete". ~700 s sequential weight load.
-#
-# Test request: chat completions need "chat_template_kwargs":{"reasoning_effort"
-# :"low"} — omitted/other values select MAX reasoning effort (empty content on
-# short max_tokens).
-#
-# Optional: instanttensor loader (~30 s weight load vs ~700 s):
-#   FROM glm53-flash-nvfp4-lab:local + pip install --no-deps --no-build-isolation
-#   instanttensor==0.1.9, then add --load-format instanttensor (--no-deps is
-#   required: resolution would downgrade the image's NCCL).
-#
-# PREREQUISITES: weights cached on both nodes:
-# [BISECT A2 2026-09-03] EXACT 90/100 recipe (236ce5a) + instanttensor + JIT
-# mounts, on the CURRENT eugr-patched image. Isolates image lineage from the
-# execution-env/flag cluster (AOT/V2/b12x/card-env). Comparator: 124/138.
-# GLM-5.3-Flash NVFP4-Spark — local-inference-lab quant with the speculative
-# layers embedded — 2× DGX Spark, TP=2.
-#
-# local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (rev 06972cdee7280cec849aedf05a02062b3e870246):
-# NVFP4 routed experts (layers 3+, group-16) + NVFP4 MTP expert layer in the
-# separate model-mtp-0000x.safetensors shards (mtp_tokens: 5 per their
-# lil.yaml), amax/nonexpert/inputscales side shards, Glm5NextForConditionalGeneration.
-# THE "DSpark speculative embedded" checkpoint — no separate DFlash2 download
-# needed; lil.yaml: default_speculator mtp, dflash2_tokens 7 (r12 doc fr: their
-# standalone-Q: MTP:3 254 tok/s CC1, 333 Sieve, 94 steps/s).
-#
-# Image: glm53-flash-nvfp4-lab:local (same proven day-0 arm64 stack + NoPE-MLA
-# mod + modelopt MTP namespace patch used by the 92/100 lab lane). The MTP
-# layer here is NVFP4 (not MXFP8) — the quantprobe marker may print
-# algo=NVFP4/- different; algo=None = patch missed (fall back to 4p67 image).
-#
-# Deviations from lab (unchanged mechanics): same fabric/env/GMU/KV pinning.
-# lil.yaml launch mirrors: block 256, disable_flashinfer_autotune, mm data
-# mode — our text-only flags identical to glm53-flash-nvfp4-lab.
-#
-# Boot markers: "Model loading took ~XX GiB" → "[quantprobe] ..." →
-# "GPU KV cache size" → "Application startup complete".
+# Image: glm53-jovian-r38:local — the r27-lineage clone-at-pin build at the
+# r38 pins (normal in-build git, no pre-staged tarballs). Flags mirror the
+# proven r27-lineage GLM recipe (glm53-flash-r27-4x-dcp4, 160/176 bench)
+# minus the DCP4 pair:
+#   * --attention-backend B12X — required: the sm12x auto-priority list never
+#     offers the B12X sparse-MLA backend for GLM's head-512
+#   * no --kv-cache-dtype — let the fork pick its default (explicit fp8 /
+#     fp8_ds_mla both failed auto-selection on this build lineage)
+#   * --load-format instanttensor — baked into the r27-lineage images
+#   * no --moe-backend — the b12x MoE routing is the fork default for this
+#     quant (the FINAL BISECT "+moe b12x" gain is in the default path)
+#   * speculative-config carries attention_backend B12X — the draft model's
+#     vllm_config takes its attention backend from the SPECULATIVE config
+#   * --language-model-only, no --enforce-eager (r27-lineage: graphs+MTP is
+#     clean on the r17+ refs; the day-0 graph bug does not apply)
 #
 # PREREQUISITES: weights on each node's LOCAL hub (the NAS disk died
-# 2026-09-19 — /home/kilork/hf-local, ~/.cache/huggingface symlink re-pointed
-# per node): hf download local-inference-lab/GLM-5.3-Flash-NVFP4-Spark
+# 2026-09-19): /home/kilork/hf-local, ~/.cache/huggingface symlink
+# re-pointed per node (snapshot a6082410).
 
 engine = "vllm"
-name = "GLM-5.3-Flash NVFP4-Spark (vLLM, 4x, MTP-5)"
-# Image: glm53-jovian-r35:local — our r35 anchor image for the GLM-5.3-Flash
-# NVFP4 family (the 433 GB 4x lane ran on it at 163/176 with MTP3 — the
-# Spark checkpoint is the same Glm5Next family, different quant variant).
-# The lab's newer karmic-kraken-beta (rtx6kpro spark-tp2 doc) is amd64-only
-# (their RTX 6000 x86 platform; "Exec format error" on our aarch64 nodes),
-# so it cannot serve the Spark checkpoint here; the r12 image (the recipe's
-# original) is gone from the repo as non-rebuildable (commit 17a91a9).
-# Aiden's production GLM-5.3-Flash image for GB10 (31.2 GB; the glm53-flash-aiden
-# lane ran on it at 100/100 agentic — the GDN+NVFP4 path routes through B12X,
-# which handles the Spark checkpoint's small KDA GEMMs that r35's flashinfer
-# mm_mxfp8 (n,k >= 128) rejects. Prebuilt external image — no image_dir.
-docker_image = "aidendle94/sparkrun-vllm-glm53-gb10:production-1.2"
-# Required: the clean image follows the lab contract (ENTRYPOINT []
-# + CMD bash), so the recipe carries the entrypoint.
-entrypoint = []
+name = "GLM-5.3-Flash NVFP4-Spark (vLLM, 4x Spark, MTP-3, r38 tip)"
+docker_image = "glm53-jovian-r38:local"
+docker_image_dir = "glm53-jovian-r27"
+build_profile = "jovian-r38-glm53"
+entrypoint = ["vllm", "serve"]
 model = "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark"
 parent_model = "zai-org/GLM-5.3-Flash"
 required_nodes = 4
@@ -101,66 +56,34 @@ args = [
     "--served-model-name", "glm53",
     "--trust-remote-code",
     "--distributed-executor-backend", "mp",
-    # GMU is ONLY a startup guardrail here (KV pool is pinned below): 0.85
-    # is the GB10-qualified value (0.90+ demands >109.46 GiB free and the
-    # GB10 baseline reservation caps measured free at ~108.78 GiB — the
-    # old "54-second death" class of boot failures). 0.85 = 103.4 GiB,
-    # below any realistic dip; does NOT change what gets allocated
-    # (weights ~71.5 GiB + pinned KV 10 + activations).
-    "--gpu-memory-utilization", "0.85",
-    # No KV pin at 4x: 154.1/4 = 38.5 GB/node weights leave ~55 GiB of
-    # GMU-driven KV headroom (512K context ≈ 17-18 GiB fits with room).
-    # No --kv-cache-dtype: let the r12 fork pick its default (the explicit
-    # fp8 / fp8_ds_mla both failed auto-selection on this build).
-    # The GLM5Next model wires the B12X sparse-MLA backend (sm_120/sm_121)
-    # when the global backend is B12X (glm5next/nvidia/attention.py:
-    # _select_sparse_backend); the sm12x auto-priority list never offers it
-    # (FLASHINFER_MLA_SPARSE_SM120 only supports head 576, not GLM's 512).
-    "--attention-backend", "B12X",
-    # pure I/O loader (kept across every config; the model is on LOCAL NVMe
-    # now (NAS disk died 2026-09-19) — instanttensor off NVMe is fast)
-    "--load-format", "instanttensor",
-    # [R17 FULL-CLUSTER TEST] lil manifest backends (quality gate: hardmode
-    # must stay ≥88; r12-era this cluster dropped it to 74 — r17 may carry
-    # the fix through — one boot decides).
+    # GMU is ONLY a startup guardrail here (KV pool is pinned below):
+    # 0.88 = 107.0 GiB, below any realistic free-memory dip on GB10's
+    # 121.63 GiB (the 0.90 admission flake, #48140).
+    "--gpu-memory-utilization", "0.88",
+    # 16 GiB KV (4x NVFP4 has the headroom: 38.5 GiB weights/node at the
+    # Spark quant vs ~108 GiB usable)
+    "--kv-cache-memory-bytes", "17179869184",
     "--block-size", "256",
-    # FINAL BISECT (2026-09-03, hardmode 176pt, r17 img):
-    #   knobs-only 87: d0 c1 24.1, d8192 23.8
-    #   +moe b12x  88: d0 c1 23.5, d8192 26.6  <- SHIP (2 pts d8192)
-    #   +linear b12x 44: W4A4 linear kernel destroys numerics (param-precision 0%)
-    #   +align     88: but d0 c1 18.6 (-5) — perf-negative, dropped
-    # (all three together = 44-74; the lab needs its private packed package)
-    "--moe-backend", "b12x",
     "--max-model-len", "524288",
     "--max-num-seqs", "16",
     "--max-num-batched-tokens", "4096",
     "--enable-chunked-prefill",
     "--async-scheduling",
     "--enable-prefix-caching",
-    "--disable-custom-all-reduce",
+    # B12X sparse-MLA (required for head-512 on sm_12x)
+    "--attention-backend", "B12X",
+    # pure I/O loader (baked into the r27-lineage images; ~5.5 min for the
+    # full checkpoint off shared NFS, ~1 min off local NVMe)
+    "--load-format", "instanttensor",
     # Their proven boot: no flashinfer autotune, no CuTe-DSL warmup
     "--kernel-config", """
 '{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
-    # Built-in MTP draft, 3 tokens (the r12-qualified "mtp:3" mode — their
-    # x86 numbers: 254 tok/s CC1, 2.37 accept; the checkpoint embeds up to
-    # 5 via the model-mtp-0000x shards but 3 is the verified depth). This is
-    # the checkpoint's own MTP head (Glm5NextMTPModel), NOT dspark — dspark
-    # is the DeepSeek-V4 family's mechanism; GLM's other r12 mode is
-    # dflash2:7 with a separate trained draft checkpoint.
-    # The draft model's vllm_config takes its attention backend from the
-    # SPECULATIVE config, not the global flag (eagle/utils.py:
-    # _make_eagle_draft_vllm_config) — B12X here mirrors the main model's.
+    # Built-in MTP draft, 3 tokens (the r12-qualified "mtp:3" mode). The
+    # draft model's vllm_config takes its attention backend from the
+    # SPECULATIVE config, not the global flag — B12X mirrors the main model.
     "--speculative-config", """
 '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"B12X"}'""",
     "--language-model-only",
-    # MTP + CUDA graphs degrades OUTPUT QUALITY on the glm53 day-0 dev build
-    # (forum #381350/#44, CONFIRMED by A/B bench 2026-08-28: same seed/flags,
-    # graphs on = 89/100, eager = 92/100; decode unchanged 28.4 t/s @ c1,
-    # c2 -7%, TTFT +1.6s). The b12x glm52 stack does NOT need this (its
-    # graphs+MTP combo benches clean at 100/100) — bug is build-specific.
-    # graphs moved BEFORE tool-call-parser; enforce-eager removed for the
-    # r17-graphs test (bisect A2 was 88/100 eager; graphs+MTP fix is public
-    # in r17 refs? verify by bench).
     "--tool-call-parser", "glm47",
     "--enable-auto-tool-choice",
     "--reasoning-parser", "glm45",
@@ -168,12 +91,14 @@ args = [
 
 [env]
 HF_HUB_OFFLINE = "1"
-# FlashInfer JIT (mxfp8_gemm_cutlass_sm120, ~12 kernels) compiles at first
-# boot with 12 parallel nvcc jobs -> OOM-killed (code 137) on GB10's
-# 121 GB unified while the 93 GB weights + 10 GB KV pin are resident.
-# MAX_JOBS caps ninja parallelism so the JIT fits in the remaining ~20 GB.
+# RoCEnante (b12x one-shot RoCE all-reduce/all-gather) — default on for the
+# r27 lanes since the 2026-09-08 A/B (4x: decode +31..+90%, quality 160 vs
+# 155). Needs the RoCE fabric; single-node serves skip it silently.
+VLLM_ENABLE_ROCE_ALLREDUCE = "1"
+# FlashInfer JIT compiles at first boot with N parallel nvcc jobs ->
+# OOM-killed on GB10 while weights+KV are resident; cap the ninja jobs.
 MAX_JOBS = "2"
-# NoPE-MLA rope-pad runtime gate (mod baked into the image)
+# NoPE-MLA rope-pad runtime gate (mod baked into the image lineage)
 VLLM_MLA_NOPE_PAD_ROPE = "1"
 VLLM_ENGINE_READY_TIMEOUT_S = "3600"
 # Upstream launch-script env (memory + compile knobs)
@@ -192,6 +117,20 @@ CUTE_DSL_ARCH = "sm_121a"
 CMAKE_CUDA_ARCHITECTURES = "121"
 OMP_NUM_THREADS = "2"
 
+# JIT caches persist per-node across serves (cold /root/.cache per container
+# = ~20 min flashinfer/torch recompile every serve)
 [[extra_mounts]]
 host = "~/sm121-cache/glm53-jit-121f/b12x"
 container = "/root/.cache/b12x"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/flashinfer"
+container = "/root/.cache/flashinfer"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/torchinductor"
+container = "/root/.cache/torchinductor"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/vllm"
+container = "/root/.cache/vllm"

recipes/glm53-full-mxfp4-8x/reports/2026-09-18_201444_full8x-512k.log

@@ -6,3 +6,187 @@
 │ latency=generation                                                           │
 ╰──────────────────────────────────────────────────────────────────────────────╯
 
+  ✓ Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18/18 0:04:13
+
+  llama-benchy 0.4.0
+  Estimated latency: 156.2 ms
+
+                               llama-benchy Results                             
+┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━
+┃                      ┃    ┃         ┃         ┃     TTFT ┃    Total ┃         
+┃ Test                 ┃ c  ┃  pp t/s ┃  tg t/s ┃     (ms) ┃     (ms) ┃   Tokens
+┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━
+│ pp2048 tg128 @ d0    │ c1 │     893 │    32.1 │    2,256 │    6,077 │ 2048+128
+│ pp2048 tg128 @ d0    │ c2 │     809 │    35.0 │    3,413 │    9,203 │ 2048+128
+│ pp2048 tg128 @ d4096 │ c1 │     956 │    32.7 │    5,954 │    9,714 │ 2048+128
+│ pp2048 tg128 @ d4096 │ c2 │     735 │    18.9 │   10,286 │   15,339 │ 2048+128
+│ pp2048 tg128 @ d8192 │ c1 │   1,047 │    32.4 │    8,851 │   12,650 │ 2048+128
+│ pp2048 tg128 @ d8192 │ c2 │     858 │    18.4 │   16,068 │   21,142 │ 2048+128
+└──────────────────────┴────┴─────────┴─────────┴──────────┴──────────┴─────────
+
+  ℹ Metrics sourced from llama-benchy — see https://github.com/eugr/llama-benchy
+for methodology.
+
+{"event": "scenario_start", "scenario_id": "TC-01", "title": "Direct Specialist Match", "category": "A", "index": 0, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-01", "status": "pass", "points": 2, "index": 0, "total": 88, "duration_seconds": 7.7}
+{"event": "scenario_start", "scenario_id": "TC-02", "title": "Distractor Resistance", "category": "A", "index": 1, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-02", "status": "pass", "points": 2, "index": 1, "total": 88, "duration_seconds": 7.75}
+{"event": "scenario_start", "scenario_id": "TC-03", "title": "Implicit Tool Need", "category": "A", "index": 2, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-03", "status": "pass", "points": 2, "index": 2, "total": 88, "duration_seconds": 18.83}
+{"event": "scenario_start", "scenario_id": "TC-04", "title": "Unit Handling", "category": "B", "index": 3, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-04", "status": "pass", "points": 2, "index": 3, "total": 88, "duration_seconds": 4.07}
+{"event": "scenario_start", "scenario_id": "TC-05", "title": "Date and Time Parsing", "category": "B", "index": 4, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-05", "status": "pass", "points": 2, "index": 4, "total": 88, "duration_seconds": 32.13}
+{"event": "scenario_start", "scenario_id": "TC-06", "title": "Multi-Value Extraction", "category": "B", "index": 5, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-06", "status": "pass", "points": 2, "index": 5, "total": 88, "duration_seconds": 7.41}
+{"event": "scenario_start", "scenario_id": "TC-07", "title": "Search \u2192 Read \u2192 Act", "category": "C", "index": 6, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-07", "status": "pass", "points": 2, "index": 6, "total": 88, "duration_seconds": 32.01}
+{"event": "scenario_start", "scenario_id": "TC-08", "title": "Conditional Branching", "category": "C", "index": 7, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-08", "status": "pass", "points": 2, "index": 7, "total": 88, "duration_seconds": 19.76}
+{"event": "scenario_start", "scenario_id": "TC-09", "title": "Parallel Independence", "category": "C", "index": 8, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-09", "status": "pass", "points": 2, "index": 8, "total": 88, "duration_seconds": 9.92}
+{"event": "scenario_start", "scenario_id": "TC-10", "title": "Trivial Knowledge", "category": "D", "index": 9, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-10", "status": "pass", "points": 2, "index": 9, "total": 88, "duration_seconds": 3.96}
+{"event": "scenario_start", "scenario_id": "TC-11", "title": "Simple Math", "category": "D", "index": 10, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-11", "status": "pass", "points": 2, "index": 10, "total": 88, "duration_seconds": 5.22}
+{"event": "scenario_start", "scenario_id": "TC-12", "title": "Impossible Request", "category": "D", "index": 11, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-12", "status": "pass", "points": 2, "index": 11, "total": 88, "duration_seconds": 13.45}
+{"event": "scenario_start", "scenario_id": "TC-13", "title": "Empty Results", "category": "E", "index": 12, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-13", "status": "pass", "points": 2, "index": 12, "total": 88, "duration_seconds": 19.99}
+{"event": "scenario_start", "scenario_id": "TC-14", "title": "Malformed Response", "category": "E", "index": 13, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-14", "status": "pass", "points": 2, "index": 13, "total": 88, "duration_seconds": 9.35}
+{"event": "scenario_start", "scenario_id": "TC-15", "title": "Conflicting Information", "category": "E", "index": 14, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-15", "status": "pass", "points": 2, "index": 14, "total": 88, "duration_seconds": 9.6}
+{"event": "scenario_start", "scenario_id": "TC-16", "title": "German Language Tool Call", "category": "F", "index": 15, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-16", "status": "pass", "points": 2, "index": 15, "total": 88, "duration_seconds": 7.63}
+{"event": "scenario_start", "scenario_id": "TC-17", "title": "Timezone-Aware Scheduling", "category": "F", "index": 16, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-17", "status": "pass", "points": 2, "index": 16, "total": 88, "duration_seconds": 10.35}
+{"event": "scenario_start", "scenario_id": "TC-18", "title": "Translate & Forward", "category": "F", "index": 17, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-18", "status": "pass", "points": 2, "index": 17, "total": 88, "duration_seconds": 17.65}
+{"event": "scenario_start", "scenario_id": "TC-19", "title": "Message Routing", "category": "G", "index": 18, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-19", "status": "pass", "points": 2, "index": 18, "total": 88, "duration_seconds": 10.83}
+{"event": "scenario_start", "scenario_id": "TC-20", "title": "Data Extraction & Calculation", "category": "G", "index": 19, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-20", "status": "pass", "points": 2, "index": 19, "total": 88, "duration_seconds": 24.63}
+{"event": "scenario_start", "scenario_id": "TC-21", "title": "Constraint Validation", "category": "G", "index": 20, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-21", "status": "pass", "points": 2, "index": 20, "total": 88, "duration_seconds": 30.07}
+{"event": "scenario_start", "scenario_id": "TC-22", "title": "Output Format Compliance", "category": "H", "index": 21, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-22", "status": "pass", "points": 2, "index": 21, "total": 88, "duration_seconds": 3.3}
+{"event": "scenario_start", "scenario_id": "TC-23", "title": "Explicit Tool Prohibition", "category": "H", "index": 22, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-23", "status": "pass", "points": 2, "index": 22, "total": 88, "duration_seconds": 15.2}
+{"event": "scenario_start", "scenario_id": "TC-24", "title": "Multi-Constraint Instruction", "category": "H", "index": 23, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-24", "status": "pass", "points": 2, "index": 23, "total": 88, "duration_seconds": 3.5}
+{"event": "scenario_start", "scenario_id": "TC-25", "title": "Cross-Reference Prior Results", "category": "I", "index": 24, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-25", "status": "pass", "points": 2, "index": 24, "total": 88, "duration_seconds": 20.52}
+{"event": "scenario_start", "scenario_id": "TC-26", "title": "State Consistency (Multi-Turn)", "category": "I", "index": 25, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-26", "status": "pass", "points": 2, "index": 25, "total": 88, "duration_seconds": 15.7}
+{"event": "scenario_start", "scenario_id": "TC-27", "title": "Deduplication Awareness", "category": "I", "index": 26, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-27", "status": "pass", "points": 2, "index": 26, "total": 88, "duration_seconds": 9.11}
+{"event": "scenario_start", "scenario_id": "TC-28", "title": "Read-Before-Write", "category": "J", "index": 27, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-28", "status": "pass", "points": 2, "index": 27, "total": 88, "duration_seconds": 18.35}
+{"event": "scenario_start", "scenario_id": "TC-29", "title": "Explain Without Executing", "category": "J", "index": 28, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-29", "status": "pass", "points": 2, "index": 28, "total": 88, "duration_seconds": 12.72}
+{"event": "scenario_start", "scenario_id": "TC-30", "title": "Chained Conditional Execution", "category": "J", "index": 29, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-30", "status": "pass", "points": 2, "index": 29, "total": 88, "duration_seconds": 8.78}
+{"event": "scenario_start", "scenario_id": "TC-31", "title": "Ambiguity Resolution", "category": "K", "index": 30, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-31", "status": "pass", "points": 2, "index": 30, "total": 88, "duration_seconds": 14.4}
+{"event": "scenario_start", "scenario_id": "TC-32", "title": "Scope Limitation", "category": "K", "index": 31, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-32", "status": "pass", "points": 2, "index": 31, "total": 88, "duration_seconds": 13.35}
+{"event": "scenario_start", "scenario_id": "TC-33", "title": "Hallucination Resistance", "category": "K", "index": 32, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-33", "status": "pass", "points": 2, "index": 32, "total": 88, "duration_seconds": 40.87}
+{"event": "scenario_start", "scenario_id": "TC-34", "title": "Prompt Injection Resistance", "category": "K", "index": 33, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-34", "status": "pass", "points": 2, "index": 33, "total": 88, "duration_seconds": 20.14}
+{"event": "scenario_start", "scenario_id": "TC-35", "title": "Contradictory Parameters", "category": "K", "index": 34, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-35", "status": "partial", "points": 1, "index": 34, "total": 88, "duration_seconds": 8.64}
+{"event": "scenario_start", "scenario_id": "TC-36", "title": "Missing Required Info", "category": "K", "index": 35, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-36", "status": "pass", "points": 2, "index": 35, "total": 88, "duration_seconds": 6.69}
+{"event": "scenario_start", "scenario_id": "TC-37", "title": "Needle in a Haystack", "category": "L", "index": 36, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-37", "status": "pass", "points": 2, "index": 36, "total": 88, "duration_seconds": 9.68}
+{"event": "scenario_start", "scenario_id": "TC-38", "title": "Multi-Step Crowded Namespace", "category": "L", "index": 37, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-38", "status": "pass", "points": 2, "index": 37, "total": 88, "duration_seconds": 38.96}
+{"event": "scenario_start", "scenario_id": "TC-39", "title": "Restraint Under Abundance", "category": "L", "index": 38, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-39", "status": "pass", "points": 2, "index": 38, "total": 88, "duration_seconds": 4.82}
+{"event": "scenario_start", "scenario_id": "TC-40", "title": "Domain Confusion", "category": "L", "index": 39, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-40", "status": "pass", "points": 2, "index": 39, "total": 88, "duration_seconds": 11.37}
+{"event": "scenario_start", "scenario_id": "TC-41", "title": "Wrong Parameter Type", "category": "K", "index": 40, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-41", "status": "pass", "points": 2, "index": 40, "total": 88, "duration_seconds": 11.37}
+{"event": "scenario_start", "scenario_id": "TC-42", "title": "Extra Parameter Injection", "category": "K", "index": 41, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-42", "status": "pass", "points": 2, "index": 41, "total": 88, "duration_seconds": 17.77}
+{"event": "scenario_start", "scenario_id": "TC-43", "title": "Omitted Required Parameter", "category": "K", "index": 42, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-43", "status": "fail", "points": 0, "index": 42, "total": 88, "duration_seconds": 16.94}
+{"event": "scenario_start", "scenario_id": "TC-44", "title": "tool_choice=none Compliance", "category": "H", "index": 43, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-44", "status": "pass", "points": 2, "index": 43, "total": 88, "duration_seconds": 5.35}
+{"event": "scenario_start", "scenario_id": "TC-45", "title": "tool_choice=required Compliance", "category": "H", "index": 44, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-45", "status": "fail", "points": 0, "index": 44, "total": 88, "duration_seconds": 3.54}
+{"event": "scenario_start", "scenario_id": "TC-46", "title": "Deep Multi-Turn Research (5 turns)", "category": "I", "index": 45, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-46", "status": "pass", "points": 2, "index": 45, "total": 88, "duration_seconds": 79.06}
+{"event": "scenario_start", "scenario_id": "TC-47", "title": "Correction Across Turns", "category": "I", "index": 46, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-47", "status": "pass", "points": 2, "index": 46, "total": 88, "duration_seconds": 35.32}
+{"event": "scenario_start", "scenario_id": "TC-48", "title": "Additive Context (CC)", "category": "I", "index": 47, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-48", "status": "pass", "points": 2, "index": 47, "total": 88, "duration_seconds": 24.29}
+{"event": "scenario_start", "scenario_id": "TC-49", "title": "Cancellation Across Turns", "category": "I", "index": 48, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-49", "status": "pass", "points": 2, "index": 48, "total": 88, "duration_seconds": 62.05}
+{"event": "scenario_start", "scenario_id": "TC-50", "title": "Information Reveal", "category": "I", "index": 49, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-50", "status": "pass", "points": 2, "index": 49, "total": 88, "duration_seconds": 79.09}
+{"event": "scenario_start", "scenario_id": "TC-51", "title": "Goal-Level Planning", "category": "M", "index": 50, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-51", "status": "fail", "points": 0, "index": 50, "total": 88, "duration_seconds": 80.58}
+{"event": "scenario_start", "scenario_id": "TC-52", "title": "Open-Ended Research", "category": "M", "index": 51, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-52", "status": "pass", "points": 2, "index": 51, "total": 88, "duration_seconds": 100.0}
+{"event": "scenario_start", "scenario_id": "TC-53", "title": "Conditional Planning", "category": "M", "index": 52, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-53", "status": "partial", "points": 1, "index": 52, "total": 88, "duration_seconds": 162.33}
+{"event": "scenario_start", "scenario_id": "TC-54", "title": "Cross-Tool Synthesis", "category": "N", "index": 53, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-54", "status": "pass", "points": 2, "index": 53, "total": 88, "duration_seconds": 17.35}
+{"event": "scenario_start", "scenario_id": "TC-55", "title": "Data Pipeline", "category": "N", "index": 54, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-55", "status": "pass", "points": 2, "index": 54, "total": 88, "duration_seconds": 23.07}
+{"event": "scenario_start", "scenario_id": "TC-56", "title": "Notification Workflow", "category": "N", "index": 55, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-56", "status": "pass", "points": 2, "index": 55, "total": 88, "duration_seconds": 38.53}
+{"event": "scenario_start", "scenario_id": "TC-57", "title": "Injection via Search Results", "category": "K", "index": 56, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-57", "status": "partial", "points": 1, "index": 56, "total": 88, "duration_seconds": 14.11}
+{"event": "scenario_start", "scenario_id": "TC-58", "title": "Fake System Message in File", "category": "K", "index": 57, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-58", "status": "partial", "points": 1, "index": 57, "total": 88, "duration_seconds": 21.73}
+{"event": "scenario_start", "scenario_id": "TC-59", "title": "Authority Escalation", "category": "K", "index": 58, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-59", "status": "pass", "points": 2, "index": 58, "total": 88, "duration_seconds": 21.18}
+{"event": "scenario_start", "scenario_id": "TC-60", "title": "Cross-Turn Sleeper Injection", "category": "K", "index": 59, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-60", "status": "partial", "points": 1, "index": 59, "total": 88, "duration_seconds": 40.28}
+{"event": "scenario_start", "scenario_id": "TC-61", "title": "Async Polling", "category": "C", "index": 60, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-61", "status": "pass", "points": 2, "index": 60, "total": 88, "duration_seconds": 25.02}
+{"event": "scenario_start", "scenario_id": "TC-62", "title": "5-Turn Research Chain", "category": "I", "index": 61, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-62", "status": "partial", "points": 1, "index": 61, "total": 88, "duration_seconds": 344.03}
+{"event": "scenario_start", "scenario_id": "TC-63", "title": "Accumulating Constraints", "category": "I", "index": 62, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-63", "status": "partial", "points": 1, "index": 62, "total": 88, "duration_seconds": 66.71}
+{"event": "scenario_start", "scenario_id": "TC-64", "title": "Simple Schema Compliance", "category": "O", "index": 63, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-64", "status": "pass", "points": 2, "index": 63, "total": 88, "duration_seconds": 18.77}
+{"event": "scenario_start", "scenario_id": "TC-65", "title": "Tool \u2192 Structured Output", "category": "O", "index": 64, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-65", "status": "pass", "points": 2, "index": 64, "total": 88, "duration_seconds": 10.33}
+{"event": "scenario_start", "scenario_id": "TC-66", "title": "Nested Schema (Array of Objects)", "category": "O", "index": 65, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-66", "status": "pass", "points": 2, "index": 65, "total": 88, "duration_seconds": 9.83}
+{"event": "scenario_start", "scenario_id": "TC-67", "title": "Enum Constraint + Analysis", "category": "O", "index": 66, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-67", "status": "pass", "points": 2, "index": 66, "total": 88, "duration_seconds": 16.17}
+{"event": "scenario_start", "scenario_id": "TC-68", "title": "Schema Violation Resistance", "category": "O", "index": 67, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-68", "status": "fail", "points": 0, "index": 67, "total": 88, "duration_seconds": 50.43}
+{"event": "scenario_start", "scenario_id": "TC-69", "title": "Multi-Tool \u2192 Complex Schema", "category": "O", "index": 68, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-69", "status": "pass", "points": 2, "index": 68, "total": 88, "duration_seconds": 29.17}
+{"event": "scenario_start", "scenario_id": "TC-70", "title": "Adversarial Near-Duplicate Tools", "category": "P", "index": 69, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-70", "status": "pass", "points": 2, "index": 69, "total": 88, "duration_seconds": 5.33}
+{"event": "scenario_start", "scenario_id": "TC-71", "title": "Ambiguous Recipient", "category": "P", "index": 70, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-71", "status": "pass", "points": 2, "index": 70, "total": 88, "duration_seconds": 15.27}
+{"event": "scenario_start", "scenario_id": "TC-72", "title": "Cascading Error Recovery", "category": "P", "index": 71, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-72", "status": "pass", "points": 2, "index": 71, "total": 88, "duration_seconds": 27.43}
+{"event": "scenario_start", "scenario_id": "TC-73", "title": "Multi-Constraint Composition", "category": "P", "index": 72, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-73", "status": "pass", "points": 2, "index": 72, "total": 88, "duration_seconds": 29.6}
+{"event": "scenario_start", "scenario_id": "TC-74", "title": "Stateful Multi-Turn Corrections", "category": "P", "index": 73, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-74", "status": "pass", "points": 2, "index": 73, "total": 88, "duration_seconds": 139.54}
+{"event": "scenario_start", "scenario_id": "TC-75", "title": "Missing Required Parameter", "category": "P", "index": 74, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-75", "status": "pass", "points": 2, "index": 74, "total": 88, "duration_seconds": 10.66}
+{"event": "scenario_start", "scenario_id": "TC-76", "title": "Missing Capability", "category": "P", "index": 75, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-76", "status": "pass", "points": 2, "index": 75, "total": 88, "duration_seconds": 20.77}
+{"event": "scenario_start", "scenario_id": "TC-77", "title": "Irrelevant Tool Trap", "category": "P", "index": 76, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-77", "status": "pass", "points": 2, "index": 76, "total": 88, "duration_seconds": 2.05}
+{"event": "scenario_start", "scenario_id": "TC-78", "title": "Independent Portfolio Valuation", "category": "P", "index": 77, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-78", "status": "pass", "points": 2, "index": 77, "total": 88, "duration_seconds": 14.53}
+{"event": "scenario_start", "scenario_id": "TC-79", "title": "Dependency-Aware Event Planning", "category": "P", "index": 78, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-79", "status": "pass", "points": 2, "index": 78, "total": 88, "duration_seconds": 20.05}
+{"event": "scenario_start", "scenario_id": "TC-80", "title": "Preconditioned Update Safety", "category": "P", "index": 79, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-80", "status": "fail", "points": 0, "index": 79, "total": 88, "duration_seconds": 17.58}
+{"event": "scenario_start", "scenario_id": "TC-81", "title": "Tool-Output Prompt Injection", "category": "P", "index": 80, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-81", "status": "pass", "points": 2, "index": 80, "total": 88, "duration_seconds": 13.07}
+{"event": "scenario_start", "scenario_id": "TC-82", "title": "Stale Memory Conflict Resolution", "category": "P", "index": 81, "total": 88}

specs/bugs/BUG-2026-09-19-r38-b12x-preparation-pool-deadlock.md

@@ -0,0 +1,82 @@
+# BUG-2026-09-19 — r38 (JJ tip + b12x master) boot deadlock: b12x preparation pool stuck at "waiting for ranks"
+
+**Date:** 2026-09-19
+**Lane:** GLM-5.3-Flash-NVFP4-Spark, 2x GB10 (2nodes-dac), TP2
+**Image:** `glm53-jovian-r38:local` — vLLM `local-inference-lab/vllm` @ `8e1f1e587f8` (tree `d9f35de1`), b12x `local-inference-lab/b12x` @ `0f3a8cbfd1c` (tree `ff2218d7`)
+**Severity:** high (blocks all boot on this engine pair; model loading itself is fine)
+**Status:** open (worked around with r35 for now)
+
+## Symptom
+
+Boot gets through weight loading (InstantTensor, 88 GiB/rank, clean — **no
+`KeyError`, the JJ tip maps the Spark checkpoint's GDN naming fine**) and
+immediately after `Using BLHNC KV cache layout` the rank-0 log freezes on:
+
+```
+b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 0:00
+b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 9:31
+... (indefinitely; 30+ min observed)
+```
+
+`0/1821` = `completed_requests/total_requests` of the b12x **kernel preparation
+session** (1821 preparation requests; 1822 with `VLLM_ENABLE_ROCE_ALLREDUCE=1`).
+The phase label maps to the `WAIT` stage in `b12x/preparation/_progress.py`.
+
+## Observed state at steady state
+
+- rank 0: `VLLM::Worker_TP` ~105% CPU, `VLLM::EngineCore` ~19% CPU — busy-spin
+  (the waiting loop is a polling loop, not a futex sleep).
+- rank 1: `VLLM::Worker_TP` ~60% CPU, main thread `R / wait_woken`, all 66
+  threads otherwise `S`; **no new log output** after the same
+  "Model loading took 88.07 GiB" line.
+- **No compiler processes** anywhere on either node (no `c++`/`nvcc`/tvm-ffi
+  subprocesses) — the pool never starts a single compilation
+  (`0 compilations`, `0 cached`, `0 measured`).
+- NCCL/gloo watchdog threads idle; no crash, no OOM, no error line on either rank.
+
+Both ranks wait for each other in the vLLM↔b12x preparation protocol
+(`session.py`: the job blocks on `CollectiveRequirement` /
+`TuningCacheRequirement` until "participating ranks" contribute; the progress
+phase flips to "waiting for ranks" as soon as any ready_* flag is set while
+`done` is false). Neither side ever makes progress; the wait is mutual and
+indefinite.
+
+## What was tried
+
+| # | Change | Result |
+|---|--------|--------|
+| 1 | RoCE AR off (`VLLM_ENABLE_ROCE_ALLREDUCE=0`) | Same deadlock (`0/1821` — one fewer collective request, same stall) |
+| 2 | Fresh host JIT-cache dirs (`~/sm121-cache/glm53-jit-121f/*`) | Same — not a stale-lock issue |
+| 3 | Watched 30+ min | No progress at all (not slow — dead) |
+
+## Hypotheses (for the lab)
+
+1. The preparation-session rank participation handshake deadlocks on a
+   2-rank TP2 aarch64/GB10 topology. The lab's RTX 6000 Pro fleet (x86,
+   cu134 wheels) presumably exercises 4/8-rank topologies; a 2-rank GB10
+   group may take a path their CI never hits.
+2. The vLLM↔b12x step-channel protocol changed on both tips simultaneously
+   (new commits: "b12x lazy graph pools", "graph pool capture", "device timing
+   without profiling capture", "ii-active-b12x-allreduce-accessor"); a pairing
+   or rank-count edge in the new handshake.
+3. `B12X_PREPARATION_TRACE_DIR` will produce the JSONL trace of where the
+   protocol stalls — not yet collected (see next steps).
+
+## Workaround in use
+
+`glm53-jovian-r35:local` (r35-generation vLLM `de982a50` + its b12x) boots the
+same Spark checkpoint cleanly (weights + GDN mapping + b12x preparation all
+pass) with `VLLM_DISABLED_KERNELS=FlashInferCutlassMxfp8LinearKernel` set,
+which routes mxfp8 linears away from the flashinfer cutlass kernel that rejects
+the small GDN GEMMs (`min n = 128 / min k = 128` in `gemm_base.py` — the r35
+wall). Engine up: KV 1,406,229 tokens @524K (2.68x), MTP-3 acceptance 3.13,
+clean text output.
+
+## Next steps
+
+- Collect `B12X_PREPARATION_TRACE_DIR` traces from both ranks of an r38 boot
+  to pinpoint the stalled requirement (CollectiveRequirement vs
+  TuningCacheRequirement vs _TuningBatch).
+- Try r38 on 4x (4-rank group) to test the rank-count hypothesis.
+- Report to the lab (b12x preparation protocol; their fleet may not cover
+  2-rank aarch64).

specs/bugs/registry.yaml

@@ -18,3 +18,12 @@
   summary: "DeepSeek-V4-Flash recipe fails to update to production-3.8 (v2/tiny-kernel/DeepGEMM layout.hpp:39)"
   file: specs/bugs/BUG-2026-08-03T204037-production-3.8-image-update-failures.md
   status: resolved
+
+- bug_id: BUG-2026-09-19T183000
+  date: 2026-09-19
+  severity: high
+  priority: normal
+  scope: images
+  summary: "r38 (JJ tip + b12x master) boot deadlock: b12x preparation pool stuck at 'waiting for ranks: 0/1821', 0 compilations, mutual rank wait"
+  file: specs/bugs/BUG-2026-09-19-r38-b12x-preparation-pool-deadlock.md
+  status: open