fix hardmode for night shift + future: version-pin tool (2.6.1.dev45, --hardmode = 88 scenarios), re-baseline anchor lane before gating; reports commit to recipes/<lane>/reports/; new eugr-4x test recipe (2x stays production default until 4x dramatic)

AGENTS.md

@@ -518,4 +518,4 @@ the detect command per node and renders `NCCL_IB_GID_INDEX=auto`.
 - Use `rg` (ripgrep) for all code search — plain `grep -rn` across this tree aborts/hangs and is unusable; `rg` is fast and skips `.git`/gitignored paths
 - `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`
 - **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.
-- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree.
+- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir). Quality-gate semantics are version-sensitive (`--hardmode` = 69+19 = 88 scenarios in 2.6.1.dev45; the older 176-pt pool is retired) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; re-baseline the anchor lane before comparing new lanes.

recipes/glm53-flash-nvfp4-eugr-4x/recipe.toml

@@ -0,0 +1,134 @@
+# GLM-5.3-Flash NVFP4 — eugr B12X 4-node test variant (vLLM) — 4× DGX Spark, TP=4.
+#
+# Night-shift A/B cell of `glm53-flash-nvfp4-eugr` (the 2x recipe is the
+# PRODUCTION default — keep it until 4x delivers a dramatic improvement).
+# Same image/model/spec/flags as the 2x lane; only the cluster shape differs
+# (4 nodes, TP4, 16 seqs / 8192 batched / 16 GiB KV pin — eugr's stock 4x
+# shape, confirmed public at forums.developer.nvidia.com/t/381350/364:
+# ~60 tok/s c1, ~73 c4 decode on this exact config). Requires the 4nodes
+# cluster to be free (the day's aiden 4x serve stopped first).
+#
+# Port of eugr's dual-Spark recipe (eugr/spark-vllm-docker @ 94b3c30,
+# 2026-09-04, recipes/glm-5.3-flash.yaml) onto our launcher, on their
+# prebuilt image. A/B cell against our r18 fork build: same lab branch line
+# (dev/jovian-judgement) — their image = 09-04 tip + lukealonso/b12x master
+# + their patch set; our r18 pin = 09-05 tip + lab b12x + eugr-derived
+# patches. Compare quality (hardmode) + decode + TTFT before trusting either.
+#
+# Image: eugr/spark-vllm-b12x:nightly-20260904 (digest c3b44d3f…, = :latest).
+#   Built from local-inference-lab/vllm dev/jovian-judgement (their e91bc76,
+#   2026-09-03, switched the ref from dev/infernal-invocation) +
+#   lukealonso/b12x @ master; torch 2.13.0, CUDA 13.0.2,
+#   nvidia-cutlass-dsl 4.7.0; their docker/patch_vllm_*.py set +
+#   instanttensor + fastsafetensors; system-NCCL symlink fix. Exact b12x
+#   commit baked in-image: /workspace/b12x-source-commit (+ build-metadata.yaml).
+#   NOTE: the image ships NO ENTRYPOINT — this recipe must carry
+#   entrypoint = ["vllm","serve"] (unlike the sparkrun/aiden images).
+#
+# Model: local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (their recipe's model —
+#   NOT the GLM-5.3-NVFP4-Spark repo the aiden lane uses). ~180 GB class:
+#   huggingface-cli download on BOTH nodes before the first serve
+#   (HF_HUB_OFFLINE=1 below).
+#
+# What this lane tests (vs our r17/r18 findings):
+#   - FULL b12x cluster: --attention-backend B12X + --moe-backend b12x +
+#     --linear-backend b12x. On our r17 image, --linear-backend b12x was THE
+#     quality TOXIN (44/100 hardmode). Their 09-04 b12x predates the SM121
+#     A16-precision commits, so the answer is open — measure, don't trust.
+#   - RoCEnante one-shot RoCE collectives: VLLM_ENABLE_ROCE_ALLREDUCE=1
+#     (their vllm #597 / b12x #295 pair; −5–19% decode per-step,
+#     +15–23% c16 measured on 4× Spark TP4).
+#   - MTP k=5 with drafter moe_backend=humming + B12X attention (vendor-card
+#     parity), AOT=1 + MEGA_AOT=1 + V2 runner, mamba-cache-mode align,
+#     instanttensor buffered loader, prefix caching ON (we run prefix-cache
+#     off elsewhere — perf-relevant, not quality-relevant).
+#
+# Deliberate parity deviations from our house lanes:
+#   - NO --language-model-only (their recipe loads the vision tower; add the
+#     flag only if memory is tight)
+#   - --kv-cache-dtype fp8 (plain), not fp8_ds_mla
+#   - no --decode-context-parallel-size (their DCP=1 is the default; no-op)
+#   - NCCL/cluster env comes from the launcher (their NCCL_* block omitted)
+#
+# REFERENCE:
+#   https://github.com/eugr/spark-vllm-docker  (recipes/glm-5.3-flash.yaml)
+#   https://hub.docker.com/r/eugr/spark-vllm-b12x/tags
+#   https://forums.developer.nvidia.com/t/381350/364 — public confirmation of
+#   this flag set (2026-09-05, tea24864 on 4x/TP4, perf-only: ~60 tok/s c1,
+#   ~73 c4 decode, prefill 2.3-2.8k tok/s) + the DFlash2 alternative spec
+#   ("method":"dflash" + "incoai/GLM-5.3-Flash-DFlash2" drafter, k=7, see
+#   night queue 3b)
+
+engine = "vllm"
+name = "GLM-5.3-Flash NVFP4 eugr B12X 4x (vLLM, 4x Spark, MTP k=5, RoCE)"
+docker_image = "eugr/spark-vllm-b12x:nightly-20260904"
+model = "local-inference-lab/GLM-5.3-Flash-NVFP4-Spark"
+required_nodes = 4
+tensor_parallel_size = 4
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+# image ships no ENTRYPOINT — the launcher supplies the vllm command
+entrypoint = ["vllm", "serve"]
+
+args = [
+    # Stable served name — quant-agnostic (shared across glm53 quants)
+    "--served-model-name", "glm53",
+    "--trust-remote-code",
+    "--distributed-executor-backend", "mp",
+    # eugr parity: FULL b12x cluster (linear-b12x = the r17 quality toxin —
+    # measure before trusting)
+    "--attention-backend", "B12X",
+    "--moe-backend", "b12x",
+    "--linear-backend", "b12x",
+    "--mamba-cache-mode", "align",
+    "--enable-prefix-caching",
+    "--enable-chunked-prefill",
+    "--dtype", "bfloat16",
+    "--kv-cache-dtype", "fp8",
+    "--quantization", "modelopt_mixed",
+    "--no-enable-flashinfer-autotune",
+    "--load-format", "instanttensor",
+    "--model-loader-extra-config", """
+'{"instanttensor_copy":false}'""",
+    "--block-size", "256",
+    "--max-model-len", "1048576",
+    "--max-num-seqs", "16",
+    "--max-num-batched-tokens", "8192",
+    # sparkring MTP3-mesh reference (DCP4 probe, night queue 1b): DCP4 +
+    # 24 GiB fp8 KV/rank gave ~4.32M logical KV tokens on 4 Sparks. Flip the
+    # next line ON for the DCP4 cell (stock vLLM flag, verified in our tree;
+    # DCP>1 requires MLA — GLM53 qualifies). Keep OFF for the plain 4x cell.
+    # "--decode-context-parallel-size", "4",
+    # MTP k=5 (eugr recipe / vendor card); drafter on humming MoE + B12X attn
+    "--speculative-config", """
+'{"method":"mtp","num_speculative_tokens":5,"moe_backend":"humming","attention_backend":"B12X"}'""",
+    "--reasoning-parser", "glm45",
+    "--tool-call-parser", "glm47",
+    "--enable-auto-tool-choice",
+    "--kv-cache-memory-bytes", "17179869184",
+    "--gpu-memory-utilization", "0.87",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+VLLM_ENGINE_READY_TIMEOUT_S = "3600"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+CUTE_DSL_ARCH = "sm_121a"
+SAFETENSORS_FAST_GPU = "1"
+# RoCEnante one-shot RoCE collectives (ON in eugr's recipe; off-by-default flag)
+VLLM_ENABLE_ROCE_ALLREDUCE = "1"
+VLLM_ROCE_ALLREDUCE_MAX_SIZE = "2MB"
+VLLM_ENABLE_PCIE_ALLREDUCE = "0"
+VLLM_WORKER_MULTIPROC_METHOD = "spawn"
+VLLM_SSM_CONV_STATE_LAYOUT = "DS"
+VLLM_USE_AOT_COMPILE = "1"
+VLLM_USE_MEGA_AOT_ARTIFACT = "1"
+VLLM_USE_V2_MODEL_RUNNER = "1"
+B12X_POLICY_MODE = "auto"
+# instanttensor buffered loader (eugr's defaults)
+INSTANTTENSOR_BACKEND = "BUFFERED"
+INSTANTTENSOR_BUFFER_SIZE = "67108864"
+INSTANTTENSOR_CHUNK_SIZE = "8388608"
+INSTANTTENSOR_CONCURRENCY = "1"
+INSTANTTENSOR_IO_DEPTH = "3"

recipes/glm53-flash-nvfp4-eugr/recipe.toml

@@ -1,5 +1,9 @@
 # GLM-5.3-Flash NVFP4 — eugr B12X full-cluster variant (vLLM) — 2× DGX Spark, TP=2.
 #
+# PRODUCTION DEFAULT for the eugr stack: run this 2x lane until the 4x test
+# variant (recipes/glm53-flash-nvfp4-eugr-4x) delivers a dramatic
+# improvement (night shift compares them).
+#
 # Port of eugr's dual-Spark recipe (eugr/spark-vllm-docker @ 94b3c30,
 # 2026-09-04, recipes/glm-5.3-flash.yaml) onto our launcher, on their
 # prebuilt image. A/B cell against our r18 fork build: same lab branch line

specs/tech-architecture/glm53-recipe-state.md

@@ -8,6 +8,16 @@ tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53 \
   --seed 42 --hardmode --perf --concurrency 1,2
 ```
 
+**Tool version pin (2026-09-05, verified against the installed tool):**
+tool-eval-bench 2.6.1.dev45+gcf54b4bfe (uv tool; git
+SeraphimSerapis/tool-eval-bench @ cf54b4bfe). Current pool semantics:
+`--hardmode` = 69 standard (A–O) + 19 Hard Mode (P) = **88 scenarios**
+(≈26 min, dry-run verified); `--hardmode-only` = 19 (≈6 min). The `/176`
+scores above (155/176 anchor, lab ref 156/176) are pinned to the OLDER
+176-point pool — scores do NOT transfer across tool versions. Every
+campaign now re-baselines the anchor lane with the CURRENT tool first
+(night queue 1a), then gates new lanes against that new baseline.
+
 All three share served name **`glm53`** — clients never changed between runs.
 
 ## Results
@@ -610,7 +620,16 @@ Order = value/duration; each item lists prereqs + expected verdict.
    a8c796f3+pr582+pr646 → 343661fe; b12x b58f34ea+pr284 → 1adb8a38),
    then
    `TAG=glm53-jovian-r12:local bash build.sh`. Test campaign:
-   (a) quality anchor glm53-flash-nvfp4-r17abi (expect ≥88/100 hardmode),
+   (a) **Re-baseline the quality anchor first** (tool version changed —
+   see the bench header): run the anchor lane glm53-flash-nvfp4-r17abi
+   (2x, known-good at 88/100 on the old pool) with the current tool:
+   `tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53
+   --seed 42 --hardmode --perf --concurrency 1,2` (88 scenarios, ≈26 min),
+   record the new baseline score in this file, save the report to
+   recipes/glm53-flash-nvfp4-r17abi/reports/; gate ALL new lanes (r18,
+   eugr 2x/4x, redhat-MTP, dsv4-vision) against the NEW baseline — never
+   against the old /176 numbers. Campaign reports always go to
+   recipes/<lane>/reports/<UTC-stamp>_<label>.md.
    (b) **re-bisect `--linear-backend b12x`** — new b12x A16 precision
    selection may clear the 44/100 toxin (29.6 c1 unlock if clean),
    (c) RoCE one-shot on the 4x lane (VLLM_ENABLE_ROCE_ALLREDUCE=1;
@@ -644,6 +663,8 @@ Order = value/duration; each item lists prereqs + expected verdict.
    local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` on both nodes. Serve
    glm53-flash-nvfp4-eugr → A/B vs r18: full b12x cluster quality
    (linear-b12x toxin retest on their stack) + RoCE 2x decode numbers.
+   4-node cell: glm53-flash-nvfp4-eugr-4x (needs all 4 nodes free) vs
+   the 2x lane — 2x stays the production default unless 4x is dramatic.
    3b. **DFlash2 spec A/B** (same eugr image, after 3 boots). Swap the
    spec to `"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2",
    "num_speculative_tokens":7,"draft_tensor_parallel_size":2,