# GLM-5.3-Flash NVFP4 — three quant variants measured on 2× DGX Spark (ds4)

Measured 2026-08-27/28 on cluster `ds4` (dgx3+dgx4, switched fabric), all three
served by sm121 through the launcher, same benchmark invocation:

```
tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53 \
  --seed 42 --hardmode --perf --concurrency 1,2 \
  --json-file <abs repo path>/recipes/<lane>/reports/<UTC-stamp>_<label>.json
```

Keep BOTH the rich `.md` report (copied from the tool's `runs/` output dir)
and the raw JSON (`--json-file`; absolute path — the bench runs in a separate
cwd, and the flag keeps stdout clean for the `.log`) in
`recipes/<lane>/reports/` under the same base name.

**Tool version pin (2026-09-05, verified against the installed tool):**
tool-eval-bench 2.6.1.dev45+gcf54b4bfe (uv tool; git
SeraphimSerapis/tool-eval-bench @ cf54b4bfe). Current pool semantics:
`--hardmode` = 69 standard (A–O) + 19 Hard Mode (P) = **88 scenarios**
(≈26 min, dry-run verified); `--hardmode-only` = 19 (≈6 min). The `/176`
scores above (155/176 anchor, lab ref 156/176) are pinned to the OLDER
176-point pool — scores do NOT transfer across tool versions. Every
campaign now re-baselines the anchor lane with the CURRENT tool first
(night queue 1a), then gates new lanes against that new baseline.

All three share served name **`glm53`** — clients never changed between runs.

## Results

| recipe | quant | quality | points | decode tg t/s (c1) | tg c2 | TTFT | deployability | MTP |
|---|---|---|---|---|---|---|---|---|
| `glm53-flash-nvfp4-lab` | local-inference-lab mixed NVFP4/MXFP8-MTP | **92** (eager) / 89 (graphs) | 162/176 | **28.4** | 40.5 | 5.6 s | **72** | ✅ k=3, accept len ~2.9–3.5, ~63–83 % |
| `glm53-flash-nvfp4-redhat` | RedHatAI W4A4 compressed-tensors | **90** | 159/176 | 13.9 | 13.8 | 2.2 s | 67 | ❌ (no MTP weights in checkpoint) |
| `glm53-flash-nvfp4` | LibertAIDAI NVFP4-A16 | **88** | 154/176 | 26.2 | 22.7 | 6.3 s | 69 | ✅ k=5 |
| `glm53-flash-nvfp4-4p67` | 4p67 W4A16-experts + MXFP8-attn, eager, MTP k=3 | **87** | 153/176 | 24.4 | 39.2 | 1.4 s | 67 | ✅ accept ~90% |
| `glm53-flash-aiden-4x` | aiden image production-1.1 (DFlash2 overlay), NVFP4, **incoai DFlash2 k=7**, eager, 16 GiB KV pin, 4× Spark TP4 (his production shape) | **89** | 156/176 | 37.6 (d0 c1; flat to d8192, 2,185 pp t/s — matches his ~2,100) | 50.7 (c2) / 63.5 (c4) | 0.9 s | **80** | ✅ DFlash2 k=7 |
| `glm53-flash-aiden-2x` | aiden image **production-1.1** (DFlash2 overlay), NVFP4, **incoai DFlash2 k=7**, eager, 4 GiB KV pin (2x profile) | **89** | 157/176 | 22.3 (d0 c1; d8192 c4 degrades — 4 GiB pin) | — | — | 66 | ✅ DFlash2 k=7 (drafter block 8−1) |
| `glm53-flash-sparkrun` | community sparkrun image (aidendle94), NVFP4, eager, NO MTP (image lacks the fix) | **89** | 154/176 | 14.3 | 23.8 | 1.4 s | 66 | ❌ (n/a in image) |
| `glm53-flash-exl3` | brandonmusic EXL3/TR3 4bpw (Entrpi image + instanttensor), DFlash2-7, graphs | **88** | 155/176 | 23.1 | 24.1 | 2.6 s d0 | **77** | ✅ accept 48–81 % |

(lab scored /174 — TC-53 was excluded from its run as an infrastructure
failure (one trial crossed the 120 s request timeout; the scenario runs ~140 s
total — harness drops timed-out scenarios from the denominator; had it scored
like redhat's 2/2, lab would be 157/176 = 89 — no change). Quality % is
comparable. LibertAIDAI is the only variant whose MTP *regresses* under c2
(26.2→22.7) — lab's k=3 draft scales, LibertAIDAI's k=5 does not.

4x note (2026-09-04): the 4x aiden run had one safety-critical failure —
TC-43 called `web_search` with an empty query (required-parameter violation);
not seen in the 2x aiden run.)

Prefill (pp2048) 0.3–1.7 K t/s depending on depth/concurrency — noisy, all three
in the same band (NAS load-influenced); decode is the stable discriminator.

## Read of the numbers

- **lab (eager) wins outright**: 92/100 best quality AND 2× decode (28–40 t/s
  vs ~14 baseline). Acceptance lengths matched upstream's 2.55–3.5 range.
- **redhat** (90/100 with graphs) is a **baseline-decode dead-end on this box:
  its checkpoint ships zero MTP tensors** (verified index: 146,745 tensors,
  none under layer 45), so it is stuck at ~14 t/s — no longer the quality
  pick either.
- **lab --enforce-eager = day-driver; redhat = fallback; LibertAIDAI =**
  proven first but k=5 MTP regresses at c2 (eager might lift it too — untested).

## Images & patches

| image dir | base | patches |
|---|---|---|
| `glm53-flash-nvfp4` | vllm/vllm-openai:glm53-flash-arm64-cu130 | kingjones30 NoPE-MLA mod (4-part, anchor-asserted) |
| `glm53-flash-nvfp4-lab` | same | + 4 FujitsuPolycom patches (Apache-2.0, see patches/UPSTREAM.md): model.py naming shim, modelopt MTP namespace fix, persistent_topk CC-12.x guards ×2 |
| redhat recipe reuses `glm53-flash-nvfp4:local` (mod is quant-independent) | | |

Boot markers `[quantprobe] layer=RoutedExperts prefix=model.layers.45.mlp.experts
algo=MXFP8` confirm the MTP fix is live; `algo=None` there = broken load.

## MTP + CUDA graphs quality bug (forum #381350/#44, CONFIRMED 2026-08-28)

`0rand`: “mtp or dflash plus cuda graphs seriously hit quality — try
disabling cuda graph capture”. A/B with identical seed/flags on lab:

| arm | quality | points | tg c1 | tg c2 | TTFT |
|---|---|---|---|---|---|
| graphs (CUDA graphs ON) | 89 | 155/174 | 28.4 | 43.4 | 3.9 s |
| **`--enforce-eager`** | **92** | **162/176** | 28.4 | 40.5 | 5.6 s |

+3 quality points for −7 % c2 decode — eager is now the lab default. Accept
metrics were healthy in BOTH arms (71.6 %, len 3.15 eager), so this is not an
acceptance collapse — graph capture corrupts sampled output silently.
Corroborating: randomllama (DFlash2 recipe) already flagged “not claiming
bit-identical greedy outputs” with graphs on. The b12x glm52 stack runs
graphs + MTP k=4 FULL_AND_PIECEWISE benches clean (100/100) — build-specific
bug in the glm53 day-0 dev build, do NOT propagate --enforce-eager to glm52.
Untested follow-ups: LibertAIDAI + eager (k=5; expect ~88→9x), redhat is
graph-neutral (no MTP, stays 90 with graphs).

## Operational notes (measured)

### redhat @ 524288 ctx (2026-08-28)

262144 booted on the base image; **512K needed three changes** (each verified
by a real boot that died without it):
1. `--kernel-config {"enable_flashinfer_autotune":false,"enable_cutedsl_warmup":false}`
   + `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` — autotune scratch at
   512K shapes OOMs the driver (`NVRM NV_ERR_NO_MEMORY _memdescAllocInternal`);
   the sparse-MLA kernels' non-autotuned fallback is fine (lab proved it).
2. `--max-num-batched-tokens 4096` + `--block-size 256` (lab's memory shape).
3. **The lab image** — the killer was `launch_persistent_topk ... total_ctas=62
   > num_sms*occupancy=48 ... fallback requires >=128KB smem (have 101376)`,
   i.e. exactly what the FujitsuPolycom `sparse_attn_indexer(_kpool)` CC-12.0
   guards fix. Indexer CTAs scale with ctx: 262K fit 48 SMs, 512K needs 62.
   The patches are arch-level → safe for the compressed-tensors checkpoint
   (quantprobe is a no-op without modelopt; output verified correct).

Result: KV pool **1,334,265 tokens / 2.54× @ 524288** (10 GiB pin; lab gets
1,164,369 from the same pin — its MTP speculator reserves memory before KV
sizing), ~17 min clean boot, correct greedy output.

Note: an orphaned **rank-1 headless worker survives rank-0 death** and keeps
the node occupied — always `docker rm -f` label-filtered leftovers before a
retry serve (or the launcher refuses with "only 1 nodes").

- lab weight load: 92.19 GiB / 700 s sequential from the NAS HF cache;
  `--kv-cache-memory-bytes 10737418240` → KV 1,164,369 tokens / 2.22× @ 512K
  (matches upstream to the digit — do NOT switch back to GMU sizing).
- First launcher serve of lab died on both ranks at ~54 s (exit 1, logs lost
  to `--rm`); re-run was clean twice. Suspected transient (cold imports +
  simultaneous launch). If it recurs: re-launch without `--rm` and keep logs.
- `sm121 serve` immediately after `sm121 stop` can fail with "cluster has only
  1 nodes" — occupancy race, nodes not yet released; wait ~10 s and retry.
- LibertAIDAI/redhat recipes still run FlashInfer autotune (60-min possible
  boot, `VLLM_ENGINE_READY_TIMEOUT_S=3600` set); the lab recipe disables it via
  `--kernel-config` per upstream's proven boot.
- Chat-template caveat: `reasoning_effort:"low"` still emits a reasoning trace
  (budget ≠ 0); `max_tokens:48` can yield `content: null, finish: length` —
  use ≥256 for smoke tests.

## Follow-ups

- eugr B12X variant recipe `glm53-flash-nvfp4-eugr` (2026-09-05): port of
  eugr/spark-vllm-docker @ 94b3c30 recipes/glm-5.3-flash.yaml on
  eugr/spark-vllm-b12x:nightly-20260904 (their dev/jovian-judgement 09-04 tip
  + lukealonso/b12x master; no ENTRYPOINT in image → recipe carries
  ["vllm","serve"]). A/B cell vs the r18 fork build: full b12x cluster
  (linear-b12x = r17 toxin retest) + RoCE one-shot all-reduce + MTP k=5.
  Model: lab GLM-5.3-Flash-NVFP4-Spark (cache on both nodes first).
  Unmeasured — dry-run verified only.

- Vendor instanttensor loader into the lab image (~30 s vs 700 s weight load).
- Consider adding `--kernel-config` autotune-off to the LibertAIDAI recipe
  (boot minutes shorter; redhat now has it, see 512K section — verify before
  trusting on LibertAIDAI).
- MTP for redhat: RESOLVED (2026-09-05) — no variant repo needed: the main
  `RedHatAI/GLM-5.3-Flash-NVFP4` repo gained `model_mtp.safetensors` (7.6 GB)
  and the card an "Enable Speculative Decoding" section (k=5, plain
  `--speculative-config {"method":"mtp",...}` — same model path, no swap).
  Recipe `glm53-flash-nvfp4-redhat` updated to MTP k=5 (watch the +7.6 GB/node
  memory cost vs the 10 GiB KV pin; k=3 fallback). The lab's 5.2 pattern
  (`-mtp-FP8` variant repos) did not apply; the cross-graft MXFP8-MTP idea is
  moot.

## 4p67: --enforce-eager vs CUDA graphs (2026-08-29, same seed/invocation)

Same recipe both arms (MTP k=3 included), only `--enforce-eager` removed;
graphs arm verified to capture (FULL 5/5 + PIECEWISE 11/11), KV identical
(1,164,369 tok pinned).

| metric | eager | graphs |
|---|---|---|
| quality | **87** (153/176) | **88** (154/176) |
| deployability | 67 | 68 |
| tg t/s c1 (d0/d4096/d8192) | 21.1 / 23.2 / **24.4** | **23.6** / 22.6 / 22.9 |
| tg t/s c2 (d0/d4096) | **39.2** / 22.9 | 33.1 / **24.0** |
| TTFT (d0 / d8192) | 1.45 s / 6.17 s | 1.41 s / 6.10 s |
| MTP draft accept | 90.0–92.2 % | 89.0–97.0 % |

Findings:

* **No graphs+MTP quality regression on 4p67** (88 vs 87 = 1 point, run-to-run
  noise; the failure sets differ only by 5 shuffled partials). The lab
  92→89 graphs regression (forum #44) does NOT reproduce on this quant —
  so the bug is build/quant-specific, not universal.
* **Graphs buy nothing on decode here**: every cell within ±5%, and the
  headline c2 number is *worse* with graphs (39.2→33.1). MTP already eats
  the launch-gap that graphs would recover.
* Prefill rows are prefix-cache-polluted (recipe enables prefix caching and
  the perf probe repeats prompts); cold prefill ~1.4–1.5 K t/s both arms.
* **Decision: `--enforce-eager` stays in `glm53-flash-nvfp4-4p67`** — it is
  measurably free here and stays immune to the lab graph bug class.

Reports: `/tmp/runs/2026/08/2026-08-29T09-09-14…--(eager)`,
`…10-27-35.818420Z_181bbd18.md` (graphs).

## sparkrun (community image) A/B (2026-08-30, seed 42)

aidendle94/sparkrun-vllm-glm53-gb10:production-1.0 (same day-0 vLLM dev
base as our lab image, arm64-only) serving the same lab quant: quality
89 (154/176) but decode tg 14.3/23.8 t/s (c1/c2) vs lab-eager 24.4/39.2
and responsiveness 12/100 (median turn 11.7 s) — works, not performant:
- MTP impossible in that image (no modelopt-MTP namespace fix).
- --moe-backend marlin must be pinned (their auto = TileLang JIT storm,
  worker OOM-killed at boot), nvrtc.h bind-mount needed, plain fp8 KV
  (their mla kernel asserts pe_dim==64 for fp8_ds_mla).
Report: /tmp/runs/2026/08/2026-08-30T09-41-57.562243Z_108a798b.md.

## EXL3 lane (Entrpi community image) — boot diagnosis (2026-08-30)

ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark:v1-dflash2 + brandonmusic
GLM-5.3-Flash-tr3-4bpw (164 GiB) + DFlash2 drafter. VERIFIED SERVING
2026-08-30 evening: the loader-side OOM chain (`--load-format instanttensor`
overlay) resolved — stock loader balloons ~80 GiB CPU-anon streaming the
checkpoint; instanttensor streams ~175 MB/s with tiny buffers. ~22.8 tok/s
with DFlash2 k=7 (accept 48-81% by content); math smoke exact. DFlash2
drafter switched to the upstream `incoai/GLM-5.3-Flash-DFlash2` (downloaded
to NAS at operator request); MTP3-language-only + v84 image lanes remain
SM120-only upstream. Fixed in-recipe: NGC
entrypoint (needs explicit `vllm serve`), drafter swap to the MXFP8
mirror (incoai raw drafter absent from the NAS hub), GMU 0.86. BLOCKED:
global OOM kills the worker at 75-95% of shard load on BOTH streamed-from-
NAS nodes (repro 2×; dmesg oom-killer 12:22:25 / 13:12:24). Upstream's
"validated NFS topology" does not hold on our fabric — the mirror
topology (per-node local NVMe) is the next step; dgx4 is 98% full
(candidate cleanup targets ~1.3T of re-downloadable zai BF16 repos
there) — awaiting operator call.

## EXL3 lane bench (2026-08-30 evening, seed 42, same invocation)

`glm53-flash-exl3` (brandonmusic GLM-5.3-Flash-tr3-4bpw 4bpw, Entrpi
v1-dflash2 image + instanttensor overlay) — quality 88 (155/176, the
failure set is the usual 16 shuffled partials/fails; nothing new flags),
decode tg 23.1/24.1 t/s (DFlash2 k=7 drafting keeps C1≈C2 — the flattest
concurrency profile of any glm53 lane), TTFT 2.6 s @d0, pp ~817-1,004
(noisy, NAS-adjacent). The headline: **deployability 77/100 — the best of
the four lanes** (lab 72, 4p67 67, sparkrun 66), driven by responsiveness
50/100 with a 3.0 s median turn. Quality still trails lab-eager-with-MTP3
(92) by ~4 points — the 4bpw price. Report:
/tmp/runs/2026/08/2026-08-30T17-43-04.770487Z_63900bf7.md.

## 0rand winning-config trial on the lab quant (2026-08-30 evening)

Forum 381350 #117/#119 report 93/100 (164/176) for the lab quant with
GMU 0.89, MNBT 2048, KV auto-sized (1.05M pool), MTP-3. Reproduced on
ds4 as `glm53-flash-nvfp4-lab-0rand` (variant in tree): KV auto-size
lands at 1,167,586 tokens — same capacity as our pinned 10 GiB, so the
pin is unnecessary; smoke exact. Bench (--seed 42 --hardmode --perf
--concurrency 1,2,4): quality 90 (158/176), tg 27.1/34.6/**37.2** c1/c2/c4,
TTFT d0 1.3 s, responsiveness 25 (6.2 s median), deployability 70.
vs our lab baseline (92 = 162/176; c1 28.4 / c2 40.5; resp 21, deploy 72):
quality −2, c2 decode −15%, c1 ≈, but c4 decode 37.2 in an arm the
baseline never ran, and responsiveness +4. On OUR stack the 0rand combo
is not a strict win; keep the baseline as the daily driver, keep the
variant in tree for concurrency-hungry serving. Possible confounder:
their run posts a single 14.8 s median turn at responsiveness 8 — an
earlier bench build. Report:
/tmp/runs/2026/08/2026-08-30T19-33-22.444571Z_4616f476.md.

## MNBT 1024-vs-2048 assertion on the lab quant (2026-08-30/31 night)

Forum #124 ("1024 = quality-first") asserted. Our c1,2,4 run with
MNBT 1024 + seqs 16 + GMU 0.89 + auto-KV (pool 1,051,792): quality 88,
denominator 146 (harness dropped ~30 pts of scenarios), tg 27.6/27.0/30.4,
TTFT d0 8.6 s, resp 24 (6.4 s). vs the same-protocol MNBT-2048 run one
hour earlier: quality 90 (158/176, full denominator), tg 27.1/34.6/37.2,
resp 25 (6.2 s). Verdict: at our protocol (reasoning_effort-low, temp 0,
seed 42) **2048 wins** on both quality percentage and decode; the 1024
advantage was measured under their thinking-max + temp-0.1 + trials-2
protocol. 0rand's #123 already noted "2048 batch + MTP4 ... loses
quality (89 vs 93)" for THEIR protocol — different regime, same lesson:
batch tuning is protocol-dependent. Variant stays in tree; record both.
Reports: 2048 = ...19-33-22Z_4616f476.md, 1024 = ...22-04-48Z_4a70e6be.md.

## DeepSeek-V4-Flash-Vision-Exp lane — campaign state (2026-08-31 night)

Day-0 port landed (sfxnz plugin @7a958128b84a vendored under
docker-images/dsv4-flash-vision/, bases digest-pinned; sm121 build works,
35-40 min roundtrip). Lane verified TEXT-serving on ds4 (+NVMe/NAS weights):
- eugr b12x base: 1024-token gen in 29.4 s = 34.9 tok/s with DSpark-6
  (accept 0.714, length 2.55), text math exact, plugin loads ViT+aligner
  weights (whole checkpoint present, 82.28 GiB/node).
- fixing their gaps so far: plugin __init__.py must ship the general-
  plugins `register` (raw.githubusercontent flaked on re-fetch — fixed via
  the contents API); wrapper class needs `supports_multimodal = True`
  (OpenAI layer refuses image_url otherwise — interfaces.supports_multimodal
  is a getattr check).
- OPEN: OpenAI-image path raises "Model class ... has no registered
  multimodal processor" — the day-0 plugin never registers a
  _processor_factory (their shipped tests exercise the loader directly,
  not the API). Two paths: implement the BaseMultiModalProcessor inside
  our plugin (grid-tokens-per-image machinery exists in
  dsv4_vision.image_processor; fork interface = MultimodalRegistry.
  register_processor factory in multimodal/registry.py), or wait for the
  upstream repo to ship it. aiden-fork A/B build launched.

## dsv4vision MM-into-OpenAI campaign — 2026-09-01 checkpoint

Built the full OpenAI multimodal stack in our plugin (processor trio,
grid/sentinel tokens, embed_multimodal + embed_input_ids merge; verified in
-image: embedding rows byte-exact vs aligner(ViT(patches))[perm]) and got the
engine through profiling+warmup with mm registered. Two hard walls remain
(before the first image_url token):

1. The fork's V2 "always-embed" mm arm passes `input_batch.input_ids[:n]` —
   a wrong-rank slice that arrives as [B, T]-shaped 2-D ids (observed
   (17,17) for a 17-token prompt, (7,7) on DSpark-6 verify steps) — and
   downstream logits degenerate (all-`<｜begin▁of▁sentence｜>`).
2. The AOT/dynamo trace of inputs_embeds freezes the dummy's 2-D shape and
   then asserts "wrong number of dimensions3 for op: input" on the native
   3-D stream (tilelang mhc kernels are rank-strict both ways).

State: the whole mm branch is env-gated (`VLLM_DSV4_MM=1`), default OFF =
the verified text-only lane — restored EXACT on the new per-node NVMe
weights (~/dsv4vision-hf on dgx3+dgx4, 167 GiB each, recipe HF_HOME flipped
from the NAS hub; shard load 7 min → <1 min, boot ~6 min end-to-end).
Bridges done tonight for the next attempt: encoder-runner + speculator
buffer shape-follow patches (patch_mm_hc_streams.py), placeholder
`<｜deepseek_image｜>` = in-vocab id 129264, `get_placeholder_str` chat
integration, AOT=0 knob for mm experiments. nid: fork A/B (aiden base)
still pending its pip-layout patch; dspark_gate patch now auto-finds the
fork's venv root.

### dsv4vision tool-eval-bench (c1, 2026-09-01)

`tool-eval-bench run --base-url http://dgx3.lan:30000/v1 --model deepseek-v4-flash-vision --seed 42 --hardmode --perf --concurrency 1` → report `/private/tmp/runs/2026/09/2026-09-01T17-46-28.764372Z_89b18250.md`:

| recipe | quality | points | decode tg t/s c1 (d0/d4096/d8192) | pp2048 t/s | TTFT | deployability |
|---|---|---|---|---|---|---|
| `dsv4-flash-vision` (text lane, local-NVMe, DSpark-6) | **81** | 143/176 | — | — | — | — |

perf block: tg 34.1 / 31.0 / 29.1, pp2048 3,907 / 2,714 / 2,449, TTFT 1.9-4.9 s (spec),
scenario-median turn 2.4 s, deployability **74** (best of any ds4 lane) — long-context
capability is the raw 1M-context draw. 3 safety-critical failures (TC-42 extra-param,
TC-43 empty required param, TC-60 sleeper injection) — the day-0 0731-style tool
formatting is the gap vs the GLM lanes (thinking-pair JSON fencing was already a
known day-0 weakness). Quality 81 < sparkrun 89 < lab 92.

## New 2026-09-01 checkpoints: Intel W4A16-AutoRound + lab NVFP4-Spark

Checkpoints on the NAS hub (hf download'd):
- `Intel/GLM-5.3-Flash-W4A16-AutoRound` rev 5eee1846f0321058ed73745f9aa16f2aaf0fc0a0 —
  AutoRound 0.15 W4A16 sym int4 g168, 99.84% of BF16 (GSM8K .9712/MMLU .8620),
  self_attn/hc_/indexer/embed lm_head FP16 by design (their ignore_layers),
  pure GPTQ packing (qweight/qzeros/scales — qweight count 37152).
- `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` rev 06972cdee7280cec849aedf05a02062b3e870246 —
  NVFP4 W4A4 *whole network* (in_proj_qkvbfg_a.weight_scale etc.) + NVFP4 MTP
  expert layer in separate model-mtp-0000x shards (lil.yaml: mtp_tokens 5,
  dflash2_tokens 7, default_speculator mtp, block 256, disable flashinfer
  autotune, INSTANTTENSOR env sizes; arch Glm5NextForConditionalGeneration +
  vision blocks in the index). This is the rtx6kpro r12-stack checkpoint
  (their doc: MTP:3 254 tok/s CC1 @ TP4, Sieve 333-403).

Recipes added: `recipes/glm53-flash-intel-w4a16` (timothystewart6/vllm-gb10
v0.28.0-gb10.2 arm64, gptq_marlin, entrypoint declared — tini AssertionError
 taught us the image has ENTRYPOINT []+CMD bash), `recipes/glm53-flash-nvfp4-spark`
(lab image). Boot outcomes:
- Intel lane died in the fork's Transformers-backend weight loader:
  "no module or parameter named 'layers.0.self_attn.q_conv1d'" — the
  tjstewart image has NO native glm5_next arch (vllm archs list lacks it;
  transformers 5.16.1 has Glm5NextForConditionalGeneration) → needs the
  local-inference-lab fork (native glm5next) or a transformers name-shim mod.
- NVFP4-Spark on our lab fork: KeyError on `in_proj_qkvbfg_a.weight_scale` at
  80% shard load — whole-network NVFP4 W4A4 is beyond the R8-era lab fork
  loader (r12 stack territory).

⇒ THE unifying next step (user directive): build our own arm64/sm_121a image
of local-inference-lab's fork — source contract from models/glm-5.3-flash.md
(local-inference-lab/vllm dev/jovian-judgement + b12x master + 6/5 PRs; kk3
build rules apply: CUDA 13.0.88, sm_121f gencode rule for block-scaled FP4).
RTX docs archive: /tmp/lil-docs (ds4-flash-v6, ds4dspark-v20-r33, infernal
r21, pcie-autotune) for the DeepSeek quality track.

## 2026-09-02: r12 fork built + NVFP4-Spark serving on it (the day's win)

Built `glm53-jovian-r12:local` in-docker on dgx3 (docker-images/glm53-jovian-r12/):
vllm 0.1.dev20489+gcc161e220 (local-inference-lab/vllm base 54f6e9826c + PR heads
550/552/553/554/555/557 merged in order — the doc's "composition commit" 7d8a09c is
not a public ref) + b12x 1.3.0 (master 139e0405 + PR heads 260-270), on the day-0
`vllm/vllm-openai:glm53-flash-arm64-cu130@905c02…` base. Build fixes applied along
the way: apt git, setuptools-rust, ninja/cmake, cu13 wheel include-header symlinks
(cusparse.h), unversioned libnvrtc.so symlink for the deepgemm_C link step.

NVFP4-Spark boot on it (recipe glm53-flash-nvfp4-spark, retargeted):
- whole-network NVFP4 W4A4 (incl. attention in_proj_qkvbfg scales) now LOADS —
  the r12 fork's modelopt handles it (our lab fork: KeyError).
- Attention wall solved: the sm12x auto-priority list only offers
  [TRITON_MLA (no sparse), FLASHINFER_MLA_SPARSE_SM120 (head 576 only)] — neither
  fits GLM head-512. Fix = `--attention-backend B12X` (model hook in
  glm5next/nvidia/attention.py: _select_sparse_backend) for the main model, and
  `"attention_backend":"B12X"` INSIDE --speculative-config for the MTP draft
  (eagle/utils.py: _make_eagle_draft_vllm_config takes the draft's backend from
  the speculative config, not the global flag — user's hint).
- Served: exact "391" verified, MTP-3 Glm5NextMTPModel resolved, KV fp8_ds_mla
  default, 2-node TP2. c1 bench running.
- Intel W4A16 recipe retargeted to the same image (gptq_marlin + B12X attn).
- patch_cuda_sm12x.py (B12X into sm12x priorities) staged in the image dir for
  the next rebuild — the image-baked version of the same fix.

## 2026-09-02 (cont): bench + clean rebuild per operator direction

- NVFP4-Spark (r12 fork image, day-0 base) c1 bench: **88/100** (120/136,
  default 69-scenario suite; deployability 67, median turn 8.1s — TP2 GB10
  is slower per-turn than the eugr vision lane's 2.2s on the same suite,
  also 88). Same-suite A/B vs vavrusa lane: 88 = 88. The lab's 92 was the
  88-scenario/176-point suite — that invocation's re-run is pending.
- Operator: docker_image_dir in the spark recipe pointed at the lab dir —
  fixed to glm53-jovian-r12 (intel recipe same).
- Operator: "start with cuda image, solar-open2 pattern but r12-specific".
  Dockerfile/versions.env rewritten: raw nvidia/cuda:13.2.0-devel-ubuntu24.04
  (a5b6256e…, solar-pinned) → apt stage (solar locks) → torch 2.13.0 cu130 +
  flashinfer 0.6.17 (whl index) → vllm wheel (fork base 54f6e9826c + PR-head
  merges 550-557, use_existing_torch, ccache) → b12x wheel (master + PR heads
  260-270, pure-python) → runner installs both wheels + deps, bakes
  patch_cuda_sm12x.py, GB10 gate (vllm/b12x/_flashkda_C imports, B12X enum,
  patch presence). No day-0 vllm contamination; ENTRYPOINT [] CMD bash.
  Build running on dgx3 (tmux r12build → /tmp/r12build2.log), same tag
  glm53-jovian-r12:local (running serve keeps the old image; re-verify lane
  after build).

## 2026-09-02 (cont 2): lil — the lab's own launcher, read

local-inference-lab/lil = standalone Go launcher (their sm121 equivalent):
reads each HF model repo's lil.yaml at runtime + embedded family policy +
discovered topology (local or spark-rdma via SSH), renders/validates/runs a
typed vLLM invocation. `lil render <model>` prints the exact env + command.

GLM family policy (configs/models/_bases.yaml) — the lab's canonical
launch for these checkpoints, vs our spark recipe today:
- --attention-backend B12X, --linear-backend b12x, --moe-backend b12x
  (we have only the global attention flag; linear/moe backends MISSING)
- spec config: {"method":"mtp","num_speculative_tokens":5 (lil.yaml
  mtp_tokens), "moe_backend":<quant-mapped: nvfp4→b12x, mxfp8/bf16→triton>,
  "attention_backend":"B12X"} — we use k=3 and no moe_backend. MTP:3 is the
  r12 x86-qualified mode; the checkpoint's manifest default is 5.
- --load-format instanttensor (+INSTANTTENSOR_BUFFER_SIZE/CHUNK_SIZE env)
- --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096,
  --max-model-len auto
- --reasoning-parser glm45, --tool-call-parser glm47
- DFlash2 (the GLM-side second speculator, method "dflash") needs the
  separate draft checkpoint local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8.

TODO: after the clean image lands, align the spark recipe to the lil policy
(moe/linear backend b12x, spec moe_backend, instanttensor, mamba-cache-mode,
parsers) and A/B k=3 vs k=5.

## Clean r12 build: fix log (builds 2-9)
1. flashinfer: whl-index path dead → flashinfer-python 0.6.17 from PyPI
   (fork's setup.py skips flashinfer-cubin at build; runtime fetches cubins).
2. git protocol v2 flake in-container → GIT_PROTOCOL=0 + protocol.version 0.
3. setuptools: fork build needs setuptools-rust + setuptools-scm (uv hint).
4. DeepGEMM FetchContent: pinned SHA on non-default nv_dev branch → GitHub
   404s raw-SHA fetch → DEEPGEMM_SRC_DIR pre-clone stage (cmake escape hatch).
5. PR 550 closed (unmerged) → refs/pull/550/head deleted; all PR heads now
   pinned by commit SHA (vllm 550-557, b12x 260-270). Raw-SHA fetches
   verified working.
6. Operator rule: no build while a serve container runs (OOM risk on 128 GB
   unified) — spark lane stopped for the build window.

## 2026-09-02 (cont 3): clean r12 image SHIPPED

Image glm53-jovian-r12:local rebuilt from raw nvidia/cuda:13.2.0-devel
(solar pattern, r12-specific content): 16.1 GB (vs 37.6 GB day-0-based),
gates green (vllm 0.1.dev20489+g06f922207, b12x, _flashkda_C, B12X enum,
sm121 patch), synced dgx3→dgx4.
Boot fixes found on first clean boot:
- tini "[FATAL] exec local-inference-lab/… failed" → clean image is
  ENTRYPOINT [] (solar contract), so the RECIPE now declares
  entrypoint = ["vllm","serve"] (the old day-0 image had it baked in).
- FlashInfer JIT OOM: 12 parallel nvcc jobs (mxfp8_gemm_cutlass_sm120 etc.)
  get OOM-killed (code 137) while 93 GB weights + 10 GB KV are resident on
  121 GB unified → MAX_JOBS=2 in recipe [env]; JIT (~15 min) then survives.
  Boot total ~35 min (NFS shards 16 min + JIT 15 min).
c1 bench (default 69-scenario suite, same as the vavrusa run): **90/100**
(124/138; quality 90, deployability 69, median 7.9 s) — vs 88 on the
day-0-based image, same suite. Clean image is now the reference GLM lane.

Remaining lab-alignment (next boot, one change-set at a time):
--linear-backend b12x + --moe-backend b12x, spec moe_backend:b12x,
A/B k=3 vs k=5 (lil default), --load-format instanttensor (+env),
--mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096,
--reasoning-parser glm45, --tool-call-parser glm47.
Then: Intel W4A16 boot on the clean image (queued, needs cluster).

## 2026-09-02 (cont 4): lab-stack perf push — eugr build pipeline adopted

Goal (user): 30 tok/s in --perf. Found the missing pieces:
1. **lil render reference** (built lil from local-inference-lab/lil @ cb58d54,
   rendered with testdata manifest GLM-5.3-Flash-NVFP4-Spark + spark.yaml):
   our recipe flags now 100% match the family policy (_bases.yaml glm: b12x
   linear/moe, B12X attention, instanttensor, block 256, seqs 8, batched 4096,
   glm45/glm47, mamba align) + spec {mtp:5, moe_backend b12x, B12X} + env
   VLLM_USE_AOT_COMPILE=1 / V2_MODEL_RUNNER=1 / FLASHINFER_SAMPLER=1 /
   BREAKABLE_CUDAGRAPH=0. Boot 35m -> ~7m (instanttensor loader, user: keep).
   Perf after engine switch: d0 c1 24.2 / c4 31.8 tok/s (was 10.3/16.5).
   Quality dropped with full lil set (16/22 at TC-22) -> bisection in
   progress; user: instanttensor is fine, problem elsewhere.
2. **eugr/spark-vllm-docker** (the lab's actual build repo, main): 804-line
   Dockerfile, CUDA 13.0.2 base, 12 build-time vllm patches (docker/patch_vllm_*.py),
   b12x from source (lukealonso/b12x; lab: local-inference-lab/b12x master),
   nightly DockerHub images eugr/spark-vllm[-b12x]:latest (stale 08-23,
   user: dockerhub is the wrong source — build from source).
   Ported all 12 patches into docker-images/glm53-jovian-r12/patches-eugr/
   + Dockerfile (VLLM_PRESERVE_SM12X_TARGET=1, VLLM_PATCH_B12X_C128A_ALIGNMENT=1;
   patches are idempotent, several no-op on our r12 ref).
   Build blocker: node github.com access flaky ("could not read Username" on
   FetchContent clones) -> pre-fetched all 9 sub-deps on the Mac into
   deps-src.tar.gz (triton_kernels@v3.5.1, MSA, FlashMLA, FlashKDA,
   flash-attention, qutlass@e74319e, tml-fa4@b206834, cutlass@v4.4.2) +
   the SRC_DIR env escapes each cmake documents. Build 14 compiling.

## r17 (20260902-r17 card) adoption + quality bisect (2026-09-03)

Latest lab card: https://github.com/local-inference-lab/rtx6kpro/blob/master/models/glm-5.3-flash.md
(names the qualified serving contract per artifact and SHA; TP4 on RTX PRO,
not Spark — the Spark lil.yaml testdata render remains our TP2 adapter).

1. **Public r17 composition** built from source:
   `/tmp/vllmr12` = local-inference-lab/vllm `dev/jovian-judgement@9c4dd054`
   + open PRs **579/580/581/582/586** (branch supersedes the r12 PR set
   550–557 — same symbols, evolved, see below) + staged-cutlass cmake patch;
   `/tmp/b12xb17` = local-inference-lab/b12x `master@2bbab479` + PRs
   **290/284**. Pre-staged as vllm-src.tar.gz / b12x-src.tar.gz on the build
   node (WAF blocks node→github git). DO NOT re-merge 550-557: creates
   conflict-marker SyntaxErrors (pr550's pooled_indexer.py conflicts; branch
   tip already contains the evolved implementation).

2. **Important correction to the 2026-08-28 eager verdict**: on the public
   r12/r17 refs, graphs+MTP is no longer a quality bug — the lab's private
   "packed GLM-5.3 package" (card SHAs 47f86087/a02841bc, not public) was the
   presumed fix carrier, but our public-tree composition measured:
   - r17 + graphs + 90/100-flags (no b12x/align): **88/100 hardmode (155/176)
     == lab-quant's own day-0 reference 156/176** (r17graphs01/…perf)
   - r17 + eager same-flags: 67/100, median turn 11.4 s, 2 scenarios timed
     out and excluded (r12eager01) — eager is now the WORST mode.
   So: graphs stay ON; enforce-eager is gone.

3. **Execution-env knobs are quality-clean**: GMU 0.90 (0.95 boot-deaths on
   GB10 — "free memory 111.96 < desired"), max-num-seqs 16, OMP 2,
   VLLM_USE_STANDALONE_COMPILE/FLASHINFER_SAMPLER/BREAKABLE envs: 87/100
   (r17knobsonly). AOT=1+V2 runner: NEUTRAL for quality (47 either way when
   backends were on) and NEUTRAL for the d0 grid — dropped (AOT=0, no V2).

4. **THE QUALITY TOXIN = the b12x backend-flag cluster** (on the PUBLIC
   r17/r12 compositions, with our b12x 1.3.0/1.2.6 builds):
   - k=5 full cluster: 74/100 (r12 img), ~74 → killed-to-16/22 (r12 img)
   - k=5 full cluster (r17 img): ~70-ish band (r17cluster01, stopped early)
   - k=3 full cluster: **46/100** (r17clusterk3)
   - k=3 cluster minus AOT/V2: **47/100** (r17noaotk3)
   - k=3 cluster minus backends (knobs only): **87/100** (r17knobsonly)
   The lab's own stack runs these backends qualified — but their composition
   carries the private packed GLM-5.3 package (RTX-PRO target); the public
   branch tip alone does NOT make the b12x W4A4/linear/align paths
   quality-safe. Also: b12x cluster wrecked deep-context speed
   (d4096 c1 12.6 @k5) while helping d0 (33-34 c1) — nets negative anyway.

5. **SHIP ANCHOR (current recipe)** = r17 image (94ac0882a8b7) + 90/100
   args (attention B12X, block 256, seqs 16, batched 4096, LM-only,
   glm47/glm45 parsers, instanttensor load, k=3 spec {mtp,attn B12X} with
   moe_backend key) + graphs (no eager) + knobs env (GMU 0.90, seqs 16,
   OMP 2, AOT=0, STANDALONE=1, SAMPLER=1, BREAKABLE=0) + JIT-cache mounts:
   **87/100 hardmode (153/176; lab-quant ref 156/176), default-suite 88
   class**; perf d0 c1 24.1 / c2 30.2 / c4 44.7; d4096 c1 23.9; d8192 c1
   23.8; pp ~1.6-1.8k. Boot 8 min (JIT warm) / ~25 min cold (nvcc re-JIT).

6. Boot-time JIT: changing the rw flashinfer cache invalidates per-config
   kernel sets; a config's first boot silently nvcc-compiles (MAX_JOBS=2)
   ~15-25 min inside "No available shared memory" log-silence; later boots
   of the same config are fast. Expect this on every NEW kernel path.

7. KV-dtype probes: fp8_ds_mla on r12-img = 75/100 default-suite (net zero,
   TC-45 flipped PASS but others flipped off); graphs here. r17 anchor keeps
   default dtype. Not a quality lever.

8. Open perf gap: c1 24 vs manifest-cluster 33-34 (which is quality-dead).
   If the lab ever publishes the packed package (or a public branch commit
   makes b12x-backends clean), the 30+ c1 target lands with manifest flags.
   Alternative to probe later: one b12x backend flag at a time (a single
   toxin member may be free).

## One-backend bisect (2026-09-03) — SOLVED (hardmode, r17 img 94ac0882, k=3 graphs)
- knobs-only: 87 (153/176); d0 c1 24.1, d4096 23.9, d8192 23.8 (r17knobsonly)
- + `--moe-backend b12x` (spec default MTP experts): **88 (154/176)**; d0 23.5,
  d4096 22.5, d8192 **26.6** (r17moeonly) — quality-clean, keep
- + `--linear-backend b12x`: **44 (77/176)**, Parameter Precision 0%, TC-21
  0/5 validations — THE toxin. B12X linear kernel path destroys numerics on
  the public composition (r17linearon)
- + `--mamba-cache-mode align`: 88 (154/176) but d0 c1 18.6 (−4.9), no quality
  gain (r17alignon) — perf-negative, dropped
- Full-cluster combos (44-74) were dominated by linear; spec's moe_backend b12x
  key and the knobs env family are all exonerated individually.
**SHIP = r17 img + 90/100 args + graphs + knobs env + `--moe-backend b12x`
only** (recipe @ 2d379f9): 88/100, d0 23.5 / c4 43.9, depth 22.5/26.6.
The 30 c1 target is only reachable with linear-b12x (29.6 d0) which is
quality-dead until the lab publishes the packed GLM-5.3 package.

## Checkpoint A/B on ds4 (2026-09-03, hardmode 176pt): Spark vs NVFP4 lab checkpoint
User-flagged must-have: our lane served the Spark-variant checkpoint with quality
losses vs lab references; A/B the checkpoint under matched stacks.

| Cell | Stack | Checkpoint | Hardmode | d0 c1 | d4096 c1 | d8192 c1 | notes |
|---|---|---|---|---|---|---|---|
| r17ablabimg | lab img + lab recipe (eager, fp8_ds_mla, k3, default MoE, day-0 fork 0.1.dev20051+g487ecf187) | NVFP4 | **93 (164/176)** ★★★★★ | 24.4 | 26.5 | 22.8 | 78 pass / 2 fail; TC-45 PASS; matches lab's own 92 ref |
| r17ablabck | **shipped r17 img** + ship flags (graphs, knobs, k3, default MoE — moe-b12x unsupported on this ckpt: layer-45 is MXFP8) | NVFP4 | **88 (154/176)** | **27.5** | **29.6** | 26.6 | 46.4/46.7 c2/c4 at d0 |
| r17moeonly (ship) | shipped r17 img | Spark | 88 (154/176) | 23.5 | 22.5 | 26.6 | — |

Verdict:
1. **Checkpoint is not the quality factor on the r17 stack** (88 = 88 both
   checkpoints) — but the NVFP4 lab checkpoint is significantly FASTER
   (+4 tok/s d0, +7 d4096) and lighter rules (no --moe-backend b12x).
2. **The lab day-0 stack retains +5 quality (93 vs 88) regardless of
   checkpoint** — its eager + fp8_ds_mla + default-everything runtime is what
   the lab's own 92-hardmode reference was measured on. Enforce-eager costs
   prefill pp at c2/c4 (0.9-1.1k vs 1.5-1.6k) but TG is equal-or-better.
3. Recipes: `glm53-flash-nvfp4-r17abi` (ship flags + NVFP4, moe flag removed)
   is the fastest quality-parity lane; `glm53-flash-nvfp4-lab` remains the
   quality champion (93).

## Night-shift queue (2026-09-05)

Prepared lanes + builds queued for the night (cluster is serving the 4x
aiden lane during the day — stop it first, then all 4 nodes are free).
Order = value/duration; each item lists prereqs + expected verdict.

1. **r18 fork build** (docker-images/glm53-jovian-r12, ~1-2 h build).
   Stage on the build node: /tmp/vllm-src.tar.gz (94 MB) +
   /tmp/b12x-src.tar.gz (29 MB) — pins per versions.env (vllm
   a8c796f3+pr582+pr646 → 343661fe; b12x b58f34ea+pr284 → 1adb8a38),
   then
   `TAG=glm53-jovian-r12:local bash build.sh`. Test campaign:
   (a) **Re-baseline the quality anchor first** — DONE 2026-09-06: ran
   glm53-flash-nvfp4-r17abi (2x, r17 image) on 2nodes-dac with the current
   tool: **NEW BASELINE = 153/176 (87/100, deployability 69)** — report in
   recipes/glm53-flash-nvfp4-r17abi/reports/2026-09-06_r18-rebaseline-r17abi-2x.md
   (2 pts below the historical 155/176 anchor: DAC fabric + NAS weights;
   scoring confirmed current-tool-compatible: 88 scenarios × 2 pts = 176).
   Gate ALL new lanes against 153/176 (≥151 = parity, <151 = regression);
   (b) **re-bisect `--linear-backend b12x`** — new b12x A16 precision
   selection may clear the 44/100 toxin (29.6 c1 unlock if clean),
   (c) RoCE one-shot on the 4x lane (VLLM_ENABLE_ROCE_ALLREDUCE=1;
   expect −5–19% decode per-step per their #597 numbers),
   (d) `--kv-cache-dtype nvfp4_ds_mla` probe (new B12X KV path),
   (e) #646 geometry check: 2048/256/256 split blocks selected +
   repeat-request prefix resume at the fine boundary (their regression
   scenario: retention 0, states 32/16, resume at 32) — if broken, drop
   the 646 pin from versions.env and re-cut (rest of r18 is independent).
   1b. **DCP4 probe** (4x lane, no build dependency — run on the current
   r17abi image first, re-run on r18 if positive). Add
   `--decode-context-parallel-size 4` (stock vLLM flag, verified in the
   r18 tree: vllm/config/parallel.py; DCP>1 requires MLA — GLM53
   qualifies). Reference (sparkring MTP3 mesh profile, 4 Sparks, DCP4 +
   24 GiB fp8 KV/rank): ~4.32M logical KV tokens (vs our ~1.25M at
   12 GiB DCP1), 231.3 aggregate decode tok/s peak, prefill ~2.7k flat
   8K→128K. Verdict: KV-token capacity per node at our pin, decode
   C1/C8/C16 vs baseline, prefill 8K/64K. Their fabric is a direct
   HW-forwarded mesh (cheaper all-reduce than our switch) — measure, do
   not extrapolate.
2. **RedHatAI MTP lane** (2x, ~30 min). Re-run `huggingface-cli download
   RedHatAI/GLM-5.3-Flash-NVFP4` on both nodes (model_mtp.safetensors
   7.6 GB is new since the initial pulls; HF_HUB_OFFLINE=1). Serve
   glm53-flash-nvfp4-redhat (MTP k=5 per vendor card) → compare vs the
   90/100 no-MTP baseline + decode speed; memory watch +7.6 GB/node
   (KV pin 10→8 GiB if boot dies; k=3 fallback).
3. **eugr b12x image verify + GLM lane** (2x, ~1 h). Pull
   eugr/spark-vllm-b12x:nightly-20260904 (10.9 GB) on the 2 nodes, read
   /workspace/build-metadata.yaml + /workspace/b12x-source-commit (record
   exact pin in the recipe header). `huggingface-cli download
   local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` on both nodes. Serve
   glm53-flash-nvfp4-eugr → A/B vs r18: full b12x cluster quality
   (linear-b12x toxin retest on their stack) + RoCE 2x decode numbers.
   4-node cell: glm53-flash-nvfp4-eugr-4x (needs all 4 nodes free) vs
   the 2x lane — 2x stays the production default unless 4x is dramatic.
   3b. **DFlash2 spec A/B** (same eugr image, after 3 boots). Swap the
   spec to `"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2",
   "num_speculative_tokens":7,"draft_tensor_parallel_size":2,
   "kv_cache_dtype":"auto"` (separate drafter repo — huggingface-cli
   download on both nodes first; exists on HF, single safetensors). VS
   the recipe's MTP k=5: decode at C1/C4 + quality spot-check (forum
   numbers are perf-only: ~60 c1 / ~73 c4 tok/s on 4x — their 4x/TP4
   config, so 2x numbers will differ).
4. **DSV4F-Vision eugr lane** (2x, prefer dgx3/dgx4 — weight reuse).
   Reuse the per-node ~/dsv4vision-hf copies (verify the HF_HOME mapping
   vs the recipe's default HF cache) or re-download Vision-Exp (168 GB).
   Serve dsv4-flash-vision-eugr → compare vs the fork lane
   (dsv4-flash-vision, DSpark-6): quality (text + vision smoke) + decode
   + TTFT; spec A/B: k=3 adaptive (vendor card) vs k=6 (fork lane's
   proven value).

## Night-shift results (2026-09-05/06, all seed 42, hardmode, c1,2)

**Re-baseline (queue 1a):** r17abi 2x/2nodes-dac = **153/176 (87/100, dep 69)**
with tool-eval-bench 2.6.1.dev45+gcf54b4bfe (88 scenarios). This is the new
canonical anchor — all lanes below gate against it.
Report: `recipes/glm53-flash-nvfp4-r17abi/reports/`.

**r18 build** (5 attempts on dgx4, log r18build*.log): three root causes —
(1) transient `snapshot.ubuntu.com` 503s; (2) `patch_vllm_spark_kv_cache_cleanup.py`
anchor gone (upstream replaced the profiling flow with
`MemorySnapshot`/`request_memory` which already cleans before the snapshot —
patch now detects the new flow and exits 0); (3) **staged-deps submodule
stubs**: both `flash-attention/csrc/cutlass` AND `MSA/python/fmha_sm100/cutlass`
were empty dirs in `deps-src.tar.gz` (the r18 tree dropped implicit cutlass
include wiring / MSA's install needs the dir) — populated with each project's
own pin (FA: nvidia/cutlass `62750a2b`, MSA: nvidia/cutlass `eb61c911`) and
purged 23k macOS AppleDouble `._*` files. Image: tag `glm53-jovian-r12:local`,
tree `67e7e6e` (base `a8c796f3` + #582 + #646), 16.1 GB, on all 4 nodes.

**Lane results** (gate = 153/176 baseline; quality is the gate, dep = deployability):

| Lane | /176 | /100 | dep | KV tokens | verdict |
|------|------|------|-----|-----------|---------|
| r17abi 2x (baseline) | 153 | 87 | 69 | ~1.25M | anchor |
| r18 2x (ds4) | 157 | 89 | 71 | — | PASS (+4) |
| RedHat MTP k=5 2x | 156 | 89 | 70 | — | PASS (MTP works, +3) |
| eugr b12x MTP k=5 2x | 143 | 81 | 65 | — | FAIL (−10) |
| r18 4x RoCE TP4 | 156 | 89 | 76 | 1.47M | PASS |
| r18 4x **DCP4** (TP4+`--decode-context-parallel-size 4`+`--cp-kv-cache-interleave-size 4`) | **161** | **91** | 75 | **4.84M** | **BEST** |
| r18 4x **nvfp4_ds_mla KV** | **161** | **91** | **77** | 2.35M | PASS (best dep) |

**--perf results** (llama-benchy, pp2048/tg128, runs=3, from each report's
Throughput Metrics table; c1 = single, c2 = 2 concurrent; t/s = tokens/s):

| Lane (img) | pp c1 | tg c1 | TTFT c1 | pp c2 | tg c2 | TTFT c2 | tg c2 @8k-ctx |
|---|---:|---:|---:|---:|---:|---:|---:|
| r17abi 2x (baseline, r17) | 1,577 | 29.0 | 1,362ms | 1,595 | 41.7 | 2,093ms | 29.7 |
| r18 2x (r18) | 1,473 | 30.7 | 1,481ms | 1,374 | 40.2 | 2,268ms | 30.6 |
| RedHat MTP k=5 2x (lab) | 1,340 | 26.8 | 1,638ms | **460** | 22.4 | 4,667ms | 14.7 |
| eugr MTP k=5 2x (eugr) | 1,292 | 31.1 | 2,397ms | 1,388 | 41.6 | 2,751ms | 17.7 |
| r18 4x RoCE (r18) | **2,904** | 47.7 | **762ms** | 2,413 | **70.9** | 1,399ms | 35.9 |
| r18 4x DCP4 (r18) | 1,639 | 43.1 | 1,521ms | 1,756 | 51.1 | 1,637ms | 30.0 |
| r18 4x nvfp4 KV (r18) | 2,614 | 45.3 | 873ms | 2,250 | 62.6 | 1,368ms | 35.6 |

Perf readings:
- **4x RoCE is the throughput king**: 2× prefill of 2x (2.9k vs 1.6k), 1.7×
  decode (70.9 c2 vs 41.7), half the TTFT (762ms). 4x wins on both axes vs
  2x — the case for the 4x production cell.
- **DCP4 pays a prefill tax**: 1.6k pp (vs 2.9k RoCE) — the DCP ring
  exchange costs prefill; decode is comparable (43–51). Its edge is the
  4.84M KV capacity + quality, not raw throughput.
- **nvfp4 KV ≈ RoCE** (2.6k pp, 62.6 c2): the KV dtype costs nothing
  measurable — combine candidates: DCP4 + nvfp4 KV.
- **RedHat MTP k=5 collapses at c2**: pp 460 t/s (vs ~1.4–1.6k everywhere
  else), TTFT 4.7s — quality is fine (156) but the lab image's MTP-k=5 path
  is perf-weak under concurrency. r18 2x dominates it on both axes.
- **eugr 2x**: c1/c2 short-ctx throughput matches the baseline, but decode
  degrades at depth (15.8–17.7 t/s @4k/8k-ctx vs 28–30) — consistent with
  the quality failure (81/100).

Key findings:
- **DCP4 works on our switch fabric** (sparkring steal confirmed): 4.84M KV
  tokens at the SAME 10 GiB/node pin — 3.3× plain 4x, and beats their
  4.32M @ 24 GiB/rank (their HW mesh gave them 231 t/s aggregate; our
  RoCE switch numbers are in the report). Best quality of the night (91).
  B12X is NOT DCP-compatible (engine auto-falls the DCP cell to FLASH_ATTN
  attention; that's what benched).
- **`nvfp4_ds_mla` KV dtype** (r18-tree value, `vllm/config/cache.py`):
  1.59× the fp8 capacity at the same pin, tied quality, best deployability.
  Combine with DCP4 next (projected ~7.7M KV).
- **#646 geometry live** (queue 1e): serve logs show
  "Setting attention block size to 2304" = the 2048+256 split-block
  geometry; no retention/states anomalies seen, no fallback needed.
- **eugr image lane fails the quality gate** (143 vs 153) on 2x — the
  linear-b12x toxin persists on their full b12x cluster; eugr-4x cell
  skipped (2x already decides it).

**Blocked (documented, not solvable from recipes):**
- **DFlash2 drafter lane** (queue 3b): the drafter inherits the engine's FP8
  KV dtype; on SM121 `B12X` is rejected for it (kv_cache_dtype constraint)
  and `FLASH_ATTN` too ("FP8 KV cache requires FA3 on SM90 or FA4 on SM100").
  With drafter `kv_cache_dtype: "auto"` it fails the FORK's own assertion:
  `ValueError: GLM MLA parent-page stride must be an exact number of C4
  pages`. Fork-side fix needed (drafter KV geometry under C4 paging) before
  the k=7 A/B can run.
- **dsv4-flash-vision-eugr** (queue 4): the eugr 0904 image's
  `DeepseekV4ForCausalLM` (vllm/models/deepseek_v4/nvidia/model.py) contains
  NO `aligner` module — the image predates V4-Vision support; the Vision-Exp
  checkpoint's vision weights have nowhere to land. Re-check on a newer
  eugr image; fork lane (dsv4-flash-vision) remains the only V4-Vision path.

**Open decisions:** (1) production 4x candidate = r18 + DCP4 (quality 91,
4.84M KV) — needs a production recipe + operator sign-off; (2) r18 2x
adoption as the new 2x default (157 vs 153 baseline); (3) DCP4+nvfp4 KV
combined cell; (4) eugr lanes stay demoted.

## r12/r18 image separation (2026-09-06, operator direction)

The `glm53-jovian-r12:local` tag pointing at r18 content was wrong — separate
now, no mixing:
- `docker-images/glm53-jovian-r18/` — self-contained r18 build dir (r18
  versions.env pins 582+646, fixed kv-cache-cleanup patch, re-staged
  deps-src with FA+MSA cutlass, source tarballs vllm `67e7e6e` / b12x
  `b58f34ea`). Image tag: `glm53-jovian-r18:local` (renamed on all 4 nodes
  from the r12 tag; content unchanged, built during the night shift).
- `docker-images/glm53-jovian-r12/` — restored via `git checkout 582343c`
  to the genuine r17 state (pins 550–557, original kv-cache-cleanup patch).
  Image tag `glm53-jovian-r12:local` currently points at NOTHING (removed
  from nodes) — an r17 rebuild is possible but NOT done: the r17-era
  deps-src staging is unrecoverable (untracked + overwritten), so a rebuild
  today would use the r18 deps staging (FA/MSA cutlass populated) and be a
  subtly different image than the original r17 anchor. If the r17 A/B
  anchor is ever needed again, re-cut the r17-era deps first.
- Recipes: `glm53-flash-nvfp4-r18{,-4x,-4x-dcp4,-4x-nvfp4kv,-dflash2}`
  (new) → r18 tag/dir; `glm53-flash-nvfp4-r17abi` unchanged → r12 tag/dir.
- Source tarballs (vllm-src/b12x-src) are local build artifacts — now
  gitignored; deps-src already was.

## 2x nvfp4-KV + Spark checkpoint extension (2026-09-06, continued)

Operator direction: nvfp4_ds_mla in the 2x setup (parity with the 4x cell)
and a Spark-checkpoint variant of each nvfp4kv cell; 4x KV pin 10 -> 16 GiB
(NVFP4 4x has the headroom: 47.1 GiB weights/node vs ~108 usable).

| Lane (r18 img) | /176 | /100 | dep | KV @ pin | verdict |
|---|---:|---:|---:|---|---|
| 2x nvfp4 KV (non-Spark) | 154 | 88 | 70 | 2.14M @ 10G (1.7x fp8) | PASS, no quality regression |
| 2x nvfp4 KV + **Spark ckpt** | **158** | **90** | 70 | 2.14M @ 10G | **BEST 2x of the campaign** |
| 4x nvfp4 KV + Spark ckpt | — | — | — | (16G) | BLOCKED: mm_mxfp8 TP4 |

- The **Spark checkpoint** (DSpark-embedded quant) beats the non-Spark lab
  quant by +1/176 at 2x (158 vs 157 r18 2x) and by +5 over the anchor —
  candidate for the 2x production default alongside r18.
- **4x + Spark is blocked**: `ValueError: Problem size is not supported for
  mm_mxfp8` — the Spark quant's MXFP8 MTP-expert layer does not tile at TP4
  (non-Spark is fine at 4x). Spark checkpoint => 2x lane only.
- 16 GiB 4x pins: recipes bumped; expected KV: RoCE 2.35M, DCP4 7.7M,
  nvfp4 3.76M. No re-bench needed (capacity-only change vs the 10G benches).
- JIT persistence first measurement: cold serve ~45 min -> **9 min warm**
  (per-node flashinfer/torchinductor/vllm caches; see next section).

## JIT cache persistence (2026-09-06, operator direction)

Every fresh container re-pays ~20 min of flashinfer/torch JIT compilation
(cold `/root/.cache` per container — seen on every lane all night). All glm53
lanes now persist the JIT caches per node (local FS, not NAS) via plain
`[[extra_mounts]]`:
- `~/sm121-cache/jit-121f/flashinfer` → `/root/.cache/flashinfer`
- `~/sm121-cache/jit-121f/torchinductor` → `/root/.cache/torchinductor`
- `~/sm121-cache/jit-121f/vllm` → `/root/.cache/vllm`
Host dirs are SHARED between lanes of the same arch — each cache
self-namespaces by its engine-version key internally (flashinfer keys by
pkg version + arch; torchinductor by torch version; vllm by compile input
hash), so mixing r17/r18/lab/eugr in one dir is safe. The pre-existing
`glm53-jit-121f/b12x` mount is untouched (keeps its accumulated cache).
First serve of each image pays the compile once per node; later serves
should start ~20 min faster.

## Ops layout from 2026-09-06 afternoon (operator direction)

- **2nodes-dac (dgx1+dgx2) = PRODUCTION**: serves the operator's own agent on
  `glm53-flash-nvfp4-r18-2x-nvfp4kv-spark` (158/176). Do NOT stop/re-serve/
  benchmark against it without explicit operator confirmation — treat like a
  live cluster (AGENTS.md warning applies).
- **ds4 (dgx3+dgx4) = DEV lane**: identical recipe serves here as the A/B
  reference for fork/tree improvement work ("improve our 2 node builds").
  Next 4x cells (which need all 4 nodes) require stopping the production
  lane — operator sign-off first.

## Image build v2: clone-at-pin (2026-09-06, operator direction)

Operator verdict: regenerating tarballs on some machine is nonsense (the
sources existed only on that machine — a fresh repo clone could not build).
The r27 build has NO context tarballs:

- `docker-images/glm53-jovian-r27/` — clone-at-pin build: the Dockerfile
  clones vLLM/b12x/LMCache at pinned commits (versions.env `*_PIN` = SHA,
  `*_TREE` = git tree hash, gates assert BOTH), builds sub-deps from the
  pins the tree itself declares (fetch-deps.sh, grep-gated against the
  tree's cmake), builds LMCache + cuMem interposer in-container, and keeps
  the proven GB10 runtime (CUDA 13.2 aarch64 / torch 2.13.0 / flashinfer
  0.6.17 — the same versions the r27 tree's requirements/cuda.txt asks for).
- Fresh-clone reproducibility: `git clone` this repo, upload the 215 KB
  context, `TAG=... bash build.sh` — everything else comes from GitHub at
  pinned SHAs (verified locally end to end; b12x/lmcache fetch-by-SHA works,
  branch-tip gates catch drift — LMCache's lock branch was already renamed
  upstream, so pins are SHAs, branches are comments only).
- Source contract = rtx6kpro r27 source.lock: vLLM
  `integration/glm53-r27-release-20260906` @ 63a82f8d (tree f54cd9ca),
  B12X @ e8ad299b (f3cd8a9e), LMCache @ 822330db (008ac3e0,
  0.5.5.dev0+glm53cache.engine.822330db), FlashKDA @ 3b225bf2, cutlass
  v4.4.2 + FA gitlink 62750a2b + MSA gitlink eb61c911, FlashMLA a8f794d1,
  qutlass e74319e3, tml-fa4 b2068346, triton v3.5.1, DeepGEMM 8b1392b.
  Wheels get deterministic versions via SETUPTOOLS_SCM_PRETEND_VERSION.
- Where r27 lands vs our r18: new in the tree are LMCache (engine-driven
  cache transfers), qualified DFlash2 support (dflash2 depth 7 via env),
  2048-token GPU pages with public block 256 (the #646 geometry, now
  qualified), recurrent-boundary prefix reuse, prefill compute sharing
  (fairness engine), DCP full-cache prefill; eugr patch set retained
  (idempotent, skips patterns that moved upstream).
- Legacy: r12/r18 build dirs are staging-era, not rebuildable (tarballs
  were context-only artifacts). Cleanup candidate once r27 qualifies.

## Night 2 results (2026-09-06→07, tool 2.6.1.dev46, seed 42, hardmode)

Operator switched the agent back to pluto → dgx1–4 all free; dual-bench window.

- **aiden production-1.2 (2x, first canonical aiden number ever): 159/176 (90)**.
  DFlash2 k=7, fp8_e4m3 KV 4 GiB pin (461K-token pool), eager, LibertAIDAI NVFP4
  97 GiB/node. Spec acceptance 4.27–4.94 (healthy). Fails: TC-43/58/61.
  Profile vs the r18-spark anchor (158): +1 quality, −32% prefill (1,271 vs 1,861
  pp c1), ~23 t/s decode, 1.8 s TTFT, 5x less KV headroom. The drafter lane is
  alive in aiden's stack — the r18-fork C4-page assert is fork-specific.
  Report: `recipes/glm53-flash-aiden-2x/reports/2026-09-06_aiden-12-2x-hardmode.*`.
- **r27 lane (2x spark, nvfp4 KV, KV 2,206,618): 156/176 (88) = −2 vs r18**
  on the identical 2x config. r27 prefill ~27% slower (1,356 vs 1,861), decode
  in line (31.4 vs 36.0 tg c2). The r27 tree has NOT beaten the r18 build on
  quality yet; its high-value cells (DFlash2 env-driven spec, DCP4, 16 GiB KV)
  remain the next windows. Fails: TC-33/43/45/76.
  Report: `recipes/glm53-flash-r27-spark/reports/2026-09-06_r27-2x-hardmode.*`.
- r27 image arch fix: attempt-3 build lost the stage-inherited
  TORCH_CUDA_ARCH_LIST=12.1a (buildkit has no GPU; torch autodetect unavailable)
  → sm120-only kernels → cudaErrorNoKernelImageForDevice at boot. Attempt 4
  declares `ENV TORCH_CUDA_ARCH_LIST=12.1a GIT_PROTOCOL=0` in vllm-builder
  explicitly; clean boot + full bench since.

Ops: old `r18-2x-nvfp4kv` containers (left over from the night-1 matrix, up
~1 h on dgx1/2) stopped; 2nodes-dac and ds4 now hold the new lanes.
Next queue: prod-3.73-vision (DSV4F-Vision aiden lane) on 2nodes-dac, then
r27 DFlash2 + DCP4-16G re-tests (need 4x → operator sign-off on the lane swap).

## Night-3 close (2026-09-07, pluto switch, dgx1–4 all free)

Final campaign picture (all seed 42 hardmode, tool 2.6.1.dev46):

| lane | score | verdict |
|---|---|---|
| aiden prod-1.2 2x (DFlash2 k=7) | **159 (90)** | first canonical aiden number; +1 vs r18-spark but 5x less KV, eager profile |
| DSV4F-Vision aiden prod-3.73-vision 2x | **161 (91.5)** | new 2x campaign record; ties best 4x cells |
| r27 2x spark | 156 (88) | −2 vs r18; engine not at parity |
| r27 4x DCP4 16G | 155 (88) | −6 vs r18 161; KV 7.74M (+60% headroom), fast decode (44–53 t/s) |
| r27 DFlash2 2x | — | **BLOCKED**: C4 parent-page stride assert (pooled_indexer, tree-level in r18 AND r27; r27's DFlash2 qualification was sm_120-only) |

Ops: launcher gained two entrypoint-image capabilities (empty `model` → no
positional model; NODE_RANK/MASTER_ADDR env injection) + the r27 image
arch fix (explicit 12.1a in vllm-builder). Hung-worker cleanup pattern
reconfirmed (rank-1 survives a head death; `docker rm -f` by name).

Queue going forward: (1) r27 DFlash2 needs a fork-level patch before re-test
(drafter KV outside the C4-aligned page, or an SM121-capable FP8-KV drafter
backend); (2) r18 16 GiB 4x boot-verify (capacity-only, no re-bench);
(3) production-lane decision remains with the operator (r18-spark 2x vs
r27 watch; aiden 159 as the community reference).

## Aiden 4x + eager A/B (2026-09-07 night close)

- **aiden production-1.2 4x (zai FP8, MTP k=5, eager): 163/176 (92.6) — new
  campaign record** (79p/5partial/4fail; fails TC-43/51/58/85). KV 2.32M
  @ 16 GiB pin. The image needed our extension layer for instanttensor
  (`docker-images/glm53-flash-aiden/`, built + on all 4 nodes). Perf run
  noisy — quality is the headline. The 4x aiden lane is now a real
  production candidate (best quality of the campaign; FP8 306 G).
- **Eager A/B (our r18 2x lane + --enforce-eager): 159 vs 158** — the eager
  quality rumor does not hold on this stack (+1 = noise); perf at parity
  (tg c1 27.1 vs 24.5, TTFT 1.17 vs 1.24 s). Graphs remain the default.

## Rust frontend (vllm-frontend-rs) — evaluated, deferred (2026-09-07)

The r27 tree ships the Rust OpenAI frontend (axum, ZMQ bridge; `vllm-rs`
binary is already in our r27 image, built with the wheel; toggle
`VLLM_USE_RUST_FRONTEND=1`). Smoke test (dgx5, Qwen3.8-27B, 1 node):
fronted boots but fails `engine input registration timed out after 600s`
— the Rust client's fixed ready_timeout is shorter than our GB10 boots
(NAS weights + JIT + cudagraphs = 10–40 min). Verdict (operator): not
pursuing — the fork stays selective about upstream imports and the
frontend is upstream-experimental (not feature-complete; parser parity
for glm45/glm47 tool/reasoning unverified). Revisit only if upstream
matures it (or if per-request CPU overhead ever becomes a bottleneck in
our benches).

## RoCEnante (b12x PR #295) — queued for research (2026-09-07)

`b12x.comm.roce`: one-shot all-reduce + all-gather over RoCE for multi-node
Spark TP (1 GPU/node). Peers stage inputs into pinned host memory and
RDMA-write them directly — no GPUDirect/dmabuf/peermem needed (unavailable
on Spark); a C proxy posts the writes (ibv_reg_mr), a CuTe kernel reduces
in fixed rank order → bit-identical results on all ranks; CUDA-graph
replay safe via a device-resident epoch; fail-stop with self-poisoning
(never a fallback). Measured on our exact production shape: GLM-5.3-Flash
TP4 + MTP5, 4 Sparks. Microbench (bf16, graph replay): 3–3.8× vs NCCL up to
1 MB (48 KB 23.6 vs 54 µs; 1 MB 238.9 vs 899 µs). Serving A/B: decode step
latency lower in 14/15 cells (c=1 62–64 vs 65–66 ms; c=16 194–212 vs
246–259 ms), throughput c=16 +22–32%, c=4 +6–22%, prefill unchanged
(64 MB all-reduces stay on NCCL above the 2 MB cap), coding-peak c=1 −2%
(noise). With both collectives routed, the decode profile shows zero NCCL
kernels — i.e. it attacks exactly our measured decode profile
(bandwidth-bound, scales to ~N=4; prefill untouched).

Integration surface — **verified already in our r27 image** (2026-09-07,
dgx5, `docker run -i` import check, all OK):
- our r27 b12x pin (e8ad299b) is NOT just the original #295 merge — the r27
  line carries it plus follow-ups: two-rail striping (aa90a277 — our
  `rocep1s0f0,roceP2p1s0f0` fabric is exactly what it targets), DSV4
  graph-collective opt (1a7e3ec2), PR #315 gather-geometry (b58f34ea). The
  r27 variant is ahead of the public line, not behind (5 of 8 roce files
  differ from the LIL A/B revision; the r27 ones add per-stripe completion
  flags + power-of-two grid counters);
- the vLLM adapter `b12x_roce_all_reduce.py` is in our vLLM pin (63a82f8d,
  blob a311ae52); `import b12x.comm.roce` + `roce_oneshot` + the adapter all
  import cleanly in the built image;
- b12x env falls back to `NCCL_IB_HCA` / `NCCL_IB_GID_INDEX` — our config
  already pins both CX-7 functions and the per-node GID auto-detect
  propagates automatically;
- enable = `VLLM_ENABLE_ROCE_ALLREDUCE=1` in recipe `[env]` (caps
  `VLLM_ROCE_ALLREDUCE_MAX_SIZE`=2M / `VLLM_ROCE_ALLGATHER_MAX_SIZE`=16M).

Repo note: **voipmonitor/b12x is the operational repo** (holds the r27/r28
qualification lines; LIL is the public community repo where PR #295 merged
and 00280f2 sits on a dangling branch of). The first bump attempt
(e8ad299b → 00280f2) was a wrong move — 00280f2 is a diverged mainline
commit (no r27 GLM line, no `integration/vllm/loader.py`), broke the
b12x-managed build gate, and was reverted; no rebuild was needed.

**A/B DONE (2026-09-07 night, 4nodes-switch, 4x DCP4 shape, arms back to
back, dev46/seed 42):** `glm53-flash-r27-4x-dcp4-roce` (clone +
`VLLM_ENABLE_ROCE_ALLREDUCE=1`, minus `--disable-custom-all-reduce` — the
fork gates the RoCEnante branch on the custom-AR enable flag; inert on
4x1-GPU nodes, so no confounder) vs the NCCL control:
- **decode tg +31…+90% in every cell** (d0 c1 29.9→46.8, d0 c2 46.3→62.4,
  d4096 c1 26.5→50.4); total request time −15…−28%. Bigger than the published
  +22–32% @ c16 — our DCP4 shape gains more at c1/c2 (the DCP ring stays on
  NCCL in both arms; the delta is purely TP collectives on RoCEnante).
- Prefill 5/6 cells within −9…+8% (d0 c1 outlier −32%, 3,319→2,248 — one
  short cell; watch if it repeats).
- Quality **160/176 (91) vs 155 (88)** — +5 from 9 borderline flips (6 up / 3
  down; RoCEnante's float32-accumulate reduction is within 6.8e-3 rel of
  NCCL). TC-51/TC-80 (anchor failures) cleared. Not systematic — net positive.
- Activation verified: tp:0 dispatch `['B12X_ROCENANTE','PYNCCL']`,
  world=4, both rails, 2MB/16MB caps; all-reduce + all-gather "live" logs;
  proxy .so built once, persisted via the b12x cache mount.
- Production 4x aiden stopped for the window (operator OK), restored after.
Reports: `recipes/glm53-flash-r27-4x-dcp4{,-roce}/reports/2026-09-0{7,8}_roce-ab-*`;
detail: `history/night-shifts/2026-09-07.md`.

**2x A/B DONE (2026-09-08, 4nodes-switch, r27-spark 2x, arms back-to-back,
dev46/seed 42):** same toggle, same convention. More modest than 4x (bigger
messages over both rails favor the 4x shape):
- Quality **162/176 (92) vs 158 (90)** — +4: 7 scenarios up (incl. TC-33
  fail→pass), 3 down (TC-40/TC-50 pass→partial, TC-88 pass→fail; TC-88 also
  failed on the aborted first RoCE run — borderline flappy).
- Decode tg: d0 c1 **+17%** (23.4→27.4); the rest within −3.8…+1%
  (d0 c2 −0.6, d4096 c1 −3.6, d4096 c2 −3.8, d8192 c1 −2.0, d8192 c2 +1.0);
  total request time −2.2…+2% (noise) at 2x.
- Verdict: RoCEnante = strict quality win + big decode win at 4x, modest
  quality win at 2x → **default ON for all r27 recipes** (operator call,
  2026-09-08). Both recipes (`glm53-flash-r27-spark`,
  `glm53-flash-r27-4x-dcp4`) now ship `VLLM_ENABLE_ROCE_ALLREDUCE=1` and drop
  `--disable-custom-all-reduce`; the interim `-roce` clone is deleted
  (its reports consolidated into `glm53-flash-r27-4x-dcp4/reports/`). The
  production aiden stack is untouched (its image lacks the r27 b12x/adapter
  surface) — RoCEnante stays an r27-line feature until the aiden image picks
  it up.
- Reports: `recipes/glm53-flash-r27-spark/reports/2026-09-08_roce-ab-2x-arm{A-nccl,B-roce}.{json,log,md}`.

**One-off crash during the 2x A/B (NOT RoCEnante):** the first RoCE-2x run
died mid-bench (after scenario ~21) with a TileLang trace crash in
`vllm/models/glm5next/nvidia/ops/glm_kpool.py` —
`AttributeError: 'int' object has no attribute 'to'` at
`block.to(tl.int64)` in a kpool pool kernel → `EngineDeadError`, clean
shutdown cascade, bench recorded a partial 43%-completion run. The immediate
re-run (same seed/scenarios) completed 100% (162/176) — not reproduced.
Suspect: JIT-trace race or a poisoned kernel-cache entry. If it repeats, the
kpool C4/decode-metadata path under the r27 vLLM pin is the suspect (same
file family as the DFlash2 C4-alignment blocker). The aborted partial report
was discarded; only the clean re-run is recorded.

**Launcher additions the same night (both operator-requested):**
- `stop` now **waits until the containers are actually stopped and removed**
  (per-node 60 s verification poll after `docker stop`; `--no-wait` opts
  out) — an immediate re-serve on the same nodes used to race the dying
  containers' GPU release (two silent worker deaths during the A/B).
- New `sm121 bench --recipe R --cluster C --label L`: runs tool-eval-bench
  (seed 42, hardmode, perf, concurrency 1,2 — the campaign convention)
  against the live head, waits for API readiness, and lands
  `<UTC-date>_<label>.{md,json,log}` in `recipes/<lane>/reports/` (scratch
cwd + absolute `--json-file` handled internally; `-d` prints the command).

Detail: `history/night-shifts/2026-09-08.md`. Sources: local-inference-lab/b12x PR #295
(+ follow-up PR #315, r27-line commits aa90a277/1a7e3ec2), docs/rocenante.md,
receipts docs/evidence/rocenante/.

## r18 16 GiB 4x boot-verify (2026-09-08) — PASSED (capacity-only, no re-bench)

Queue item closed. `glm53-flash-nvfp4-r18-4x-dcp4` (KV pin 16 GiB,
`--kv-cache-memory-bytes 17179869184`) boots clean on 4nodes-switch in
~6 min: 47.48 GiB weights/node, **GPU KV cache 7,738,490 tokens** (≈ the
expected 7.7M @ 16 GiB DCP4 — exact hit), CUDA graphs captured, API live,
completion generating. No OOM, no re-bench (quality 161/176 already
recorded 09-06). Stale "10 GiB" comments in the four r18-4x* recipes
corrected. AGENTS.md gained the "all node operations via sm121 commands"
rule (raw ssh/docker only for one-off diagnostics).

## r28 evaluation started (2026-09-08)

`integration/glm53-r28-cache-fixes-20260906` @ voipmonitor/vllm
`62f6385c` (tree `6361a741`) — 13 commits over the r27 pin, 29 files,
**zero requirements/cmake changes** (GitHub compare API), so the r27 image
tree's dep gates hold. Content: prefill/decode compute-time sharing +
interleaved bounded parallel prefills (the admission-block pain from the
agent-serving findings), chat-template rendering parity (instruction
checkpoints), GLM boundary restore across DCP/DFlash groups, partial
prefix-cache hits. b12x unchanged (e8ad299b, RoCEnante already in).

New recipe `glm53-flash-r28-4x-dcp4`: clone of the r27 4x-dcp4 recipe
(160/176, 46.8/62.4 t/s) with `[build.versions]` overriding
`VLLM_PIN`/`VLLM_TREE`/`VLLM_PRETEND_VERSION` — same build dir
(`glm53-jovian-r27`), distinct image tag `glm53-jovian-r28:local`,
override folds into the build checksum (separate image, no r27 rebuild).
A/B target: r27 4x DCP4 160/176 + 46.8 t/s; 2x (162/176, 27.4 t/s) after
the 4x verdict.

## SparkRing shared image (09-09)

The operator gave access to the public SparkRing stack image:
`ghcr.io/fujitsupolycom/sparkring@sha256:f25cdb6bf7df85754ea5c62445b139dac3143bc6e910a0442d681b6e63c31c4b`
(digest-only, no tags). Inside: their vLLM fork `0.1.dev1+gd377796e8`
(revision 17bd258075f4), b12x 2883a5df, sparkcache d0cf7296, patched NCCL
2.30.7, transport `tp2-rocenante-adaptive` (b12x.comm.roce — the origin of
our RoCEnante). Model `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` @
df116c4f is already in the NFS cache. Profiles: **TP2/DCP1** (RoCEnante
2-path, KV 8.75 GiB, max-len 262K, GMU 0.87, MTP3, block 256) and **TP4
switched** (plain NCCL, RoCEnante OFF, KV 24 GiB, max-len 1M, GMU 0.80,
fastsafetensors).

`docker-images/sparkring-shared/` — a thin `FROM <digest>` wrapper under the
standard launcher build/sync flow (a digest-only raw pull leaves an untagged
image that `docker image prune` reaps; the wrapper tags it). The Dockerfile
also bridges the b12x loader JIT (`-L/opt/cuda-13.3/lib64 -lcudart`, but the
CUDA 13 image ships libcudart only as pip `nvidia/cu13/lib/libcudart.so.13`
and no lib64 dir at all) and pre-builds the loader extension into the image.

Recipes `glm53-flash-sparkring-2x` / `-4x` — the profiles verbatim; `model`
= the standard repo alias (vllm resolves it from the HF cache refs/main =
latest snapshot); the launcher injects `PORT=<recipe.port>` (the
serve-with-warmup wrapper polls 127.0.0.1:$PORT); `SERVED_MODEL_NAME` env
must match `--served-model-name` (the wrapper's warmup model name); cold
first-run JIT needs `DFLASH_WARMUP_TIMEOUT_SECONDS=1800`.

Measured (09-09, hardmode d0, seed 42): 2x 89/100 (server died at TC-57,
739 pp / 31.9 tg / TTFT 3,741 ms), 4x 89/100, 96.6% completion (died at
TC-85/86/88, 1,230 pp / 59.2 tg / TTFT 2,630 ms). Quality on par, decode up
to +26% (4x), prefill/TTFT behind; **reliability is the failure** — both
profiles die under sustained load (their sparkring-memory-guard 4GB floor,
no kernel OOM), while r28 passes 88/88 stably. Details:
`history/night-shifts/2026-09-08.md`.

**Running either profile requires a prod-downtime window**: the physical 2x
pair (dgx1+dgx2) is the same hardware as the production r28 4x (88.x/100.x/
200.x are different fabrics of the same boxes); the sm121-label probe sees
the prod containers → 0 free. `sm121 stop --recipe glm53-flash-r28-4x-dcp4`
first, always.

## R31 campaign + eugr lane (09-10 night, seed 42, hardmode, c1,2)

Update scan (09-09/10) found the R28.1→R29→R30→**R31 (qualified)** release
line in the local-inference-lab source locks (rtx6kpro
`models/glm-5.3-flash/validation/*`). R31 stack: vLLM 55769270 (integration/
jovian-warmup-buffer-reuse-20260909 — carries the GLM Flash MTP projection
loader fix for our upstream eh_proj issue, FlashInfer GDN prefill on SM12x,
scheduler hardening) + b12x 3edbcbce (frozen since 09-08, incl. TP2 graph
peer-push fix) + LMCache f868580a. Clone-at-pin build via the new version
profile `docker-images/glm53-jovian-r27/profiles/glm53-r31.env` (empty
`FLASH_KDA_SRC_DIR` — the r28.1+ tree gates FlashKDA on the packed
checkpoint API and must use its own pinned 3b225bf2 + in-tree patch
fallback; the Dockerfile now takes `FLASH_KDA_SRC_DIR` as an ARG).

Measured (09-10, tool 2.6.1.dev46, 88 scenarios):
- **r31 4x DCP4 (prod lane, GLM-5.3-Flash-NVFP4, 16 GiB KV): 159/176 (90),
  0 errors, all 88 stable** — 1,484 pp / **51.6 tg** (+10% over r28 4x) /
  1,785 ms TTFT @d0 c1. Scenario A/B vs r28 4x: 5 fixed (TC-40/47/49/51/88),
  5 regressed (TC-33 safety, TC-50/68/80/82), 8 common — quality a wash,
  decode up. **Promoted to production** (09:50 UTC, window close).
  `recipes/glm53-flash-r31-4x-dcp4/` (r28 4x fork + `--prefill-compute-share
  0.4` + `NCCL_NCHANNELS=16`).
- **r31 2x DAC (GLM-5.3-Flash-NVFP4-Spark, 10 GiB KV): 154/176 (88)** — 4
  points BELOW the r28 2x baseline (158/176), 1,544 pp / 25.9 tg / 1,289 ms
  @d0 c1. The r31 delta flags (fixed prefill share, 16 channels) do not help
  the 2x lane — **the 2x lane stays on r28** (`glm53-flash-r28-spark`)
  unless the delta is dropped for 2x and re-benched.
- **eugr 2x** — recipe ready (`glm53-flash-eugr-2x`, verbatim mirror of
  eugr/spark-vllm-docker's cluster_only GLM yaml: b12x loader, MTP k=5
  humming, 1M ctx, 8 GiB KV, GMU 0.87), image
  `eugr/spark-vllm-b12x:nightly-20260909` pulled on dgx1+dgx2 — **not
  benched** (window closed by operator); next window on 2nodes-dac.

Rollback: `sm121 stop --recipe glm53-flash-r31-4x-dcp4` + `sm121 serve
--recipe glm53-flash-r28-4x-dcp4` (r28 image still tagged on all nodes).
Full shift log: `history/night-shifts/2026-09-10.md`.

### R35 adoption (2026-09-12, tool 2.6.1.dev65, 4nodes-switch 4x DCP4)

Latest lab rev (qualified 2026-09-11): vLLM `de982a50` / b12x `98086604` /
LMCache `29bc5a2e` (`profiles/glm53-r35.env`, recipe
`glm53-flash-r35-4x-dcp4`). R33's shared-input split MoE prefill arrives with
R35's merged B12X SwiGLU-clamp correction (R33/R34 shipped the split path
without the model's clamp — the runaway-generation defect; we skipped
r33/r34). Build-dep set identical to r31.

Measured (09-12 daytime window, warm JIT after a ~25 min cold inductor
recompile on first boot):
- **r35 4x DCP4: 163/176 (93)** — 1,554 pp / 49.6 tg / 1,544 ms TTFT @d0 c1
  vs r31 4x 159/176 (90) at 1,519/50.0/1,524 (09-10, dev46 — harness-
  confounded; dev65 credit fixes lean up). Scenario delta: r31's TC-33/TC-80
  fails + TC-46/61/67 partials gone (TC-68 partial→pass); new on r35: TC-51
  fail (step-ordering), TC-35/TC-40 partials. Perf flat (their +8.8% prefill
  gain was a 32K-prefill RTX cell — outside our matrix's 8192 max prefix).
  KV 8,060,928 tokens @512K (15.4x concurrency). TC-51 = variance vs
  regression unresolved. **Promoted to production 09-12** (operator: r35
  stays; r31 = rollback lane, image tagged on all nodes). Full log:
  `history/night-shifts/2026-09-12.md` ("Daytime follow-up: R35 adoption
  + A/B").

### DFlash2 A/B on the r35 lane (2026-09-12 night, 4nodes-switch)

The r35 image ships the DFlash2 machinery natively (DFlash2DraftModel,
dedicated `dflash_attn` graph-safe split-KV backend, FlashInferCutlassMxfp8
linear for the fused draft K/V projection) and the 1.2 GB lab MXFP8 draft
was already on the shared NAS — so the A/B was a recipe-only swap (MTP3 →
`method:dflash`, K7, kv auto), no build. The 08-29 2x blocker (page
unification) is gone in the r35 tree: it pads the mamba page size (+0.70%)
to unify draft/target geometry.

4x DCP4 (same dev65 harness/seed as the prod run — clean A/B):

| | r35 MTP3 (prod) | **r35 DFlash2 K7** |
|---|---|---|
| score | 163/176 (93) | **165/176 (94)** |
| pp / tg / TTFT @d0 c1 | 1,554 / 49.6 / 1,544 ms | **1,977 / 47.3 / 1,071 ms** |
| tg @d0 c2 | 53.0 | 55.8 |
| KV @512K | 8.06M tokens (15.4x) | 6.33M tokens (12.1x) |

DFlash2 = prefill +27%, TTFT −31%, decode −4.6% c1 / +5.3% c2, +2 points
(TC-40/50/63 pass; new partial TC-60 + 1). Operator keeps MTP3 as prod of
record for now; **DFlash2 K7 is the 4x candidate** (better latency story,
−22% KV). Context: aiden's 1.1 image (different tree, incoai draft) already
ran DFlash2 on 2x (157/176) — the 2x block below is specific to the r35
tree's C4 page model, not to DFlash2 per se.

**2x Spark is BLOCKED on the r35 tree** (both gates hit, both ranks):
bare TP2 → `GLM MLA parent-page stride must be an exact number of C4
pages`; +DCP2/interleave-2 → `GLM C4 DCP requires
cp_kv_cache_interleave_size to be divisible by 4`. The interleave-multiple-
of-4 rule makes the certified TP4/DCP1 + TP4/DCP4 shapes (4x only) the
working ones — exactly the lab's qualification set. 2x stays MTP3; recipe
`glm53-flash-r35-2x-dflash2` kept as a canary. Ops note: sm121 `docker run`
is `--rm` by default (crashed rank-0 logs vanished); `serve --keep` keeps
containers + logs for post-mortem.

---

## 2026-09-13: FULL GLM-5.3 retargeted to RedHatAI MXFP4 (r35 stack) — old variant deleted

Operator: "download RedHatAI/GLM-5.3-MXFP4, select the base stack to serve it,
update our recipes for this variant, as our variant did not work, no reason to
keep it. Do not start it — night shift."

**Deleted** (all three served the abandoned full-model checkpoint
`local-inference-lab/GLM-5.3-NVFP4-Spark`, 435.6 GB / 406 GiB → 101.5 GiB
weights/node → OOM-proven on 4x 2026-09-08/09: post-load crash + auto-restart
began a SECOND full load on top of the first → 2×101.5 > 121.63 GiB → all four
nodes OOM-thrashed, sshd down on 3 of 4, HA power-cycle. See the 09-08/09-09
history entries):
`recipes/glm53-full-nvfp4-spark-lab`, `recipes/glm53-full-nvfp4-spark-aiden`,
`recipes/glm53-spark-nvfp4-4x`. Kept in git history.

**New**: `recipes/glm53-full-mxfp4-4x` — `RedHatAI/GLM-5.3-MXFP4`
(403.4 GB / 375.6 GiB, 141 shards, compressed-tensors
`mxfp4-pack-quantized`, W4A4 group-32 E8M0; card reports 98.7% gsm8k /
95.4% ifeval recovery vs the FP8 parent).

Geometry (verified from config.json): `GlmMoeDsaForCausalLM` /
`model_type=glm_moe_dsa`, 78 layers, hidden 6144, 256 routed experts + 1
shared, top-8, moe_inter 2048, MLA (kv_lora 512), DSA indexer (index_topk
2048), 1M native ctx, MTP head in-checkpoint (`num_nextn_predict_layers=1`),
no separate drafter. ~740B total / ~40B active — the config geometry and the
file size agree at MXFP4's 4.25 bits/param; note the 09-08 history entry's
"386B" was an estimate that does not match either (a different model class
from the 320B/18B Flash variant we serve in prod: 45 layers, hidden 4096).

**BASE STACK SELECTED: our own jovian r35 build** —
`docker_image_dir = "glm53-jovian-r27"` + `build_profile = "glm53-r35"` →
image `glm53-jovian-r35:local`, the same image as the Flash 4x prod. Evidence
(read-only image inspection, 2026-09-13):
- `registry.py` maps `"GlmMoeDsaForCausalLM" → vllm.models.deepseek_v32`, the
  NEW implementation ("MLA + a lightning indexer selecting top-k for sparse
  MLA") that dispatches to an **NVIDIA-specific path**
  (`deepseek_v32/nvidia/model.py: DeepseekV32ForCausalLM as
  GlmMoeDsaForCausalLM`) — the right target for GB10/sm_121a. The lab fork and
  aiden's `production-1.2` image register the same class from the older
  `deepseek_v2` module (vLLM 0.1.dev20051+g487ecf187) and needed two SM120
  `mla_attention` hotfix mods to boot the full model at all (never re-tested;
  lane parked).
- ships `compressed_tensors_w4a4_mxfp4` (mxfp4-pack-quantized) → the RedHat
  checkpoint loads natively.
- it is our reproducible clone-at-pin build (r35 lock 2026-09-11: vLLM
  de982a50 + b12x 98086604), all gates green, and it carries the GB10-tuned
  b12x backends our GLM prod uses (DCP4, B12X, RoCEnante, MTP3 — all proven
  on the Flash model). One stack to maintain; clean A/B vs the Flash prod;
  JIT caches warm.
Lab fork / aiden image stay as a documented fallback (port the two sed mods
onto `glm53-flash-nvfp4-lab:local` if the deepseek_v32 DSA/indexer path trips).

**Memory plan** (the binding constraint): 121.63 GiB/node, 116–117 GiB idle
free. Weights 93.9 GiB (403.4 GB/4) — 7.6 GiB/node lighter than the failed
attempt, which is what makes 4x plausible. + vLLM/docker ~12 GiB + KV pin
8 GiB + activations/graphs ~4 GiB ≈ 118 GiB. GMU 0.85 is the admission
guardrail only (0.88 = 107.0 GiB fails once the ~12 GiB of starting processes
count — the 2026-09-08 #48140 class); the KV pool is pinned so GMU does not
change capacity.

**Why DCP4**: the full model's KV is NOT the Flash lane's compact MLA cache —
the parked run measured ~54 KB/token (4 GiB ↔ ~79.6K tokens; "524288 would
need 26.34 GiB") vs the Flash's ~2 KB/token at 512K. DCP4 shards KV across
the 4 TP ranks, so the 8 GiB pin buys 32 GiB aggregate → one 524288-token
request needs ~27 GiB aggregate (≈6.8 GiB/rank) and just fits. Without DCP,
512K behind these weights is impossible. CAVEAT: DCP4 is qualified in r35 for
the Flash model; for the full model it is UNVERIFIED — first fallback is to
drop `--decode-context-parallel-size`/`--cp-kv-cache-interleave-size` and
lower `--max-model-len` (~180K at ~10 GiB aggregate KV).

**First-boot plan / hazard**: the 09-08 disaster was a post-load crash +
auto-restart starting a second full load, not the footprint alone. Watch the
first boot; a second "Loading safetensors" pass = kill immediately
(`sm121 stop`), do not wait. Always `--keep` for the first boot (default is
`--rm`, which wipes the crash logs). Ladder: (1) no-DCP + 131072 + KV 4 GiB
(prove stack+checkpoint) → (2) DCP4 + 524288 → (3) add MTP k=3 (k=5 after).
Boot markers to record: "Model loading took X GiB", the KV line, "Application
startup complete". MAX_JOBS=2 is the compile-OOM guard (~3 GiB × 12 nvcc jobs
behind 94 GiB of weights, 2026-09-04 lesson).

No speculative decoding on the first boot (memory + full-model MTP
unproven); `mtp k=3` is documented in the recipe header for step 3.

Prereq: `hf download RedHatAI/GLM-5.3-MXFP4` into the shared NAS HF cache on
all 4 nodes (was not on the NAS as of 2026-09-13; operator downloading).
Note: the r35 image now reads as STALE (the r27 build dir gained the r36
apt-lock/profile edits) — `warn_if_stale` is warn-only and the deployed image
(label 8bb87348) is the proven Flash-prod build, so the night shift can
either serve as-is or rebuild with `sm121 build --recipe
glm53-full-mxfp4-4x --build-node dgx5.lan`.