r12 image: adopt eugr lab build patches (12x patch_vllm_*.py) + pre-staged build sub-deps (SRC_DIR escapes) for flaky node github; ledger: lil render reference + perf push state

.gitignore

@@ -9,3 +9,4 @@ __pycache__/
 .gen/
 publish/
 docker-images/*/jit-cache*
+docker-images/*/deps-src.tar.gz

docker-images/glm53-jovian-r12/Dockerfile

@@ -71,6 +71,18 @@ RUN set -eux; \
     git checkout ${DEEPGEMM_REF}; \
     git submodule update --init --depth 1 -- third-party/cutlass third-party/fmt
 
+FROM deepgemm-builder AS deps-src
+# Pre-fetched vllm build sub-deps (deps-src.tar.gz in the build context, staged
+# on the build host): the node's direct github.com access is flaky (FetchContent
+# git clones get "could not read Username" / hung-up) — these SRC_DIR env vars
+# are each cmake's documented local-source escape hatch.
+COPY deps-src.tar.gz /tmp/deps-src.tar.gz
+RUN set -eux; \
+    mkdir -p /deps2; \
+    tar xzf /tmp/deps-src.tar.gz -C /deps2; \
+    mv /deps2/triton/python/triton_kernels /deps2/triton_kernels; \
+    rm -rf /deps2/triton /tmp/deps-src.tar.gz
+
 FROM deepgemm-builder AS vllm-builder
 ARG VLLM_REPO
 ARG VLLM_BASE_REF
@@ -94,11 +106,50 @@ RUN set -eux; \
     done; \
     git rev-parse HEAD
 WORKDIR /workspace/vllm
+# eugr/spark-vllm-docker build-time patches (main, 2026-09-02): the lab's
+# Spark-tuned vllm fixes. All are source-aware/idempotent (skip when the
+# vulnerable pattern is absent), so running the full set is safe. The two env
+# gates are opt-in and ON for this build (sm_121a + b12x):
+#   - preserve_sm12x_target: keep 12.1a in CUDA_SUPPORTED_ARCHS (their
+#     standard CUDA-13 build collapses 12.1a -> 12.0)
+#   - sm120_cooperative_topk: cooperative_topk fails on SM12.x at launch;
+#     keep SM90-only (Spark fix)
+#   - mrv2_speculator_cudagraph_pool: MTP speculator cudagraph pool fix
+#   - spark_kv_cache_cleanup: DGX Spark UMA allocator cleanup
+#   - b12x_c128a_topk_alignment / flashinfer_b12x_swigluoai: b12x kernel fixes
+#   - routed_experts_weight_shape: modelopt MoE 2-element weight_shape load
+#   - topk_softplus_sqrt_control_flow: dead CUDA kernel fix (upstream bug)
+# Provenance: https://github.com/eugr/spark-vllm-docker (docker/patch_vllm_*.py)
+COPY patches-eugr/patch_vllm_*.py /tmp/eugr-patches/
+RUN VLLM_PRESERVE_SM12X_TARGET=1 \
+    VLLM_PATCH_B12X_C128A_ALIGNMENT=1 \
+    python3 /tmp/eugr-patches/patch_vllm_mrv2_speculator_cudagraph_pool.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_b12x_c128a_topk_alignment.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_flashinfer_b12x_swigluoai.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_topk_softplus_sqrt_control_flow.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_preserve_sm12x_target.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_diffusion_tensor_causal.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_gemma4_mtp_embedding_share.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_sm120_cooperative_topk.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_autogptq_symmetric_moe_qzeros.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_disable_minimax_qk_rmsnorm_ipc.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_routed_experts_weight_shape.py . \
+    && python3 /tmp/eugr-patches/patch_vllm_spark_kv_cache_cleanup.py .
+COPY --from=deps-src /deps2 /deps2
 RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv \
     --mount=type=cache,id=ccache,target=/root/.ccache \
-    uv pip install setuptools-rust setuptools-scm \
+    git config --global protocol.version 0 \
+    && uv pip install setuptools-rust setuptools-scm \
     && _j="${BUILD_JOBS:-$(nproc)}" \
-    && export MAX_JOBS="${_j}" CMAKE_BUILD_PARALLEL_LEVEL="${_j}" \
+    && export GIT_PROTOCOL=0 MAX_JOBS="${_j}" CMAKE_BUILD_PARALLEL_LEVEL="${_j}" \
+        TRITON_KERNELS_SRC_DIR=/deps2/triton_kernels \
+        FMHA_SM100_SRC_DIR=/deps2/MSA \
+        FLASH_MLA_SRC_DIR=/deps2/FlashMLA \
+        FLASH_KDA_SRC_DIR=/deps2/FlashKDA \
+        VLLM_FLASH_ATTN_SRC_DIR=/deps2/flash-attention \
+        QUTLASS_SRC_DIR=/deps2/qutlass \
+        TML_FA4_SRC_DIR=/deps2/tml-fa4 \
+        VLLM_CUTLASS_SRC_DIR=/deps2/cutlass \
     && python3 use_existing_torch.py \
     && uv build --no-build-isolation --wheel . --out-dir=/wheels -v
 

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_autogptq_symmetric_moe_qzeros.py

@@ -0,0 +1,29 @@
+#!/usr/bin/env python3
+"""Do not pass AutoGPTQ MoE zero-points for symmetric quantization."""
+
+import sys
+from pathlib import Path
+
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "vllm/model_executor/layers/quantization/auto_gptq.py"
+bad = '''            w1_zp=getattr(layer, "w13_qzeros", None),
+            w2_zp=getattr(layer, "w2_qzeros", None),'''
+fixed = '''            w1_zp=getattr(layer, "w13_qzeros", None)
+            if not self.quant_config.is_sym
+            else None,
+            w2_zp=getattr(layer, "w2_qzeros", None)
+            if not self.quant_config.is_sym
+            else None,'''
+
+if not target.exists():
+    print(f"{target} not found; skipping AutoGPTQ MoE qzeros workaround")
+else:
+    text = target.read_text()
+    if fixed in text:
+        print("AutoGPTQ MoE qzeros workaround already present; skipping")
+    elif bad in text:
+        target.write_text(text.replace(bad, fixed, 1))
+        print("Applied AutoGPTQ symmetric MoE qzeros workaround")
+    else:
+        print("Known vulnerable AutoGPTQ MoE qzeros pattern not found; skipping")

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_b12x_c128a_topk_alignment.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""Import the C128A top-k alignment used by the B12X DeepSeek V4 path.
+
+The local-inference-lab B12X branch added a dynamic C128A top-k width that
+references ``_C128A_TOPK_ALIGNMENT`` without importing it from
+``compressor_utils``.  Keep this workaround opt-in, idempotent, and strict
+about the vulnerable source shape so regular vLLM builds remain untouched.
+"""
+
+from __future__ import annotations
+
+import ast
+import os
+import sys
+from pathlib import Path
+
+
+FLAG_NAME = "VLLM_PATCH_B12X_C128A_ALIGNMENT"
+ALIGNMENT_NAME = "_C128A_TOPK_ALIGNMENT"
+TARGET_REL = Path("vllm/models/deepseek_v4/sparse_mla.py")
+HELPER_REL = Path("vllm/v1/attention/backends/mla/compressor_utils.py")
+IMPORT_BLOCK = (
+    "from vllm.v1.attention.backends.mla.compressor_utils import (\n"
+    "    get_c128a_topk_width,\n"
+    "    get_compressed_slot_mapping,\n"
+    ")"
+)
+PATCHED_IMPORT_BLOCK = (
+    "from vllm.v1.attention.backends.mla.compressor_utils import (\n"
+    f"    {ALIGNMENT_NAME},\n"
+    "    get_c128a_topk_width,\n"
+    "    get_compressed_slot_mapping,\n"
+    ")"
+)
+TRUE_VALUES = {"1", "true", "TRUE", "yes", "YES"}
+FALSE_VALUES = {"", "0", "false", "FALSE", "no", "NO"}
+
+
+def module_binds_name(tree: ast.Module, name: str) -> bool:
+    """Return whether the module establishes ``name`` before runtime use."""
+    for node in tree.body:
+        if isinstance(node, ast.ImportFrom):
+            if any((alias.asname or alias.name) == name for alias in node.names):
+                return True
+        elif isinstance(node, ast.Assign):
+            if any(
+                isinstance(target, ast.Name) and target.id == name
+                for target in node.targets
+            ):
+                return True
+        elif isinstance(node, ast.AnnAssign):
+            if isinstance(node.target, ast.Name) and node.target.id == name:
+                return True
+    return False
+
+
+def module_constant_is_128(tree: ast.Module, name: str) -> bool:
+    """Return whether ``name`` is assigned the expected value at module scope."""
+    for node in tree.body:
+        value: ast.expr | None = None
+        if isinstance(node, ast.Assign) and any(
+            isinstance(target, ast.Name) and target.id == name
+            for target in node.targets
+        ):
+            value = node.value
+        elif (
+            isinstance(node, ast.AnnAssign)
+            and isinstance(node.target, ast.Name)
+            and node.target.id == name
+        ):
+            value = node.value
+        if isinstance(value, ast.Constant) and value.value == 128:
+            return True
+    return False
+
+
+raw_flag = os.environ.get(FLAG_NAME, "0")
+if raw_flag not in TRUE_VALUES | FALSE_VALUES:
+    raise SystemExit(f"Invalid {FLAG_NAME} value: {raw_flag!r}")
+if raw_flag not in TRUE_VALUES:
+    print("B12X C128A alignment workaround not requested; skipping")
+    raise SystemExit(0)
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / TARGET_REL
+helper = source_root / HELPER_REL
+
+if not target.exists():
+    print(f"{TARGET_REL} is absent; B12X C128A workaround is not applicable")
+    raise SystemExit(0)
+
+text = target.read_text()
+try:
+    tree = ast.parse(text, filename=str(target))
+except SyntaxError as exc:
+    raise SystemExit(f"Cannot inspect invalid Python source {target}: {exc}") from exc
+
+uses_alignment = any(
+    isinstance(node, ast.Name)
+    and node.id == ALIGNMENT_NAME
+    and isinstance(node.ctx, ast.Load)
+    for node in ast.walk(tree)
+)
+if not uses_alignment:
+    print("DeepSeek V4 source does not use the missing C128A alignment; skipping")
+    raise SystemExit(0)
+if module_binds_name(tree, ALIGNMENT_NAME):
+    print("DeepSeek V4 C128A alignment is already defined or imported; skipping")
+    raise SystemExit(0)
+
+if not helper.exists():
+    raise SystemExit(
+        f"{target} uses {ALIGNMENT_NAME}, but {helper} is missing; refusing to guess"
+    )
+helper_text = helper.read_text()
+try:
+    helper_tree = ast.parse(helper_text, filename=str(helper))
+except SyntaxError as exc:
+    raise SystemExit(f"Cannot inspect invalid Python source {helper}: {exc}") from exc
+if not module_constant_is_128(helper_tree, ALIGNMENT_NAME):
+    raise SystemExit(
+        f"{target} uses {ALIGNMENT_NAME}, but {helper} does not define it as 128; "
+        "refusing to patch an unknown source shape"
+    )
+
+anchor_count = text.count(IMPORT_BLOCK)
+if anchor_count != 1:
+    raise SystemExit(
+        f"Expected one DeepSeek V4 compressor_utils import block, found "
+        f"{anchor_count}; refusing to patch an unknown source shape"
+    )
+
+updated = text.replace(IMPORT_BLOCK, PATCHED_IMPORT_BLOCK, 1)
+compile(updated, str(target), "exec")
+updated_tree = ast.parse(updated, filename=str(target))
+if not module_binds_name(updated_tree, ALIGNMENT_NAME):
+    raise SystemExit("B12X C128A alignment import was not established after patching")
+
+target.write_text(updated)
+print("Applied B12X DeepSeek V4 C128A top-k alignment import workaround")

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_diffusion_tensor_causal.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+"""Allow DiffusionGemma Tensor causal masks after vLLM PR #47914."""
+
+import sys
+from pathlib import Path
+
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "vllm/v1/worker/gpu/attn_utils.py"
+bad_signature = "causal: bool | Mapping[int, bool] = True,"
+fixed_signature = "causal: bool | Mapping[int, bool] | torch.Tensor = True,"
+bad_group_causal = (
+    "        group_causal = causal if isinstance(causal, bool) else "
+    "causal.get(i, True)"
+)
+fixed_group_causal = """        if isinstance(causal, (bool, torch.Tensor)):
+            group_causal = causal
+        else:
+            group_causal = causal.get(i, True)"""
+
+if not target.exists():
+    raise SystemExit(f"{target} not found; cannot apply DiffusionGemma causal patch")
+
+text = target.read_text()
+if fixed_signature in text and fixed_group_causal in text:
+    print("DiffusionGemma Tensor causal workaround already present; skipping")
+elif bad_signature in text and bad_group_causal in text:
+    text = text.replace(bad_signature, fixed_signature, 1)
+    text = text.replace(bad_group_causal, fixed_group_causal, 1)
+    target.write_text(text)
+    print("Applied DiffusionGemma Tensor causal workaround for vLLM PR #47914")
+else:
+    print("Known vLLM PR #47914 causal regression pattern not found; skipping")

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_disable_minimax_qk_rmsnorm_ipc.py

@@ -0,0 +1,28 @@
+#!/usr/bin/env python3
+"""Disable the MiniMax QK RMSNorm CUDA IPC fusion on multi-node DGX Spark."""
+
+import sys
+from pathlib import Path
+
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "vllm/model_executor/layers/minimax_rms_norm/rms_norm_tp.py"
+marker = (
+    '_MINIMAX_FUSED_AR_RMS_QK = getattr(torch.ops._C, '
+    '"minimax_allreduce_rms_qk", None)'
+)
+replacement = (
+    "_MINIMAX_FUSED_AR_RMS_QK = None  "
+    "# Disabled for DGX Spark multi-node TP"
+)
+
+if target.exists() and marker in (text := target.read_text()):
+    print("MiniMax QK norm fusion found; disabling CUDA IPC fused path")
+    target.write_text(text.replace(marker, replacement))
+elif target.exists() and replacement in target.read_text():
+    print("MiniMax QK norm fusion already disabled")
+else:
+    print("MiniMax QK norm fusion marker not present; skipping patch")
+
+if target.exists() and marker in target.read_text():
+    raise SystemExit("ERROR: MiniMax QK norm fusion marker is still present")

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_flashinfer_b12x_swigluoai.py

@@ -0,0 +1,349 @@
+#!/usr/bin/env python3
+"""Apply the runtime subset of vLLM PR #47392 needed by FlashInfer B12x.
+
+The upstream PR also changes tests and documentation.  This downstream patch
+only teaches the production backend to pass SwiGLU-OAI parameters through to
+FlashInfer and lets the NVFP4 oracle select it when that FlashInfer API exists.
+
+The patch is idempotent.  It skips refs that predate FlashInfer B12x and fails
+on unknown source shapes instead of making a best-effort rewrite.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+
+EXPERT_REL = Path(
+    "vllm/model_executor/layers/fused_moe/experts/flashinfer_b12x_moe.py"
+)
+ORACLE_REL = Path("vllm/model_executor/layers/fused_moe/oracle/nvfp4.py")
+FLASHINFER_UTIL_REL = Path("vllm/utils/flashinfer.py")
+
+
+class PatchError(RuntimeError):
+    pass
+
+
+def replace_once(text: str, old: str, new: str, description: str) -> str:
+    count = text.count(old)
+    if count != 1:
+        raise PatchError(
+            f"expected one {description} source anchor, found {count}; "
+            "the vLLM source shape has changed"
+        )
+    return text.replace(old, new, 1)
+
+
+def patch_flashinfer_util(text: str) -> str:
+    helper_name = "has_flashinfer_b12x_moe_activation"
+    if f"def {helper_name}()" not in text:
+        if "\nimport inspect\n" not in text:
+            text = replace_once(
+                text,
+                "import importlib.util\n",
+                "import importlib.util\nimport inspect\n",
+                "FlashInfer import",
+            )
+
+        anchor = (
+            "    return True\n\n\n"
+            "@functools.cache\n"
+            "def has_nvidia_artifactory() -> bool:"
+        )
+        helper = (
+            "    return True\n\n\n"
+            "@functools.cache\n"
+            f"def {helper_name}() -> bool:\n"
+            '    """Return whether B12xMoEWrapper accepts SwiGLU parameters."""\n'
+            "    if not has_flashinfer_b12x_moe():\n"
+            "        return False\n"
+            '    mod = _get_submodule("flashinfer.fused_moe")\n'
+            '    if mod is None or not hasattr(mod, "B12xMoEWrapper"):\n'
+            "        return False\n"
+            "    return (\n"
+            '        "swiglu_limit" '
+            "in inspect.signature(mod.B12xMoEWrapper).parameters\n"
+            "    )\n\n\n"
+            "@functools.cache\n"
+            "def has_nvidia_artifactory() -> bool:"
+        )
+        text = replace_once(text, anchor, helper, "B12x capability helper")
+
+    export = f'    "{helper_name}",\n'
+    if export not in text:
+        text = replace_once(
+            text,
+            '    "has_flashinfer_b12x_moe",\n',
+            '    "has_flashinfer_b12x_moe",\n' + export,
+            "FlashInfer __all__ entry",
+        )
+
+    if f"def {helper_name}()" not in text or export not in text:
+        raise PatchError("FlashInfer B12x activation capability helper is incomplete")
+    return text
+
+
+def patch_expert(text: str) -> str:
+    helper_name = "has_flashinfer_b12x_moe_activation"
+    helper_import = f"    {helper_name},\n"
+    if helper_import not in text:
+        text = replace_once(
+            text,
+            "    has_flashinfer_b12x_moe,\n",
+            "    has_flashinfer_b12x_moe,\n" + helper_import,
+            "FlashInfer B12x helper import",
+        )
+
+    activation_entry = (
+        '        MoEActivation.SWIGLUOAI_UNINTERLEAVE: '
+        '"swigluoai_uninterleave",\n'
+    )
+    if activation_entry not in text:
+        text = replace_once(
+            text,
+            '        MoEActivation.RELU2_NO_MUL: "relu2",\n',
+            '        MoEActivation.RELU2_NO_MUL: "relu2",\n' + activation_entry,
+            "B12x activation map entry",
+        )
+
+    if "        self.swiglu_alpha = _resolve(" not in text:
+        config_block = """
+        # ModelOpt NVFP4 checkpoints carry SwiGLU parameters on moe_config,
+        # while other quantization paths may put them on quant_config.
+        def _resolve(
+            quant_val: float | None, moe_val: float | None
+        ) -> float | None:
+            return quant_val if quant_val is not None else moe_val
+
+        self.swiglu_alpha = _resolve(
+            quant_config.gemm1_alpha, moe_config.swiglu_alpha
+        )
+        self.swiglu_beta = _resolve(
+            quant_config.gemm1_beta, moe_config.swiglu_beta
+        )
+        self.swiglu_limit = _resolve(
+            quant_config.gemm1_clamp_limit, moe_config.swiglu_limit
+        )
+        if activation == MoEActivation.SWIGLUOAI_UNINTERLEAVE:
+            if self.swiglu_limit is None:
+                raise ValueError(
+                    "swigluoai_uninterleave requires swiglu_limit "
+                    "(gemm1_clamp_limit), but none was provided."
+                )
+        elif self.swiglu_limit is not None:
+            raise ValueError(
+                "FlashInferB12xExperts only applies swiglu_limit with the "
+                f"swigluoai_uninterleave activation, got {activation}."
+            )
+"""
+        text = replace_once(
+            text,
+            "        self._activation_str = self._ACTIVATION_MAP[activation]\n",
+            "        self._activation_str = self._ACTIVATION_MAP[activation]\n"
+            + config_block,
+            "B12x activation configuration",
+        )
+
+    support_check = "            and has_flashinfer_b12x_moe_activation()\n"
+    if support_check not in text:
+        legacy_support = (
+            "    def _supports_activation(activation: MoEActivation) -> bool:\n"
+            "        return activation in ("
+            "MoEActivation.SILU, MoEActivation.RELU2_NO_MUL)\n"
+        )
+        patched_legacy_support = (
+            "    def _supports_activation(activation: MoEActivation) -> bool:\n"
+            "        if activation in ("
+            "MoEActivation.SILU, MoEActivation.RELU2_NO_MUL):\n"
+            "            return True\n"
+            "        return (\n"
+            "            activation == MoEActivation.SWIGLUOAI_UNINTERLEAVE\n"
+            + support_check
+            + "        )\n"
+        )
+        gelu_support = (
+            "    def _supports_activation(activation: MoEActivation) -> bool:\n"
+            "        return activation in (\n"
+            "            MoEActivation.SILU,\n"
+            "            MoEActivation.GELU_TANH,\n"
+            "            MoEActivation.RELU2_NO_MUL,\n"
+            "        )\n"
+        )
+        patched_gelu_support = (
+            "    def _supports_activation(activation: MoEActivation) -> bool:\n"
+            "        if activation in (\n"
+            "            MoEActivation.SILU,\n"
+            "            MoEActivation.GELU_TANH,\n"
+            "            MoEActivation.RELU2_NO_MUL,\n"
+            "        ):\n"
+            "            return True\n"
+            "        return (\n"
+            "            activation == MoEActivation.SWIGLUOAI_UNINTERLEAVE\n"
+            + support_check
+            + "        )\n"
+        )
+        known_support_shapes = (
+            (legacy_support, patched_legacy_support),
+            (gelu_support, patched_gelu_support),
+        )
+        matches = [(old, new) for old, new in known_support_shapes if old in text]
+        match_count = sum(text.count(old) for old, _ in known_support_shapes)
+        if match_count != 1:
+            raise PatchError(
+                "expected one B12x activation support predicate source anchor, "
+                f"found {match_count}; the vLLM source shape has changed"
+            )
+        old_support, new_support = matches[0]
+        text = text.replace(old_support, new_support, 1)
+
+    if "        swiglu_kwargs: dict[str, float] = {}\n" not in text:
+        wrapper_kwargs = """        swiglu_kwargs: dict[str, float] = {}
+        if self.swiglu_limit is not None:
+            # vLLM defaults unset alpha/beta to 1.0/0.0, while FlashInfer's
+            # wrapper defaults are 1.702/1.0, so pass the resolved values.
+            swiglu_kwargs = {
+                "swiglu_limit": self.swiglu_limit,
+                "swiglu_alpha": (
+                    1.0 if self.swiglu_alpha is None else self.swiglu_alpha
+                ),
+                "swiglu_beta": (
+                    0.0 if self.swiglu_beta is None else self.swiglu_beta
+                ),
+            }
+
+"""
+        text = replace_once(
+            text,
+            "        self._wrapper = B12xMoEWrapper(\n",
+            wrapper_kwargs + "        self._wrapper = B12xMoEWrapper(\n",
+            "B12x wrapper construction",
+        )
+
+    if "            **swiglu_kwargs,\n" not in text:
+        text = replace_once(
+            text,
+            "            activation=self._activation_str,\n",
+            "            activation=self._activation_str,\n"
+            "            **swiglu_kwargs,\n",
+            "B12x wrapper SwiGLU keyword forwarding",
+        )
+
+    required = (
+        helper_import,
+        activation_entry,
+        "        self.swiglu_alpha = _resolve(",
+        support_check,
+        "            **swiglu_kwargs,\n",
+    )
+    missing = [marker.strip() for marker in required if marker not in text]
+    if missing:
+        raise PatchError(f"FlashInferB12xExperts patch is incomplete: {missing}")
+    return text
+
+
+def patch_oracle(text: str) -> str:
+    activation_import = (
+        "from vllm.model_executor.layers.fused_moe.activation import "
+        "MoEActivation\n"
+    )
+    if activation_import not in text:
+        text = replace_once(
+            text,
+            "from vllm.logger import init_logger\n",
+            "from vllm.logger import init_logger\n" + activation_import,
+            "MoEActivation import",
+        )
+
+    helper_import = (
+        "from vllm.utils.flashinfer import "
+        "has_flashinfer_b12x_moe_activation\n"
+    )
+    if helper_import not in text:
+        text = replace_once(
+            text,
+            "\nlogger = init_logger(__name__)\n",
+            helper_import + "\nlogger = init_logger(__name__)\n",
+            "B12x activation helper import",
+        )
+
+    set_marker = "    NVFP4_BACKENDS_WITH_CLAMP = {\n"
+    set_start = text.find(set_marker)
+    if set_start == -1:
+        raise PatchError("NVFP4_BACKENDS_WITH_CLAMP is absent from the oracle")
+    set_end = text.find("\n    }", set_start + len(set_marker))
+    if set_end == -1:
+        raise PatchError("NVFP4_BACKENDS_WITH_CLAMP has an unknown source shape")
+    set_end += len("\n    }")
+    clamp_set = text[set_start:set_end]
+
+    add_marker = (
+        "NVFP4_BACKENDS_WITH_CLAMP.add("
+        "NvFp4MoeBackend.FLASHINFER_B12X)"
+    )
+    if (
+        "NvFp4MoeBackend.FLASHINFER_B12X" not in clamp_set
+        and add_marker not in text
+    ):
+        condition = """
+
+    # B12x applies the clamp only for SwiGLU-OAI and only when the installed
+    # FlashInfer wrapper exposes the corresponding activation parameters.
+    if (
+        config.activation == MoEActivation.SWIGLUOAI_UNINTERLEAVE
+        and has_flashinfer_b12x_moe_activation()
+    ):
+        NVFP4_BACKENDS_WITH_CLAMP.add(NvFp4MoeBackend.FLASHINFER_B12X)"""
+        text = text[:set_end] + condition + text[set_end:]
+
+    set_start = text.find(set_marker)
+    set_end = text.find("\n    }", set_start + len(set_marker)) + len("\n    }")
+    clamp_set = text[set_start:set_end]
+    if (
+        "NvFp4MoeBackend.FLASHINFER_B12X" not in clamp_set
+        and add_marker not in text
+    ):
+        raise PatchError("NVFP4 oracle patch is incomplete")
+    return text
+
+
+def main() -> int:
+    source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+    expert_path = source_root / EXPERT_REL
+    if not expert_path.exists():
+        print(f"{EXPERT_REL} is absent; targeted PR #47392 patch is not applicable")
+        return 0
+
+    paths_and_patchers = (
+        (source_root / FLASHINFER_UTIL_REL, patch_flashinfer_util),
+        (expert_path, patch_expert),
+        (source_root / ORACLE_REL, patch_oracle),
+    )
+    updates: list[tuple[Path, str, str]] = []
+    for path, patcher in paths_and_patchers:
+        if not path.exists():
+            raise PatchError(f"{path} is missing while {EXPERT_REL} exists")
+        original = path.read_text()
+        updated = patcher(original)
+        compile(updated, str(path), "exec")
+        updates.append((path, original, updated))
+
+    changed = []
+    for path, original, updated in updates:
+        if updated != original:
+            path.write_text(updated)
+            changed.append(str(path.relative_to(source_root)))
+
+    if changed:
+        print("Applied targeted vLLM PR #47392 runtime patch: " + ", ".join(changed))
+    else:
+        print("Targeted vLLM PR #47392 runtime behavior is already present")
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        raise SystemExit(main())
+    except PatchError as exc:
+        raise SystemExit(f"targeted vLLM PR #47392 patch failed: {exc}") from exc

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_gemma4_mtp_embedding_share.py

@@ -0,0 +1,32 @@
+#!/usr/bin/env python3
+"""Keep the EAGLE embedding-width guard from breaking Gemma4 MTP."""
+
+import sys
+from pathlib import Path
+
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "vllm/v1/spec_decode/llm_base_proposer.py"
+old = """            if share_embeddings:
+                draft_embed = self.model.model.embed_tokens
+                # Only share when both models use the same embedding width.
+                # Guard with isinstance so non-Tensor weights (e.g. in tests)
+"""
+new = (
+    '            if share_embeddings and hasattr(self.model, "has_own_embed_tokens"):\n'
+    "                draft_embed = self.model.model.embed_tokens\n"
+    "                # Only share when both models use the same embedding width.\n"
+    "                # Guard with isinstance so non-Tensor weights (e.g. in tests)\n"
+)
+
+if not target.exists():
+    print(f"{target} not found; skipping Gemma4 MTP embedding-share workaround")
+else:
+    text = target.read_text()
+    if 'if share_embeddings and hasattr(self.model, "has_own_embed_tokens"):' in text:
+        print("Gemma4 MTP embedding-share workaround already present; skipping")
+    elif old in text:
+        target.write_text(text.replace(old, new, 1))
+        print("Applied Gemma4 MTP embedding-share workaround")
+    else:
+        print("Known Gemma4 MTP embedding-share pattern not found; skipping")

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_mrv2_speculator_cudagraph_pool.py

@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+"""Isolate MRV2 speculator graphs during CUDA-graph memory profiling."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+
+TARGET_REL = Path("vllm/v1/worker/gpu/cudagraph_utils.py")
+MARKER = "spark-vllm-docker: isolate speculator profiling graphs"
+
+DECLARATION_ANCHOR = """    all_wrappers: list[Any] = []
+    original_pools: dict[int, Any] = {}
+    try:
+"""
+
+DECLARATION_REPLACEMENT = """    all_wrappers: list[Any] = []
+    original_pools: dict[int, Any] = {}
+    speculator_managers: list[tuple[str, CudaGraphManager]] = []
+    try:
+"""
+
+POOL_ANCHOR = """        manager.pool = current_platform.graph_pool_handle()
+        if manager.use_breakable_cg:
+"""
+
+POOL_REPLACEMENT = f"""        manager.pool = current_platform.graph_pool_handle()
+        # {MARKER}. Speculator prefill and decode use separate
+        # CudaGraphManager instances, so redirect them before their lazy capture
+        # runners are created. Otherwise their discarded profiling graphs use
+        # the persistent global pool and leave its allocator handle stale.
+        if runner.speculator is not None:
+            speculator_managers = [
+                (name, value)
+                for name, value in vars(runner.speculator).items()
+                if isinstance(value, CudaGraphManager)
+            ]
+            for _, speculator_manager in speculator_managers:
+                speculator_manager.pool = manager.pool
+        if manager.use_breakable_cg:
+"""
+
+CLEANUP_ANCHOR = """        CUDAGraphWrapper.clear_all_graphs()
+        BreakableCUDAGraphWrapper.clear_all_graphs()
+        for wrapper in all_wrappers:
+"""
+
+CLEANUP_REPLACEMENT = """        CUDAGraphWrapper.clear_all_graphs()
+        BreakableCUDAGraphWrapper.clear_all_graphs()
+        # Drop speculator FULL graphs and their references to the throwaway
+        # pool before the profiling state is garbage-collected. The real KV
+        # initialization constructs fresh managers immediately afterwards.
+        for name, speculator_manager in speculator_managers:
+            speculator_manager.graphs.clear()
+            speculator_manager.pool = None
+            if (
+                runner.speculator is not None
+                and getattr(runner.speculator, name, None) is speculator_manager
+            ):
+                setattr(runner.speculator, name, None)
+        for wrapper in all_wrappers:
+"""
+
+
+class PatchError(RuntimeError):
+    """The vulnerable profiler is present in an unsupported source layout."""
+
+
+def is_fixed(source: str) -> bool:
+    local_fix = (
+        MARKER in source
+        and "speculator_manager.pool = manager.pool" in source
+        and "speculator_manager.graphs.clear()" in source
+        and "setattr(runner.speculator, name, None)" in source
+    )
+    manager_collection_fix = all(
+        anchor in source
+        for anchor in (
+            "def _profiling_cudagraph_managers(",
+            "speculator = runner.speculator",
+            "candidate = getattr(speculator, name, None)",
+            "graph_managers = _profiling_cudagraph_managers(runner)",
+            "graph_manager.pool = manager.pool",
+            "graph_manager.graphs.clear()",
+            "graph_manager.pool = original_manager_pools[id(graph_manager)]",
+        )
+    )
+    return local_fix or manager_collection_fix
+
+
+def is_affected_profiler(source: str) -> bool:
+    return (
+        "def profile_cudagraph_memory" in source
+        and "manager.pool = current_platform.graph_pool_handle()" in source
+        and "PIECEWISE, encoder and speculator graphs are measured in full" in source
+    )
+
+
+def replace_once(source: str, anchor: str, replacement: str, label: str) -> str:
+    count = source.count(anchor)
+    if count != 1:
+        raise PatchError(
+            f"expected exactly one {label} anchor, found {count}; "
+            "refusing to patch an unknown MRV2 profiler layout"
+        )
+    return source.replace(anchor, replacement, 1)
+
+
+def patch_source(source: str) -> tuple[str, str]:
+    if is_fixed(source):
+        return source, "Equivalent MRV2 speculator CUDA-graph pool fix is present; skipping"
+    if not is_affected_profiler(source):
+        return source, "Affected MRV2 CUDA-graph memory profiler is absent; skipping"
+
+    patched = replace_once(
+        source,
+        DECLARATION_ANCHOR,
+        DECLARATION_REPLACEMENT,
+        "profiling-state declaration",
+    )
+    patched = replace_once(
+        patched,
+        POOL_ANCHOR,
+        POOL_REPLACEMENT,
+        "throwaway-pool assignment",
+    )
+    patched = replace_once(
+        patched,
+        CLEANUP_ANCHOR,
+        CLEANUP_REPLACEMENT,
+        "profiling cleanup",
+    )
+    if not is_fixed(patched):
+        raise PatchError("MRV2 speculator CUDA-graph pool patch postcondition failed")
+    return patched, "Applied MRV2 speculator CUDA-graph profiling-pool fix"
+
+
+def main() -> None:
+    source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+    target = source_root / TARGET_REL
+    if not target.exists():
+        print(f"{TARGET_REL} is absent; targeted MRV2 profiling patch is not applicable")
+        return
+
+    source = target.read_text()
+    try:
+        patched, message = patch_source(source)
+    except PatchError as exc:
+        raise SystemExit(f"MRV2 speculator CUDA-graph pool patch failed: {exc}") from exc
+
+    if patched != source:
+        target.write_text(patched)
+    print(message)
+
+
+if __name__ == "__main__":
+    main()

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_preserve_sm12x_target.py

@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+"""Preserve explicitly selected CUDA 13 Blackwell subarchitecture targets."""
+
+import os
+import sys
+from pathlib import Path
+
+
+if os.environ.get("VLLM_PRESERVE_SM12X_TARGET", "") not in {
+    "1",
+    "true",
+    "TRUE",
+    "yes",
+    "YES",
+}:
+    print("Blackwell target preservation not requested; skipping")
+    raise SystemExit(0)
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "CMakeLists.txt"
+cuda13_default = (
+    'set(CUDA_SUPPORTED_ARCHS '
+    '"7.5;8.0;8.6;8.7;8.9;9.0;10.0;11.0;12.0")'
+)
+cuda13_sm121 = (
+    'set(CUDA_SUPPORTED_ARCHS '
+    '"7.5;8.0;8.6;8.7;8.9;9.0;10.0;11.0;12.0;12.1")'
+)
+cuda13_subarchitectures = (
+    'set(CUDA_SUPPORTED_ARCHS '
+    '"7.5;8.0;8.6;8.7;8.9;9.0;10.0;10.3;11.0;12.0;12.1")'
+)
+
+text = target.read_text()
+if cuda13_subarchitectures in text:
+    print("CUDA 13 Blackwell subarchitecture allow-list already present; skipping")
+elif text.count(cuda13_sm121) == 1:
+    target.write_text(text.replace(cuda13_sm121, cuda13_subarchitectures, 1))
+    print("Enabled selected SM103 and SM12x targets for CUDA 13 vLLM build")
+elif text.count(cuda13_default) == 1:
+    target.write_text(text.replace(cuda13_default, cuda13_subarchitectures, 1))
+    print("Enabled selected SM103 and SM12x targets for CUDA 13 vLLM build")
+else:
+    raise SystemExit(
+        "Unable to preserve the selected Blackwell target: expected CUDA 13 "
+        "CUDA_SUPPORTED_ARCHS entry was not found exactly once"
+    )

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_routed_experts_weight_shape.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+"""Preserve vector weight_shape metadata in RoutedExperts loading."""
+
+import sys
+from pathlib import Path
+
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "vllm/model_executor/layers/fused_moe/routed_experts.py"
+old = '''    def _load_single_value(
+        self, param: torch.nn.Parameter, loaded_weight: torch.Tensor, expert_id: int
+    ):
+        param_data = param.data
+
+        # Input scales can be loaded directly and should be equal.
+        param_data[expert_id] = self._to_scalar(loaded_weight)
+'''
+new = '''    def _load_single_value(
+        self, param: torch.nn.Parameter, loaded_weight: torch.Tensor, expert_id: int
+    ):
+        param_data = param.data
+        target = param_data[expert_id]
+
+        if target.ndim > 0 and target.numel() == loaded_weight.numel():
+            target.copy_(loaded_weight.reshape_as(target).to(
+                device=target.device, dtype=target.dtype))
+            return
+
+        # Scalar input scales can be loaded directly and should be equal.
+        param_data[expert_id] = self._to_scalar(loaded_weight)
+'''
+
+if not target.exists():
+    print(f"{target} not found; skipping RoutedExperts weight_shape workaround")
+else:
+    text = target.read_text()
+    if "target = param_data[expert_id]" in text:
+        print("RoutedExperts weight_shape workaround already present; skipping")
+    elif old in text:
+        target.write_text(text.replace(old, new, 1))
+        print("Applied RoutedExperts weight_shape workaround")
+    else:
+        print(
+            "Known vulnerable RoutedExperts _load_single_value pattern "
+            "not found; skipping"
+        )

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_sm120_cooperative_topk.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+"""Restrict vLLM's cooperative sparse-attention top-k path to SM90."""
+
+import sys
+from pathlib import Path
+
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "vllm/model_executor/layers/sparse_attn_indexer.py"
+old = '''        use_cooperative_topk = (
+            current_platform.is_cuda()
+            and topk_tokens in (512, 1024, 2048)
+            and num_rows <= 32
+            and logits.stride(0) % 4 == 0  # TMA 16-byte alignment
+            and current_platform.has_device_capability(90)
+        )'''
+new = '''        device_capability = current_platform.get_device_capability()
+        use_cooperative_topk = (
+            current_platform.is_cuda()
+            and topk_tokens in (512, 1024, 2048)
+            and num_rows <= 32
+            and logits.stride(0) % 4 == 0  # TMA 16-byte alignment
+            and device_capability is not None
+            and device_capability.to_int() == 90
+        )'''
+
+if not target.exists():
+    print(f"{target} not found; skipping SM120 cooperative_topk workaround")
+else:
+    text = target.read_text()
+    if "device_capability.to_int() == 90" in text:
+        print("SM120 cooperative_topk workaround already present; skipping")
+    elif old in text:
+        target.write_text(text.replace(old, new, 1))
+        print("Applied SM120 cooperative_topk workaround")
+    else:
+        print("Known cooperative_topk selector pattern not found; skipping")

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_spark_kv_cache_cleanup.py

@@ -0,0 +1,170 @@
+#!/usr/bin/env python3
+"""Clean profiling allocations before vLLM sizes and creates the KV cache."""
+
+import re
+import sys
+from pathlib import Path
+
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "vllm/v1/worker/gpu_worker.py"
+
+if not target.exists():
+    raise SystemExit(f"{target} not found; cannot apply KV cache cleanup patch")
+
+text = target.read_text()
+lines = text.splitlines(keepends=True)
+changed = False
+
+profile_cleanup_present = (
+    "profile_result.after_profile.measure()" in text
+    and "diff_from_create.non_torch_memory" in text
+)
+prealloc_cleanup_present = (
+    "memory_reserved(self.device)" in text
+    and "memory_allocated(self.device)" in text
+    and "empty_cache()" in text
+)
+needs_cleanup = not (profile_cleanup_present and prealloc_cleanup_present)
+
+if needs_cleanup and not re.search(r"(?m)^import gc$", text):
+    insert_at = None
+    last_future_import = None
+    for i, line in enumerate(lines):
+        if line.startswith("from __future__ import "):
+            last_future_import = i
+        elif insert_at is None and (
+            line.startswith("import ") or line.startswith("from ")
+        ):
+            insert_at = i
+    if last_future_import is not None:
+        lines.insert(last_future_import + 1, "import gc\n")
+    elif insert_at is not None:
+        lines.insert(insert_at, "import gc\n")
+    else:
+        lines.insert(0, "import gc\n")
+    changed = True
+
+
+def find_line(pattern: str) -> tuple[int, re.Match[str]]:
+    regex = re.compile(pattern)
+    for index, line in enumerate(lines):
+        match = regex.match(line)
+        if match:
+            return index, match
+    raise SystemExit(f"Could not find expected vLLM pattern: {pattern}")
+
+
+def insert_after_docstring(func_index: int, func_indent: str, block: list[str]) -> None:
+    insert_at = func_index + 1
+    if insert_at < len(lines):
+        stripped = lines[insert_at].lstrip()
+        quote = None
+        if stripped.startswith(chr(34) * 3):
+            quote = chr(34) * 3
+        elif stripped.startswith(chr(39) * 3):
+            quote = chr(39) * 3
+
+        if quote is not None:
+            if stripped.count(quote) >= 2 and not stripped.startswith(quote * 2):
+                insert_at += 1
+            else:
+                insert_at += 1
+                while insert_at < len(lines):
+                    if quote in lines[insert_at]:
+                        insert_at += 1
+                        break
+                    insert_at += 1
+
+    lines[insert_at:insert_at] = block
+
+
+if not profile_cleanup_present:
+    snapshot_line = (
+        r"^(?P<indent>[ \t]+)free_gpu_memory = "
+        r"profile_result\.after_profile\.free_memory\n$"
+    )
+    index, match = find_line(snapshot_line)
+    indent = match.group("indent")
+    lines[index:index] = [
+        f"{indent}# spark-vllm-docker: post-profile cleanup before KV sizing\n",
+        f'{indent}if self.device_config.device_type == "cuda":\n',
+        f"{indent}    before_cleanup = profile_result.after_profile.free_memory\n",
+        f'{indent}    if hasattr(self.model_runner, "_cleanup_profiling_kv_cache"):\n',
+        f"{indent}        self.model_runner._cleanup_profiling_kv_cache()\n",
+        f"{indent}    gc.collect()\n",
+        f"{indent}    torch.cuda.synchronize(self.device)\n",
+        f"{indent}    torch.cuda.empty_cache()\n",
+        f"{indent}    profile_result.after_profile.measure()\n",
+        f"{indent}    diff_from_create = (\n",
+        f"{indent}        profile_result.after_profile "
+        "- profile_result.before_create\n",
+        f"{indent}    )\n",
+        f"{indent}    profile_result.non_torch_increase = (\n",
+        f"{indent}        diff_from_create.non_torch_memory\n",
+        f"{indent}    )\n",
+        f"{indent}    profile_result.non_kv_cache_memory = (\n",
+        f"{indent}        profile_result.non_torch_increase\n",
+        f"{indent}        + profile_result.torch_peak_increase\n",
+        f"{indent}        + profile_result.weights_memory\n",
+        f"{indent}    )\n",
+        f"{indent}    cleanup_freed = (\n",
+        f"{indent}        profile_result.after_profile.free_memory - before_cleanup\n",
+        f"{indent}    )\n",
+        f"{indent}    if cleanup_freed > 0:\n",
+        f"{indent}        logger.info_once(\n",
+        f'{indent}            "Freed %.2f GiB before KV cache sizing; "\n',
+        f'{indent}            "non-torch profile increase is %.2f GiB.",\n',
+        f"{indent}            cleanup_freed / (1024**3),\n",
+        f"{indent}            profile_result.non_torch_increase / (1024**3),\n",
+        f"{indent}        )\n",
+        "\n",
+    ]
+    changed = True
+
+if not prealloc_cleanup_present:
+    func_index = None
+    func_indent = None
+    for i, line in enumerate(lines):
+        match = re.match(
+            r"^(?P<indent>[ \t]+)def initialize_from_config"
+            r"\(self,\s*kv_cache_config\b",
+            line,
+        )
+        if match:
+            func_index = i
+            func_indent = match.group("indent")
+            break
+
+    if func_index is None or func_indent is None:
+        raise SystemExit("Could not find initialize_from_config in vLLM gpu_worker.py")
+
+    body_indent = func_indent + "    "
+    block = [
+        f"{body_indent}# spark-vllm-docker: pre-KV cache allocator cleanup\n",
+        f'{body_indent}if self.device_config.device_type == "cuda":\n',
+        f"{body_indent}    gc.collect()\n",
+        f"{body_indent}    torch.cuda.synchronize(self.device)\n",
+        f"{body_indent}    cached_memory = max(\n",
+        f"{body_indent}        torch.cuda.memory_reserved(self.device)\n",
+        f"{body_indent}        - torch.cuda.memory_allocated(self.device),\n",
+        f"{body_indent}        0,\n",
+        f"{body_indent}    )\n",
+        f"{body_indent}    torch.cuda.empty_cache()\n",
+        f"{body_indent}    if cached_memory > 0:\n",
+        f"{body_indent}        logger.info_once(\n",
+        f'{body_indent}            "Cleared %.2f GiB of cached CUDA allocator '
+        'memory before "\n',
+        f'{body_indent}            "KV cache allocation.",\n',
+        f"{body_indent}            cached_memory / (1024**3),\n",
+        f"{body_indent}        )\n",
+        "\n",
+    ]
+    insert_after_docstring(func_index, func_indent, block)
+    changed = True
+
+if changed:
+    target.write_text("".join(lines))
+    print("Applied Spark KV cache cleanup patch")
+else:
+    print("Equivalent Spark KV cache cleanup already present; skipping")

docker-images/glm53-jovian-r12/patches-eugr/patch_vllm_topk_softplus_sqrt_control_flow.py

@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+"""Fix the misplaced XPU-only return introduced by vLLM PR #49408."""
+
+import sys
+from pathlib import Path
+
+
+source_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path.cwd()
+target = source_root / "vllm/_custom_ops.py"
+function_marker = "def topk_hash_softplus_sqrt("
+direct_call = "\n    torch.ops._moe_C.topk_softplus_sqrt("
+misplaced_return = "\n\n    return" + direct_call
+fixed_return = "\n        return\n" + direct_call
+
+if not target.exists():
+    raise SystemExit(f"{target} not found; cannot inspect vLLM PR #49408 regression")
+
+text = target.read_text()
+start = text.find(function_marker)
+if start == -1:
+    print("topk_hash_softplus_sqrt is absent; vLLM PR #49408 is not applicable")
+else:
+    end = text.find("\ndef ", start + len(function_marker))
+    end = len(text) if end == -1 else end
+    function = text[start:end]
+
+    if "is_padding" not in function:
+        print("topk_hash_softplus_sqrt predates vLLM PR #49408; skipping workaround")
+    elif fixed_return in function:
+        print("vLLM topk_softplus_sqrt non-XPU path already fixed; skipping")
+    elif (
+        direct_call in function
+        and "current_platform.is_xpu()" not in function
+        and "\n    return" not in function
+    ):
+        print("topk_hash_softplus_sqrt has no XPU workaround; patch is not applicable")
+    elif function.count(misplaced_return) == 1:
+        function = function.replace(misplaced_return, fixed_return, 1)
+        updated = text[:start] + function + text[end:]
+        compile(updated, str(target), "exec")
+        target.write_text(updated)
+        print("Applied vLLM PR #49452 topk_softplus_sqrt control-flow fix")
+    else:
+        raise SystemExit(
+            "Unknown is_padding-aware topk_hash_softplus_sqrt layout; "
+            "refusing to guess whether vLLM PR #49408 is fixed"
+        )

recipes/glm53-flash-nvfp4-spark/recipe.toml

@@ -74,13 +74,15 @@ args = [
     # _select_sparse_backend); the sm12x auto-priority list never offers it
     # (FLASHINFER_MLA_SPARSE_SM120 only supports head 576, not GLM's 512).
     "--attention-backend", "B12X",
-    # lil family policy: GDN linear-attention + routed-expert MoE on the
-    # B12X kernels (the GB10 fast path for this quant).
-    "--linear-backend", "b12x",
-    "--moe-backend", "b12x",
     # lab loader: instanttensor (0.1.9 is already in the r12 image) —
-    # ~1 min weight load vs ~16 min from NFS
+    # ~5 min weight load vs ~16 min from NFS; stays in every bisection run
     "--load-format", "instanttensor",
+    # PERF RUN (2026-09-02): full lab execution engine per lil render
+    # (vllm-node-eugr reference): b12x linear+MoE kernels, mamba state
+    # cache align, no enforce-eager (lab runs CUDA graphs; the enforce-eager
+    # quality bug was day-0-build-specific, r12 is the lab-qualified build).
+    "--linear-backend", "b12x",
+    "--moe-backend", "b12x",
     "--mamba-cache-mode", "align",
     "--block-size", "256",
     "--max-model-len", "524288",
@@ -93,23 +95,12 @@ args = [
     # Their proven boot: no flashinfer autotune, no CuTe-DSL warmup
     "--kernel-config", """
 '{"enable_cutedsl_warmup":false,"enable_flashinfer_autotune":false}'""",
-    # Built-in MTP draft, 5 tokens — lil's mtp_tokens for this checkpoint
-    # (all 5 MTP layers are embedded in the model-mtp-0000x shards). The
-    # draft model's vllm_config takes its attention backend from the
-    # SPECULATIVE config, not the global flag (eagle/utils.py:
-    # _make_eagle_draft_vllm_config) — B12X mirrors the main model's;
-    # moe_backend b12x per lil's MTPMoEBackendFromQuantization (nvfp4→b12x).
-    # A/B vs k=3 if accept-rate/quality regresses (k=3 was the r12 x86
-    # reference: 254 tok/s CC1, 2.37 accept).
+    # lab spec config verbatim (lil render): k=5, moe_backend b12x
+    # (nvfp4->b12x), attention B12X (draft's backend comes from the spec
+    # config, not the global flag — eagle/utils.py _make_eagle_draft_vllm_config)
     "--speculative-config", """
 '{"method":"mtp","num_speculative_tokens":5,"moe_backend":"b12x","attention_backend":"B12X"}'""",
     "--language-model-only",
-    # MTP + CUDA graphs degrades OUTPUT QUALITY on the glm53 day-0 dev build
-    # (forum #381350/#44, CONFIRMED by A/B bench 2026-08-28: same seed/flags,
-    # graphs on = 89/100, eager = 92/100; decode unchanged 28.4 t/s @ c1,
-    # c2 -7%, TTFT +1.6s). The b12x glm52 stack does NOT need this (its
-    # graphs+MTP combo benches clean at 100/100) — bug is build-specific.
-    "--enforce-eager",
     "--tool-call-parser", "glm47",
     "--enable-auto-tool-choice",
     "--reasoning-parser", "glm45",
@@ -117,7 +108,11 @@ args = [
 
 [env]
 HF_HUB_OFFLINE = "1"
-# lab lil.yaml INSTANTTENSOR knobs
+# lab INSTANTTENSOR loader knobs (lil render: BACKEND/CONCURRENCY/IO_DEPTH
+# + the lil.yaml BUFFER_SIZE/CHUNK)
+INSTANTTENSOR_BACKEND = "BUFFERED"
+INSTANTTENSOR_CONCURRENCY = "1"
+INSTANTTENSOR_IO_DEPTH = "3"
 INSTANTTENSOR_BUFFER_SIZE = "67108864"
 INSTANTTENSOR_CHUNK_SIZE = "8388608"
 # FlashInfer JIT guard: if a kernel is missing from the mounted JIT cache it
@@ -127,9 +122,15 @@ MAX_JOBS = "2"
 # NoPE-MLA rope-pad runtime gate (mod baked into the image)
 VLLM_MLA_NOPE_PAD_ROPE = "1"
 VLLM_ENGINE_READY_TIMEOUT_S = "3600"
-# Upstream launch-script env (memory + compile knobs)
+# PERF RUN: lab execution engine (lil render reference)
+VLLM_USE_AOT_COMPILE = "1"
+VLLM_USE_STANDALONE_COMPILE = "1"
+VLLM_USE_V2_MODEL_RUNNER = "1"
+VLLM_USE_FLASHINFER_SAMPLER = "1"
+VLLM_USE_BREAKABLE_CUDAGRAPH = "0"
+# NOTE: no VLLM_USE_MEGA_AOT_ARTIFACT — the prebuilt mega-AOT artifact ships
+# in the lab image, not in ours; the flag would only serialize.
 PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
-VLLM_USE_AOT_COMPILE = "0"
 TORCH_USE_RTLD_GLOBAL = "1"
 VLLM_NO_USAGE_STATS = "1"
 

specs/tech-architecture/glm53-recipe-state.md

@@ -448,3 +448,30 @@ A/B k=3 vs k=5 (lil default), --load-format instanttensor (+env),
 --mamba-cache-mode align, --max-num-seqs 8, --max-num-batched-tokens 4096,
 --reasoning-parser glm45, --tool-call-parser glm47.
 Then: Intel W4A16 boot on the clean image (queued, needs cluster).
+
+## 2026-09-02 (cont 4): lab-stack perf push — eugr build pipeline adopted
+
+Goal (user): 30 tok/s in --perf. Found the missing pieces:
+1. **lil render reference** (built lil from local-inference-lab/lil @ cb58d54,
+   rendered with testdata manifest GLM-5.3-Flash-NVFP4-Spark + spark.yaml):
+   our recipe flags now 100% match the family policy (_bases.yaml glm: b12x
+   linear/moe, B12X attention, instanttensor, block 256, seqs 8, batched 4096,
+   glm45/glm47, mamba align) + spec {mtp:5, moe_backend b12x, B12X} + env
+   VLLM_USE_AOT_COMPILE=1 / V2_MODEL_RUNNER=1 / FLASHINFER_SAMPLER=1 /
+   BREAKABLE_CUDAGRAPH=0. Boot 35m -> ~7m (instanttensor loader, user: keep).
+   Perf after engine switch: d0 c1 24.2 / c4 31.8 tok/s (was 10.3/16.5).
+   Quality dropped with full lil set (16/22 at TC-22) -> bisection in
+   progress; user: instanttensor is fine, problem elsewhere.
+2. **eugr/spark-vllm-docker** (the lab's actual build repo, main): 804-line
+   Dockerfile, CUDA 13.0.2 base, 12 build-time vllm patches (docker/patch_vllm_*.py),
+   b12x from source (lukealonso/b12x; lab: local-inference-lab/b12x master),
+   nightly DockerHub images eugr/spark-vllm[-b12x]:latest (stale 08-23,
+   user: dockerhub is the wrong source — build from source).
+   Ported all 12 patches into docker-images/glm53-jovian-r12/patches-eugr/
+   + Dockerfile (VLLM_PRESERVE_SM12X_TARGET=1, VLLM_PATCH_B12X_C128A_ALIGNMENT=1;
+   patches are idempotent, several no-op on our r12 ref).
+   Build blocker: node github.com access flaky ("could not read Username" on
+   FetchContent clones) -> pre-fetched all 9 sub-deps on the Mac into
+   deps-src.tar.gz (triton_kernels@v3.5.1, MSA, FlashMLA, FlashKDA,
+   flash-attention, qutlass@e74319e, tml-fa4@b206834, cutlass@v4.4.2) +
+   the SRC_DIR env escapes each cmake documents. Build 14 compiling.