recipes: dsv4-vision-exp-vavrusa — native image_url lane (eugr nightly 20260810 + instanttensor-draft + vision-exp mods), DSpark-6, NVMe weights; verified text-exact + real-photo caption (325 img tokens)

recipes/dsv4-vision-exp-vavrusa/mods/deepseek-v4-vision-exp/README.md

@@ -0,0 +1,94 @@
+# DeepSeek-V4-Flash-Vision-Exp native vision mod
+
+Adds **native image input** for `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` to a
+vLLM that otherwise serves the text-only `DeepseekV4ForCausalLM`. Ported from
+[MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark)
+(`patches/hotfix-dsv4-vision-exp.py` + `patches/vision_exp/`).
+
+## Why a patch is required
+
+The Vision-Exp checkpoint declares the **same**
+`architectures: ["DeepseekV4ForCausalLM"]` string as the text-only 0731 model
+while carrying 316 extra tensors — a 32-block ViT, a 2-layer aligner, four
+learned image embeddings, and a `bias_vl` MoE routing bias on every layer.
+vLLM's `DeepseekV4ForCausalLM` is text-only, so loading the vision checkpoint
+fails with `ValueError: There is no module or parameter named 'aligner'`.
+
+DeepSeek ships only a reference implementation (`inference/` in their repo —
+"a readable reference implementation rather than a production serving engine"),
+so there is nothing to configure: the ViT + aligner and the multimodal
+processor must be added to vLLM's vendored `deepseek_v4/nvidia/model.py`.
+
+## Mechanism
+
+`run.sh` stages `vision_exp/` to `/opt/dspark-patches/vision_exp` (the overlay
+package root — the hotfix's `DEFAULT_PATCHES` points there) and runs
+`hotfix-dsv4-vision-exp.py`, which:
+
+1. **Injects a fail-closed import hook** at the end of `nvidia/model.py` that
+   constructs `vision` / `aligner` / `image_{start,end,newline,pad}` on
+   `DeepseekV4Model` when `config.vision_n_layers > 0`, maps the `vision.*` /
+   `aligner.*` / `image_*` / `bias_vl` weights, and registers a vLLM multimodal
+   processor (`vision_exp/processor.py`).
+2. **Remaps DSpark draft `ffn.gate.bias_vl`** → `e_score_correction_bias_vl` in
+   `dspark.py` (the draft loader only rewrote names ending in `.ffn.gate.bias`).
+3. Clears `__pycache__` and the stale `$VLLM_CACHE_ROOT/modelinfos/` cache (the
+   disk-cached multimodal inspection — a prior text-only boot leaves a stale
+   entry that silently reuses the "text-only" classification).
+
+The whole thing is idempotent and fail-closed on anchor drift.
+
+## Recipe usage
+
+Point the recipe at the **vision checkpoint** and add the mod:
+
+```yaml
+model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
+mods:
+  - mods/deepseek-v4-vision-exp
+```
+
+`served_model_name` should stay `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp`.
+
+## b12x fused-MHC `rms_norm_eps` fix (eugr b12x images only)
+
+The Vision-Exp checkpoint ships `rms_norm_eps: 1e-20` (0731 used `1e-6`). With
+`VLLM_USE_B12X_MHC=1`, the b12x fused MHC Gram **decode** kernel hard-requires
+`rms_eps == 1e-6` (`b12x.norm.mhc._impl._supports_fused_mhc_gram`); 1e-20 falls
+through to a `ValueError` during `determine_available_memory` ("b12x_mhc_pre is
+served only by the fused Gram kernel"). Lineages that route MHC through a
+TileLang kernel (`mhc_pre_tilelang`) accept any eps and are unaffected.
+
+`apply_vision_exp`'s `model_init` hook forces `config.rms_norm_eps = 1e-6`
+before `DeepseekV4Model.__init__` reads it, **guarded on `vision_n_layers > 0`**
+so text-only checkpoints are untouched. The DSpark MTP draft is built from the
+same shared `hf_config` (speculative `target_model_config.hf_config`), so it
+inherits the override.
+
+**Correctness of the override:** RMSNorm computes
+`x * rsqrt(mean(x²) + eps)`; eps is a div-by-zero stabilizer. On realistic
+DeepSeek V4 hidden states (hidden_size 4096, bf16, variance O(1)), RMSNorm with
+`eps=1e-6` vs `eps=1e-20` produced **bit-identical bf16 output** (0/131072
+elements differ). So the override is exact for bf16, not merely approximately
+equal.
+
+To instead keep `rms_norm_eps=1e-20` exactly, set `VLLM_USE_B12X_MHC=0` in the
+recipe env — this routes MHC to the TileLang path (no eps constraint) but drops
+the b12x fused decode kernel.
+
+## Notes / limitations
+
+- **Images in `user` messages only** — `system` / `assistant` images return
+  HTTP 400 (official Chat Completions restriction).
+- **Max 384 image tokens** per image (`vision_max_n_token`).
+- **DSpark k must be a multiple of 3.** Vision-Exp has
+  `num_nextn_predict_layers=3` (0731 had 1), so k=5 is rejected
+  (`k % 3 == 0` when `k > n_predict`). Use `num_speculative_tokens: 6` — the
+  smallest k that is also ≥ dspark block size 5.
+- **Fidelity gap (documented, not silent):** image tokens use the standard
+  causal sparse attention pattern — the reference's bidirectional attention
+  *within* each `[IMAGE_START, IMAGE_END]` span is not ported. Expect some
+  quality loss on image-heavy prompts.
+- `bias_vl` is loaded but not applied at routing time — image tokens route
+  through the text bias. Known limitation in the reference ports; text-only
+  requests are byte-identical to 0731.

recipes/dsv4-vision-exp-vavrusa/mods/deepseek-v4-vision-exp/hotfix-dsv4-vision-exp.py

@@ -0,0 +1,283 @@
+#!/usr/bin/env python3
+"""Install native DeepSeek-V4-Flash-Vision-Exp image support into Anemll vLLM.
+
+The 0.1.1 image's ``DeepseekV4ForCausalLM`` is text-only. Vision-Exp ships a
+32-layer ViT + Aligner and ``<｜deepseek_image｜>`` prompt tokens. This
+startup patch:
+
+1. Appends a fail-closed import hook to ``nvidia/model.py`` that constructs
+   the tower, maps ``vision.*`` / ``aligner.*`` / ``image_*`` / ``bias_vl``
+   weights, and registers a vLLM multimodal processor.
+2. Remaps DSpark draft ``ffn.gate.bias_vl`` → ``e_score_correction_bias_vl``
+   (Anemll only rewrote names ending in ``.ffn.gate.bias``).
+3. Relaxes the Vision-Exp encoder's rejection of already-substituted
+   ``<｜deepseek_image｜>`` text so OpenAI ``image_url`` parts survive
+   vLLM's chat parser, then restores the official Chat Completions rule:
+   images in ``user`` messages only (``system`` / ``assistant`` → 400).
+
+Video is not wired: the official weights, ``encoding/``, and ``inference/``
+have no video encoder. GIF is decoded as a still RGB frame.
+
+Usage (inside the container, after the encoder copy):
+  python3 hotfix-dsv4-vision-exp.py
+"""
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+DEFAULT_MODEL = Path(
+    "/usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v4/nvidia/model.py"
+)
+DEFAULT_ENCODING = Path(
+    "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/deepseek_v4_encoding.py"
+)
+DEFAULT_DSPARK = Path(
+    "/usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v4/nvidia/dspark.py"
+)
+DEFAULT_PATCHES = Path("/opt/dspark-patches/vision_exp")
+
+MODEL_MARK = "# [vision-exp-hotfix] native DeepSeek-V4-Flash-Vision-Exp image tower"
+ENC_MARK = "# [vision-exp-hotfix] allow vLLM-inserted image placeholders"
+ENC_ROLE_MARK = "# [vision-exp-hotfix] images only in user messages"
+ENC_ROLE_PAIRED_MARK = "# [vision-exp-hotfix] paired <image> tag (issue 165)"
+DSPARK_MARK = "# [vision-exp-hotfix] remap ffn.gate.bias_vl"
+
+DSPARK_GATE_BIAS_OLD = '''                if name.endswith(".ffn.gate.bias"):
+                    name = name.replace(
+                        ".ffn.gate.bias", ".ffn.gate.e_score_correction_bias"
+                    )
+                param = params_dict[name]'''
+
+DSPARK_GATE_BIAS_NEW = f'''                if name.endswith(".ffn.gate.bias_vl"):
+                    name = name.replace(
+                        ".ffn.gate.bias_vl",
+                        ".ffn.gate.e_score_correction_bias_vl",
+                    )
+                elif name.endswith(".ffn.gate.bias"):
+                    name = name.replace(
+                        ".ffn.gate.bias", ".ffn.gate.e_score_correction_bias"
+                    )
+                if name not in params_dict:
+                    continue  {DSPARK_MARK}
+                param = params_dict[name]'''
+
+MODEL_INJECT = f'''
+{MODEL_MARK}
+import sys as _dspark_vision_sys
+if "/opt/dspark-patches" not in _dspark_vision_sys.path:
+    _dspark_vision_sys.path.insert(0, "/opt/dspark-patches")
+from vision_exp.apply import apply_vision_exp as _dspark_apply_vision_exp
+_dspark_apply_vision_exp(
+    DeepseekV4Model=DeepseekV4Model,
+    DeepseekV4ForCausalLM=DeepseekV4ForCausalLM,
+    DeepseekV4MoE=DeepseekV4MoE,
+)
+'''
+
+CONTENT_CHECK = (
+    "if isinstance(content, str) and IMAGE_PLACEHOLDER in content:"
+)
+CONTENT_CHECK_NEW = (
+    "if False and isinstance(content, str) and IMAGE_PLACEHOLDER in content:"
+    f"  {ENC_MARK}"
+)
+REASONING_CHECK = (
+    "if isinstance(reasoning_content, str) and IMAGE_PLACEHOLDER in reasoning_content:"
+)
+REASONING_CHECK_NEW = (
+    "if False and isinstance(reasoning_content, str) and IMAGE_PLACEHOLDER in reasoning_content:"
+    f"  {ENC_MARK}"
+)
+TEXT_CHECK = "if IMAGE_PLACEHOLDER in text:"
+TEXT_CHECK_NEW = f"if False and IMAGE_PLACEHOLDER in text:  {ENC_MARK}"
+
+ENC_ROLE_INJECT = f'''
+{ENC_ROLE_MARK}
+{ENC_ROLE_PAIRED_MARK}
+def _dspark_vision_text_has_image(text: str) -> bool:
+    if IMAGE_PLACEHOLDER in text:
+        return True
+    needle, close = "<image>", "</image>"
+    start = 0
+    while True:
+        i = text.find(needle, start)
+        if i < 0:
+            return False
+        if text.find(close, i + len(needle)) >= 0:
+            return True
+        start = i + len(needle)
+
+
+def _dspark_vision_value_has_image(value) -> bool:
+    if isinstance(value, str):
+        return _dspark_vision_text_has_image(value)
+    if not isinstance(value, list):
+        return False
+    for block in value:
+        if not isinstance(block, dict):
+            continue
+        if block.get("type") in ("image", "image_url"):
+            return True
+        text = block.get("text") or ""
+        if isinstance(text, str) and _dspark_vision_text_has_image(text):
+            return True
+        nested = block.get("content")
+        if isinstance(nested, list) and _dspark_vision_value_has_image(nested):
+            return True
+    return False
+
+
+def _validate_no_image_sp_tokens(msg):
+    """Official restriction: images in user messages only (system/assistant → 400)."""
+    reasoning_content = msg.get("reasoning_content")
+    if isinstance(reasoning_content, str) and IMAGE_PLACEHOLDER in reasoning_content:
+        raise ValueError(
+            "reasoning_content contains image special token "
+            + repr(IMAGE_PLACEHOLDER)
+        )
+    role = msg.get("role")
+    if role in ("user", "developer"):
+        return
+    if _dspark_vision_value_has_image(msg.get("content")) or _dspark_vision_value_has_image(
+        msg.get("content_blocks")
+    ):
+        raise ValueError(
+            "Images are supported in user messages only: "
+            "images in " + repr(role) + " messages return a 400 error."
+        )
+'''
+
+
+def patch_model_text(source: str) -> tuple[str, str]:
+    if MODEL_MARK in source:
+        return source, "skipped"
+    if "class DeepseekV4ForCausalLM" not in source or "class DeepseekV4MoE" not in source:
+        return source, "drift:missing-dsv4-class"
+    updated = source.rstrip() + "\n" + MODEL_INJECT
+    compile(updated, "model.py", "exec")
+    return updated, "applied"
+
+
+def patch_encoding_text(source: str) -> tuple[str, str]:
+    if (
+        ENC_MARK in source
+        and ENC_ROLE_MARK in source
+        and ENC_ROLE_PAIRED_MARK in source
+        and CONTENT_CHECK_NEW in source
+    ):
+        return source, "skipped"
+    if "IMAGE_PLACEHOLDER" not in source:
+        # Lineages whose deepseek_v4_encoding.py has no image-placeholder logic
+        # have nothing to relax here. The Anemll image this hotfix was written
+        # for rejects already-substituted placeholders at the encoder; a
+        # lineage without them makes the encoding patch a no-op.
+        return source, "skipped-no-placeholder"
+    if ENC_MARK not in source:
+        missing = []
+        for old, new in (
+            (CONTENT_CHECK, CONTENT_CHECK_NEW),
+            (REASONING_CHECK, REASONING_CHECK_NEW),
+            (TEXT_CHECK, TEXT_CHECK_NEW),
+        ):
+            if source.count(old) != 1:
+                missing.append(f"{old!r}={source.count(old)}")
+                continue
+            source = source.replace(old, new, 1)
+        if missing:
+            return source, "drift:" + ",".join(missing)
+    if ENC_ROLE_MARK in source and ENC_ROLE_PAIRED_MARK not in source:
+        source = source[: source.rfind(ENC_ROLE_MARK)].rstrip() + "\n"
+    if ENC_ROLE_MARK not in source:
+        source = source.rstrip() + "\n" + ENC_ROLE_INJECT
+    compile(source, "encoding.py", "exec")
+    return source, "applied"
+
+
+def patch_dspark_text(source: str) -> tuple[str, str]:
+    if DSPARK_MARK in source and DSPARK_GATE_BIAS_NEW in source:
+        return source, "skipped"
+    if source.count(DSPARK_GATE_BIAS_OLD) != 1:
+        return source, (
+            "drift:dspark-gate-bias-remap="
+            f"{source.count(DSPARK_GATE_BIAS_OLD)}"
+        )
+    updated = source.replace(DSPARK_GATE_BIAS_OLD, DSPARK_GATE_BIAS_NEW, 1)
+    compile(updated, "dspark.py", "exec")
+    return updated, "applied"
+
+
+def _write(path: Path, original: str, updated: str, status: str) -> None:
+    if status == "applied":
+        path.write_text(updated)
+    elif status not in ("skipped", "skipped-no-placeholder"):
+        raise SystemExit(f"FATAL: {path} {status}")
+    print(f"vision-exp hotfix {path.name:40s}: {status}")
+
+
+def main() -> int:
+    if len(sys.argv) > 1 and sys.argv[1] == "--status":
+        model = DEFAULT_MODEL.read_text() if DEFAULT_MODEL.is_file() else ""
+        encoding = DEFAULT_ENCODING.read_text() if DEFAULT_ENCODING.is_file() else ""
+        dspark = DEFAULT_DSPARK.read_text() if DEFAULT_DSPARK.is_file() else ""
+        print(
+            "vision-exp model.py                    :",
+            "APPLIED" if MODEL_MARK in model else "NOT APPLIED",
+        )
+        print(
+            "vision-exp encoding.py                 :",
+            "APPLIED"
+            if ENC_MARK in encoding and ENC_ROLE_MARK in encoding
+            and ENC_ROLE_PAIRED_MARK in encoding
+            else "NOT APPLIED",
+        )
+        print(
+            "vision-exp dspark.py                   :",
+            "APPLIED" if DSPARK_MARK in dspark else "NOT APPLIED",
+        )
+        ok = (
+            MODEL_MARK in model
+            and ENC_MARK in encoding
+            and ENC_ROLE_MARK in encoding
+            and ENC_ROLE_PAIRED_MARK in encoding
+            and DSPARK_MARK in dspark
+        )
+        return 0 if ok else 1
+
+    patches = Path(sys.argv[1]) if len(sys.argv) > 1 else DEFAULT_PATCHES
+    model_path = Path(sys.argv[2]) if len(sys.argv) > 2 else DEFAULT_MODEL
+    encoding_path = Path(sys.argv[3]) if len(sys.argv) > 3 else DEFAULT_ENCODING
+    dspark_path = Path(sys.argv[4]) if len(sys.argv) > 4 else DEFAULT_DSPARK
+
+    if not (patches / "apply.py").is_file() or not (patches / "vision.py").is_file():
+        print(f"FATAL: Vision-Exp overlay missing under {patches}", file=sys.stderr)
+        return 1
+    if not model_path.is_file():
+        print(f"FATAL: {model_path} missing", file=sys.stderr)
+        return 1
+    if not encoding_path.is_file():
+        print(
+            f"FATAL: {encoding_path} missing (encoder copy must run first)",
+            file=sys.stderr,
+        )
+        return 1
+    if not dspark_path.is_file():
+        print(f"FATAL: {dspark_path} missing", file=sys.stderr)
+        return 1
+
+    model_src = model_path.read_text()
+    model_new, model_status = patch_model_text(model_src)
+    _write(model_path, model_src, model_new, model_status)
+
+    enc_src = encoding_path.read_text()
+    enc_new, enc_status = patch_encoding_text(enc_src)
+    _write(encoding_path, enc_src, enc_new, enc_status)
+
+    dspark_src = dspark_path.read_text()
+    dspark_new, dspark_status = patch_dspark_text(dspark_src)
+    _write(dspark_path, dspark_src, dspark_new, dspark_status)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

recipes/dsv4-vision-exp-vavrusa/mods/deepseek-v4-vision-exp/run.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+# run.sh — DeepSeek-V4-Flash-Vision-Exp native vision mod
+#
+# Ported from MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark
+# (patches/hotfix-dsv4-vision-exp.py + patches/vision_exp/). Adds native image
+# input to a vLLM that serves deepseek-ai/DeepSeek-V4-Flash-Vision-Exp.
+#
+# Target layout (Anemll / eugr-style DeepSeek-V4 fork):
+#   model.py at /usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v4/
+#     nvidia/model.py with DeepseekV4ForCausalLM / DeepseekV4MoE /
+#     DeepseekV4Model classes (apply.py's monkeypatch anchors), plus
+#     nvidia/dspark.py and tokenizers/deepseek_v4_encoding.py. If the encoding
+#     file has no IMAGE_PLACEHOLDER the encoding patch is skipped gracefully.
+#
+# What it does:
+#   1. Stage patches/vision_exp -> /opt/dspark-patches/vision_exp (the injected
+#      import hook reads from there).
+#   2. Run hotfix-dsv4-vision-exp.py: injects a fail-closed import hook into
+#      nvidia/model.py that builds the ViT+Aligner, maps vision./aligner./
+#      image_*/bias_vl weights, registers a vLLM multimodal processor, makes
+#      embed_input_ids scatter image-block embeddings, and remaps the DSpark
+#      draft ffn.gate.bias_vl -> e_score_correction_bias_vl in dspark.py.
+#   3. Clear __pycache__ and the stale $VLLM_CACHE_ROOT/modelinfos/ cache (the
+#      disk-cached multimodal inspection keyed by module+class — a prior
+#      text-only boot would leave a stale entry).
+#
+# Notes: images go in user messages only (official restriction). Max 384 image
+# tokens per image. num_nextn_predict_layers=3 on Vision-Exp -> DSpark k must be
+# a multiple of 3 (k=6 recommended; k=5 is rejected).
+set -euo pipefail
+
+PREFIX="[deepseek-v4-vision-exp]"
+MOD_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SITE="/usr/local/lib/python3.12/dist-packages/vllm"
+MODEL="$SITE/models/deepseek_v4/nvidia/model.py"
+DSPARK="$SITE/models/deepseek_v4/nvidia/dspark.py"
+ENCODING="$SITE/tokenizers/deepseek_v4_encoding.py"
+PATCHES_DIR="/opt/dspark-patches"
+HOTFIX="$MOD_DIR/hotfix-dsv4-vision-exp.py"
+
+echo "=== deepseek-v4-vision-exp native vision mod ==="
+
+if [ ! -d "$SITE" ]; then
+    echo "$PREFIX vLLM tree not found at $SITE" >&2
+    exit 1
+fi
+if [ ! -f "$MODEL" ]; then
+    echo "$PREFIX nvidia/model.py not found at $MODEL" >&2
+    exit 1
+fi
+if [ ! -f "$DSPARK" ]; then
+    echo "$PREFIX nvidia/dspark.py not found at $DSPARK" >&2
+    exit 1
+fi
+if [ ! -f "$HOTFIX" ] || [ ! -d "$MOD_DIR/vision_exp" ]; then
+    echo "$PREFIX hotfix or vision_exp package missing under $MOD_DIR" >&2
+    exit 1
+fi
+
+# Stage the vision_exp package where the injected import hook looks for it.
+mkdir -p "$PATCHES_DIR"
+rm -rf "$PATCHES_DIR/vision_exp"
+cp -r "$MOD_DIR/vision_exp" "$PATCHES_DIR/vision_exp"
+echo "$PREFIX staged vision_exp -> $PATCHES_DIR/vision_exp"
+
+# Run the hotfix. It patches model.py + dspark.py in place (idempotent,
+# fail-closed on anchor drift) and clears the encoding.py only if that file
+# actually has placeholder logic (no-op on lineages without it).
+python3 "$HOTFIX"
+
+# Clear stale bytecode + the disk-cached multimodal inspection.
+find "$SITE" -name "__pycache__" -type d -prune -exec rm -rf {} + 2>/dev/null || true
+CACHE_ROOT="${VLLM_CACHE_ROOT:-$HOME/.cache/vllm}"
+if [ -d "$CACHE_ROOT/modelinfos" ]; then
+    rm -rf "$CACHE_ROOT/modelinfos"
+    echo "$PREFIX cleared stale $CACHE_ROOT/modelinfos"
+fi
+
+echo "=== OK: DeepSeek-V4-Flash-Vision-Exp native vision installed ==="

recipes/dsv4-vision-exp-vavrusa/mods/deepseek-v4-vision-exp/vision_exp/__init__.py

@@ -0,0 +1 @@
+"""DeepSeek-V4-Flash-Vision-Exp tower + vLLM processor (Anemll 0.1.1 hotfix)."""

recipes/dsv4-vision-exp-vavrusa/mods/deepseek-v4-vision-exp/vision_exp/apply.py

@@ -0,0 +1,315 @@
+"""Monkey-patch Anemll DeepseekV4ForCausalLM for Vision-Exp images.
+
+Called at the end of ``vllm/models/deepseek_v4/nvidia/model.py`` after the
+stock classes are defined. Video is not registered: the checkpoint has no
+video tower.
+"""
+from __future__ import annotations
+
+from typing import Any
+
+import torch
+from torch import nn
+
+from .image_processor import (
+    IMAGE,
+    is_unregistered_router_bias,
+    is_vision_exp_weight_name,
+    vision_args_from_config,
+)
+from .processor import IMAGE_PLACEHOLDER, register_vision_exp_processor
+from .vision import Aligner, ViT
+
+VISION_MAPPER_PREFIXES = {
+    "vision.": "model.vision.",
+    "aligner.": "model.aligner.",
+    "image_start": "model.image_start",
+    "image_end": "model.image_end",
+    "image_newline": "model.image_newline",
+    "image_pad": "model.image_pad",
+}
+VISION_MAPPER_SUFFIXES = {
+    ".ffn.gate.bias_vl": ".ffn.gate.e_score_correction_bias_vl",
+}
+
+
+def _as_rows(value: Any) -> list[Any]:
+    if value is None:
+        return []
+    if isinstance(value, (list, tuple)):
+        return list(value)
+    if isinstance(value, torch.Tensor):
+        if value.ndim == 0:
+            return [value]
+        return [value[i] for i in range(value.shape[0])]
+    return [value]
+
+
+def _scalar(value: Any) -> int:
+    if hasattr(value, "reshape"):
+        return int(value.reshape(-1)[0].item())
+    return int(value)
+
+
+def _extend_weights_mapper(mapper):
+    from vllm.model_executor.models.utils import WeightsMapper
+
+    extra = WeightsMapper(
+        orig_to_new_prefix=dict(VISION_MAPPER_PREFIXES),
+        orig_to_new_suffix=dict(VISION_MAPPER_SUFFIXES),
+    )
+    return mapper | extra
+
+
+def _install_vision_tower(model: nn.Module, config) -> None:
+    if getattr(model, "vision", None) is not None:
+        return
+    args = vision_args_from_config(config)
+    if args.vision_n_layers <= 0:
+        model.vision = None
+        model.aligner = None
+        return
+    model.vision = ViT(args)
+    model.aligner = Aligner(args)
+    hidden = args.dim
+    model.image_start = nn.Parameter(torch.empty(hidden))
+    model.image_end = nn.Parameter(torch.empty(hidden))
+    model.image_newline = nn.Parameter(torch.empty(hidden))
+    model.image_pad = nn.Parameter(torch.empty(hidden))
+
+
+@torch.inference_mode()
+def encode_image(model: nn.Module, patches: torch.Tensor, n_vit_h: int, n_vit_w: int):
+    return model.aligner(model.vision(patches, n_vit_h, n_vit_w), n_vit_h, n_vit_w)
+
+
+def merge_one_image(
+    model: nn.Module,
+    patches: torch.Tensor,
+    n_vit_h: int,
+    n_vit_w: int,
+    types: torch.Tensor,
+    perm: torch.Tensor,
+    num_tokens: int,
+) -> torch.Tensor:
+    types = types[:num_tokens].to(dtype=torch.long)
+    perm = perm.to(dtype=torch.long)
+    perm = perm[perm >= 0]
+    n_patches = int(n_vit_h) * int(n_vit_w)
+    patches = patches[:n_patches]
+    device = next(model.vision.parameters()).device
+    dtype = next(model.aligner.parameters()).dtype
+    patches = patches.to(device=device, dtype=dtype)
+    embeds = encode_image(model, patches, int(n_vit_h), int(n_vit_w))[perm.to(device)]
+    params = torch.stack(
+        [
+            model.image_start,
+            model.image_pad,
+            model.image_pad,
+            model.image_newline,
+            model.image_end,
+        ]
+    ).to(device=device, dtype=embeds.dtype)
+    block = params[types.to(device)]
+    image_mask = types.to(device) == IMAGE
+    if int(image_mask.sum()) != int(embeds.shape[0]):
+        raise RuntimeError(
+            f"Vision-Exp layout mismatch: {int(image_mask.sum())} IMAGE slots, "
+            f"{int(embeds.shape[0])} aligner tokens"
+        )
+    block = block.clone()
+    block[image_mask] = embeds.to(block.dtype)
+    return block
+
+
+def embed_multimodal(self, **kwargs: object):
+    pixel_values = kwargs.get("pixel_values")
+    if pixel_values is None:
+        return []
+    pixels = _as_rows(pixel_values)
+    n_vit_h = _as_rows(kwargs.get("n_vit_h"))
+    n_vit_w = _as_rows(kwargs.get("n_vit_w"))
+    types = _as_rows(kwargs.get("types"))
+    perm = _as_rows(kwargs.get("perm"))
+    num_tokens = _as_rows(kwargs.get("num_tokens"))
+    inner = self.model
+    if getattr(inner, "vision", None) is None:
+        raise RuntimeError("Vision-Exp tower was not constructed on DeepseekV4Model")
+    out = []
+    for i, patches in enumerate(pixels):
+        out.append(
+            merge_one_image(
+                inner,
+                patches,
+                _scalar(n_vit_h[i]),
+                _scalar(n_vit_w[i]),
+                types[i],
+                perm[i],
+                _scalar(num_tokens[i]),
+            )
+        )
+    return out
+
+
+@classmethod
+def get_placeholder_str(cls, modality: str, i: int) -> str | None:
+    if modality.startswith("image"):
+        return IMAGE_PLACEHOLDER
+    raise ValueError(
+        f"DeepSeek-V4-Flash-Vision-Exp supports images only, got {modality!r}"
+    )
+
+
+def apply_vision_exp(
+    *,
+    DeepseekV4Model,
+    DeepseekV4ForCausalLM,
+    DeepseekV4MoE,
+) -> None:
+    from vllm.model_executor.models.interfaces import SupportsMultiModal
+
+    orig_model_init = DeepseekV4Model.__init__
+
+    def model_init(self, *, vllm_config, prefix: str = ""):
+        config = vllm_config.model_config.hf_config
+        if int(getattr(config, "vision_n_layers", 0) or 0) > 0:
+            # b12x fused MHC Gram kernel requires rms_eps == 1e-6 (hard-coded
+            # in b12x.norm.mhc._impl._supports_fused_mhc_gram). The Vision-Exp
+            # checkpoint ships rms_norm_eps=1e-20, which fails that guard with
+            # "b12x_mhc_pre is served only by the fused Gram kernel". Normalize
+            # to 1e-6: RMSNorm eps is a div-by-zero stabilizer, both are far
+            # below the activation variance scale (negligible for bf16), and
+            # it matches the b12x decode config the kernel was built for.
+            # Guarded on vision_n_layers so text-only 0731 is unchanged.
+            config.rms_norm_eps = 1.0e-6
+        orig_model_init(self, vllm_config=vllm_config, prefix=prefix)
+        _install_vision_tower(self, vllm_config.model_config.hf_config)
+
+    DeepseekV4Model.__init__ = model_init
+    DeepseekV4Model.encode_image = lambda self, patches, n_h, n_w: encode_image(
+        self, patches, n_h, n_w
+    )
+
+    orig_model_load = DeepseekV4Model.load_weights
+
+    def model_load_weights(self, weights):
+        try:
+            from vllm.model_executor.model_loader.weight_utils import (
+                default_weight_loader,
+            )
+        except ImportError:
+            default_weight_loader = None
+        params_dict = dict(self.named_parameters())
+        loaded: set[str] = set()
+
+        def _filtered():
+            for name, weight in weights:
+                if is_unregistered_router_bias(name, params_dict):
+                    continue
+                if not is_vision_exp_weight_name(name):
+                    yield name, weight
+                    continue
+                if name not in params_dict:
+                    raise KeyError(
+                        f"Vision-Exp weight {name!r} has no module parameter; "
+                        "the ViT/Aligner tower was not constructed"
+                    )
+                param = params_dict[name]
+                loader = getattr(param, "weight_loader", None) or default_weight_loader
+                if loader is not None:
+                    loader(param, weight)
+                else:
+                    param.data.copy_(weight)
+                loaded.add(name)
+
+        loaded |= orig_model_load(self, _filtered())
+        return loaded
+
+    DeepseekV4Model.load_weights = model_load_weights
+
+    orig_moe_init = DeepseekV4MoE.__init__
+
+    def moe_init(self, vllm_config, prefix: str = ""):
+        orig_moe_init(self, vllm_config, prefix)
+        config = vllm_config.model_config.hf_config
+        if getattr(config, "vision_n_layers", 0) > 0:
+            self.gate.e_score_correction_bias_vl = nn.Parameter(
+                torch.empty(config.n_routed_experts, dtype=torch.float32),
+                requires_grad=False,
+            )
+
+    DeepseekV4MoE.__init__ = moe_init
+
+    orig_lm_init = DeepseekV4ForCausalLM.__init__
+
+    def lm_init(self, *, vllm_config, prefix: str = ""):
+        orig_lm_init(self, vllm_config=vllm_config, prefix=prefix)
+        self.hf_to_vllm_mapper = _extend_weights_mapper(self.hf_to_vllm_mapper)
+
+    DeepseekV4ForCausalLM.__init__ = lm_init
+    DeepseekV4ForCausalLM.hf_to_vllm_mapper = _extend_weights_mapper(
+        DeepseekV4ForCausalLM.hf_to_vllm_mapper
+    )
+
+    orig_lm_embed = DeepseekV4ForCausalLM.embed_input_ids
+
+    def embed_input_ids(
+        self,
+        input_ids: torch.Tensor,
+        multimodal_embeddings: Any = None,
+        *,
+        is_multimodal: Any = None,
+    ):
+        text_embeds = orig_lm_embed(self, input_ids)
+        if multimodal_embeddings is None:
+            return text_embeds
+        try:
+            empty = len(multimodal_embeddings) == 0
+        except TypeError:
+            empty = False
+        if empty:
+            return text_embeds
+        from vllm.model_executor.models.interfaces import _require_is_multimodal
+        from vllm.model_executor.models.utils import _merge_multimodal_embeddings
+
+        return _merge_multimodal_embeddings(
+            inputs_embeds=text_embeds,
+            multimodal_embeddings=multimodal_embeddings,
+            is_multimodal=_require_is_multimodal(is_multimodal),
+        )
+
+    DeepseekV4ForCausalLM.embed_input_ids = embed_input_ids
+
+    if SupportsMultiModal not in DeepseekV4ForCausalLM.__mro__:
+        DeepseekV4ForCausalLM.__bases__ = (
+            DeepseekV4ForCausalLM.__bases__[0],
+            SupportsMultiModal,
+            *DeepseekV4ForCausalLM.__bases__[1:],
+        )
+    DeepseekV4ForCausalLM.supports_multimodal = True
+    DeepseekV4ForCausalLM.supports_multimodal_raw_input_only = False
+    # Hash MoE (layers 0–2) looks up tid2eid with input_ids. The MM runner
+    # otherwise sets input_ids=None whenever inputs_embeds is present.
+    DeepseekV4ForCausalLM.requires_raw_input_tokens = True
+    DeepseekV4ForCausalLM.get_placeholder_str = get_placeholder_str
+    DeepseekV4ForCausalLM.embed_multimodal = embed_multimodal
+
+    # SupportsMultiModal.get_language_model() returns the first child with
+    # embed_input_ids (DeepseekV4Model). DSpark/Eagle3 then read `.model` on
+    # that child. This class *is* the LM wrapper, same as 0731.
+    def get_language_model(self):
+        return self
+
+    def set_aux_hidden_state_layers(self, layers):
+        self.model._set_aux_hidden_state_layers(layers)
+
+    def get_eagle3_default_aux_hidden_state_layers(self):
+        num_layers = len(self.model.layers)
+        return (2, num_layers // 2, num_layers - 3)
+
+    DeepseekV4ForCausalLM.get_language_model = get_language_model
+    DeepseekV4ForCausalLM.set_aux_hidden_state_layers = set_aux_hidden_state_layers
+    DeepseekV4ForCausalLM.get_eagle3_default_aux_hidden_state_layers = (
+        get_eagle3_default_aux_hidden_state_layers
+    )
+    register_vision_exp_processor(DeepseekV4ForCausalLM)

recipes/dsv4-vision-exp-vavrusa/mods/deepseek-v4-vision-exp/vision_exp/image_processor.py

@@ -0,0 +1,333 @@
+"""Official DeepSeek-V4-Flash-Vision-Exp image preprocessor.
+
+Adapted from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp ``inference/image_processor.py``
+(MIT). Adds a PIL entry point so vLLM can feed decoded images without going
+back through URLs. Video is not part of this checkpoint: GIF is decoded as a
+still RGB frame (PIL's first frame).
+"""
+from __future__ import annotations
+
+import base64
+import io
+import math
+from dataclasses import dataclass
+from types import SimpleNamespace
+from typing import Any, TYPE_CHECKING
+from urllib.request import urlopen
+
+if TYPE_CHECKING:
+    import torch
+    from PIL import Image as PILImage
+
+
+def _pil():
+    """Load Pillow on first image decode. CPU CI does not install it."""
+    try:
+        from PIL import Image, ImageOps
+    except ImportError as exc:
+        raise ModuleNotFoundError(
+            "Pillow is required to decode Vision-Exp images"
+        ) from exc
+    return Image, ImageOps
+
+
+IMAGE_START, IMAGE_PAD, IMAGE, IMAGE_NEW_LINE, IMAGE_END = range(5)
+COMPRESS_PAD_TO = 4
+IMAGE_PLACEHOLDER = "<｜deepseek_image｜>"
+
+
+def is_vision_exp_weight_name(name: str) -> bool:
+    """Checkpoint keys for the ViT/Aligner/image tokens.
+
+    Anemll ``load_weights`` treats any name containing ``w1`` as a stacked
+    ``gate_up_proj`` shard. Aligner's ``w1`` and ViT MLP ``w1`` are full
+    Linear layers, so those keys must bypass the stacked mapping.
+    """
+    return (
+        name.startswith("aligner.")
+        or name.startswith("vision.")
+        or name.startswith("image_")
+        or name.startswith("model.aligner.")
+        or name.startswith("model.vision.")
+        or name.startswith("model.image_")
+    )
+
+
+def is_unregistered_router_bias(name: str, param_names: Any) -> bool:
+    """True when a gate bias has no Parameter.
+
+    Hash MoE (layers 0–2) does not register ``e_score_correction_bias``.
+    0731 omitted those tensors; Vision-Exp dumps ``ffn.gate.bias`` /
+    ``bias_vl`` there anyway. Anemll's mapper rewrites them to
+    ``e_score_correction_bias`` / ``_vl``, then ``load_weights`` KeyErrors.
+    The DSpark draft loader uses ``endswith(".ffn.gate.bias")``, which
+    misses ``bias_vl``, so unmapped ``.ffn.gate.bias_vl`` is also unused
+    until that remap is patched.
+    """
+    if name in param_names:
+        return False
+    return name.endswith(
+        (
+            ".ffn.gate.e_score_correction_bias",
+            ".ffn.gate.e_score_correction_bias_vl",
+            ".ffn.gate.bias_vl",
+            ".ffn.gate.bias",
+        )
+    )
+
+
+def as_pil(item: Any) -> PILImage.Image:
+    """Normalize vLLM image items (PIL, HWC/CHW array, tensor, dict) to RGB PIL."""
+    Image, _ImageOps = _pil()
+    if isinstance(item, Image.Image):
+        return item.convert("RGB")
+    if isinstance(item, dict):
+        for key in ("image_pil", "pil", "image"):
+            value = item.get(key)
+            if value is None or isinstance(value, dict):
+                continue
+            return as_pil(value)
+        raise TypeError(f"Unsupported image dict keys: {sorted(item)!r}")
+    array = item
+    try:
+        import torch
+    except ImportError:
+        torch = None  # type: ignore[assignment]
+    if torch is not None and isinstance(array, torch.Tensor):
+        array = array.detach().cpu().numpy()
+    try:
+        import numpy as np
+    except ImportError as exc:
+        raise TypeError(f"Unsupported image item type: {type(item)!r}") from exc
+    if not hasattr(array, "ndim"):
+        raise TypeError(f"Unsupported image item type: {type(item)!r}")
+    arr = np.asarray(array)
+    if arr.ndim == 2:
+        arr = np.stack([arr, arr, arr], axis=-1)
+    elif arr.ndim == 3 and arr.shape[-1] not in (1, 3, 4) and arr.shape[0] in (1, 3, 4):
+        arr = np.transpose(arr, (1, 2, 0))
+    if arr.ndim != 3:
+        raise TypeError(f"Unsupported image array shape: {arr.shape!r}")
+    if arr.shape[-1] == 1:
+        arr = np.repeat(arr, 3, axis=-1)
+    if np.issubdtype(arr.dtype, np.floating):
+        peak = float(arr.max()) if arr.size else 0.0
+        if peak <= 1.0:
+            arr = arr * 255.0
+        arr = np.clip(arr, 0, 255).astype("uint8")
+    else:
+        arr = np.clip(arr, 0, 255).astype("uint8")
+    if arr.shape[-1] == 4:
+        return Image.fromarray(arr, mode="RGBA").convert("RGB")
+    return Image.fromarray(arr[..., :3], mode="RGB")
+
+
+@dataclass
+class ImageInput:
+    start: int
+    patches: Any
+    n_vit_h: int
+    n_vit_w: int
+    types: Any
+    perm: Any
+
+
+def vision_args_from_config(config: Any) -> SimpleNamespace:
+    return SimpleNamespace(
+        vision_patch_size=int(getattr(config, "vision_patch_size", 14)),
+        vision_downsample_ratio=int(getattr(config, "vision_downsample_ratio", 3)),
+        vision_max_n_token=int(getattr(config, "vision_max_n_token", 384)),
+        vision_min_pixels=int(getattr(config, "vision_min_pixels", 147456)),
+        vision_max_wh_ratio=getattr(config, "vision_max_wh_ratio", 8),
+        vision_n_layers=int(getattr(config, "vision_n_layers", 0)),
+        vision_dim=int(getattr(config, "vision_dim", 1024)),
+        vision_n_heads=int(getattr(config, "vision_n_heads", 16)),
+        vision_inter_dim=int(getattr(config, "vision_inter_dim", 2816)),
+        vision_rope_theta=float(getattr(config, "vision_rope_theta", 10000.0)),
+        vocab_size=int(getattr(config, "vocab_size", 129280)),
+        dim=int(getattr(config, "hidden_size", getattr(config, "dim", 4096))),
+        hidden_size=int(getattr(config, "hidden_size", getattr(config, "dim", 4096))),
+    )
+
+
+def grid_tokens(best_height, best_width, patch_size, downsample_ratio):
+    """Number of LLM tokens the aligner grid occupies (N-layout, incl. row/align padding)."""
+    n_llm_h = math.ceil((best_height // patch_size) / downsample_ratio)
+    n_llm_w = math.ceil((best_width // patch_size) / downsample_ratio)
+    num_tokens = n_llm_h * (n_llm_w + 1) + 2
+    if n_llm_h % 2 == 1:
+        num_tokens += n_llm_w + 1
+    num_tokens += (n_llm_h + 1) // 2 * (n_llm_w + 1) % 2 * 2
+    return n_llm_h, n_llm_w, num_tokens
+
+
+def solve_resize_ratio(height, width, patch_size, downsample_ratio, max_n_token):
+    r = height / width
+    max_w_float = math.sqrt((max_n_token - 2) / r + 0.25) - 0.5
+    max_h_float = max_w_float * r
+    if max_w_float < 1.0:
+        max_w = 1
+        max_h = (max_n_token - 2) // (max_w + 1)
+        if max_h % 2 == 1:
+            max_h -= 1
+        best_width = max_w * patch_size * downsample_ratio
+        best_height = max_h * patch_size * downsample_ratio
+    elif max_h_float < 2.0:
+        max_h = 2
+        max_w = ((max_n_token - 2) // max_h) - 1
+        assert max_w > 1
+        best_width = max_w * patch_size * downsample_ratio
+        best_height = max_h * patch_size * downsample_ratio
+    else:
+        max_w = math.floor(max_w_float)
+        max_h = math.floor(max_h_float)
+        if max_h % 2 == 1:
+            max_h -= 1
+        beta = min(
+            max_w * patch_size * downsample_ratio / width,
+            max_h * patch_size * downsample_ratio / height,
+        )
+        best_width = math.floor(width * beta / patch_size) * patch_size
+        best_height = math.floor(height * beta / patch_size) * patch_size
+    n_llm_h, n_llm_w, num_tokens = grid_tokens(
+        best_height, best_width, patch_size, downsample_ratio
+    )
+    return n_llm_h, n_llm_w, best_height, best_width, num_tokens
+
+
+def safe_resize(height, width, best_height, best_width, patch_size, downsample_ratio, max_n_token):
+    max_n_token -= COMPRESS_PAD_TO - 1
+    n_llm_h, n_llm_w, num_tokens = grid_tokens(
+        best_height, best_width, patch_size, downsample_ratio
+    )
+    budget = max_n_token
+    while num_tokens > max_n_token:
+        n_llm_h, n_llm_w, best_height, best_width, num_tokens = solve_resize_ratio(
+            height, width, patch_size, downsample_ratio, budget
+        )
+        budget -= 1
+    return n_llm_h, n_llm_w, best_height, best_width
+
+
+def load_image_bytes(record) -> bytes:
+    """Load image bytes from raw/base64 data, an Anthropic source, URL, or path."""
+    data = record.get("data")
+    if isinstance(data, bytes):
+        return data
+    if isinstance(data, str):
+        return base64.b64decode(data)
+
+    source = record.get("source")
+    if isinstance(source, dict):
+        if source.get("data") is not None:
+            return base64.b64decode(source["data"])
+        if source.get("url"):
+            return load_image_bytes({"url": source["url"]})
+
+    url = record.get("url")
+    if isinstance(url, str) and url:
+        if url.startswith("data:"):
+            header, _, payload = url.partition(",")
+            if ";base64" not in header:
+                raise ValueError(f"Unsupported data URL encoding: {header}")
+            return base64.b64decode(payload)
+        if url.startswith(("http://", "https://")):
+            with urlopen(url, timeout=30) as response:
+                return response.read()
+        with open(url, "rb") as file:
+            return file.read()
+
+    raise ValueError(f"Cannot load image from record: {list(record.keys())}")
+
+
+def pil_to_patches(image: PILImage.Image, args) -> tuple[Any, int, int, int, int]:
+    """Resize/pad one RGB image and return ViT patches plus LLM grid sizes."""
+    import torch
+
+    _Image, ImageOps = _pil()
+
+    p = args.vision_patch_size
+    image = image.convert("RGB")
+    width, height = image.size
+    if args.vision_max_wh_ratio is not None and width > height * args.vision_max_wh_ratio:
+        width = height * args.vision_max_wh_ratio
+    if 0 < width * height < args.vision_min_pixels:
+        ratio = (args.vision_min_pixels / (width * height)) ** 0.5
+        width = int(width * ratio)
+        height = int(height * ratio)
+    best_width = math.ceil(width / p) * p
+    best_height = math.ceil(height / p) * p
+    n_llm_h, n_llm_w, best_height, best_width = safe_resize(
+        height,
+        width,
+        best_height,
+        best_width,
+        p,
+        args.vision_downsample_ratio,
+        args.vision_max_n_token,
+    )
+    n_vit_h, n_vit_w = best_height // p, best_width // p
+    src_w, src_h = image.size
+    if args.vision_max_wh_ratio is not None and src_w >= args.vision_max_wh_ratio * src_h:
+        image = image.resize((best_width, best_height))
+    else:
+        image = ImageOps.pad(image, (best_width, best_height), color=(127, 127, 127))
+    try:
+        import numpy as np
+
+        x = torch.from_numpy(np.asarray(image, dtype=np.float32)).permute(2, 0, 1) / 255
+    except ImportError:
+        pixels = (
+            image.get_flattened_data()
+            if hasattr(image, "get_flattened_data")
+            else image.getdata()
+        )
+        x = torch.tensor(pixels, dtype=torch.float32)
+        x = x.view(image.height, image.width, 3).permute(2, 0, 1) / 255
+    x = ((x - 0.5) / 0.5).to(torch.bfloat16)
+    patches = (
+        x.reshape(3, n_vit_h, p, n_vit_w, p)
+        .permute(1, 3, 0, 2, 4)
+        .reshape(n_vit_h * n_vit_w, 3, p, p)
+    )
+    return patches, n_vit_h, n_vit_w, n_llm_h, n_llm_w
+
+
+def load_image(record, args):
+    """Load and transform one image record into ViT patches."""
+    Image, _ImageOps = _pil()
+    with Image.open(io.BytesIO(load_image_bytes(record))) as source:
+        image = source.convert("RGB")
+        return pil_to_patches(image, args)
+
+
+def build_image_block(n_llm_h: int, n_llm_w: int, start_pos: int):
+    """Builds the N-layout token types (final order) and the aligner-row order for IMAGE slots."""
+    import torch
+
+    compress_pad = COMPRESS_PAD_TO - 1 - start_pos % COMPRESS_PAD_TO
+    pad_h = n_llm_h % 2
+    rows = n_llm_h + pad_h
+    row_len = n_llm_w + 1
+    pad_last = rows // 2 * row_len % 2 * 2
+    types = torch.tensor(
+        ([IMAGE] * n_llm_w + [IMAGE_NEW_LINE]) * n_llm_h + [IMAGE_PAD] * (row_len * pad_h),
+        dtype=torch.int64,
+    )
+    order = torch.arange(rows * row_len).view(rows // 2, 2, row_len).transpose(1, 2).reshape(-1)
+    image_idx = torch.full((rows * row_len,), -1, dtype=torch.int64)
+    image_idx.view(rows, row_len)[:n_llm_h, :n_llm_w] = torch.arange(n_llm_h * n_llm_w).view(
+        n_llm_h, n_llm_w
+    )
+    perm = image_idx[order]
+    perm = perm[perm >= 0]
+    types = torch.cat(
+        [
+            torch.full((compress_pad,), IMAGE_PAD, dtype=torch.int64),
+            torch.tensor([IMAGE_START]),
+            types[order],
+            torch.full((pad_last,), IMAGE_PAD, dtype=torch.int64),
+            torch.tensor([IMAGE_END]),
+        ]
+    )
+    return types, perm

recipes/dsv4-vision-exp-vavrusa/mods/deepseek-v4-vision-exp/vision_exp/processor.py

@@ -0,0 +1,268 @@
+"""vLLM 0.25 multimodal processor for DeepSeek-V4-Flash-Vision-Exp.
+
+Images only. The checkpoint has no video encoder; GIF is a still frame via PIL.
+"""
+from __future__ import annotations
+
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+import torch
+from transformers.feature_extraction_utils import BatchFeature
+
+from vllm.inputs import MultiModalDataDict
+from vllm.multimodal import MULTIMODAL_REGISTRY
+from vllm.multimodal.inputs import MultiModalFieldConfig
+from vllm.multimodal.processing import (
+    BaseDummyInputsBuilder,
+    BaseMultiModalProcessor,
+    BaseProcessingInfo,
+    ProcessorInputs,
+    PromptReplacement,
+    PromptUpdate,
+    TimingContext,
+)
+
+from .image_processor import (
+    IMAGE_PAD,
+    IMAGE_PLACEHOLDER,
+    as_pil,
+    build_image_block,
+    pil_to_patches,
+    vision_args_from_config,
+)
+
+
+def _image_token_id(tokenizer) -> int:
+    convert = getattr(tokenizer, "convert_tokens_to_ids", None)
+    if convert is not None:
+        token_id = convert(IMAGE_PLACEHOLDER)
+        if token_id is not None and token_id != getattr(tokenizer, "unk_token_id", None):
+            return int(token_id)
+    vocab = getattr(tokenizer, "get_vocab", lambda: {})()
+    if IMAGE_PLACEHOLDER in vocab:
+        return int(vocab[IMAGE_PLACEHOLDER])
+    # Vision-Exp added-token id (vocab_size 129280; placeholder is in the tail).
+    return 129264
+
+
+def _collect_images(mm_data: Mapping[str, object]) -> list[Any]:
+    for key in ("images", "image"):
+        value = mm_data.get(key)
+        if value is None:
+            continue
+        if isinstance(value, (list, tuple)):
+            return list(value)
+        return [value]
+    return []
+
+
+class DeepseekV4VisionExpProcessingInfo(BaseProcessingInfo):
+    def get_supported_mm_limits(self) -> Mapping[str, int | None]:
+        return {"image": 16}
+
+    def get_hf_processor(self, **kwargs: object):
+        raise RuntimeError(
+            "DeepSeek-V4-Flash-Vision-Exp has no Hugging Face processor; "
+            "the vLLM Vision-Exp processor handles images directly."
+        )
+
+    def get_max_image_tokens(self) -> int:
+        return int(getattr(self.get_hf_config(), "vision_max_n_token", 384))
+
+
+class DeepseekV4VisionExpDummyInputsBuilder(
+    BaseDummyInputsBuilder[DeepseekV4VisionExpProcessingInfo]
+):
+    def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
+        return IMAGE_PLACEHOLDER * mm_counts.get("image", 0)
+
+    def get_dummy_mm_data(
+        self,
+        seq_len: int,
+        mm_counts: Mapping[str, int],
+        mm_options: Mapping[str, Any],
+    ) -> MultiModalDataDict:
+        num_images = mm_counts.get("image", 0)
+        image_overrides = mm_options.get("image")
+        # Oversized square; official safe_resize caps at vision_max_n_token.
+        return {
+            "image": self._get_dummy_images(
+                width=2048,
+                height=2048,
+                num_images=num_images,
+                overrides=image_overrides,
+            )
+        }
+
+
+class DeepseekV4VisionExpMultiModalProcessor(
+    BaseMultiModalProcessor[DeepseekV4VisionExpProcessingInfo]
+):
+    def _hf_processor_applies_updates(
+        self,
+        prompt_text: str,
+        mm_items,
+        hf_processor_mm_kwargs: Mapping[str, object],
+        tokenization_kwargs: Mapping[str, object],
+    ) -> bool:
+        return False
+
+    def _cached_apply_hf_processor(
+        self,
+        inputs: ProcessorInputs,
+        timing_ctx: TimingContext,
+    ) -> tuple[list[int], Any, bool]:
+        # compress_pad depends on the expanded token position of each image,
+        # so a content-only processor cache would reuse the wrong layout.
+        if inputs.mm_data_items.get_count("image", strict=False) > 0:
+            return self._apply_hf_processor(inputs, timing_ctx)
+        return super()._cached_apply_hf_processor(inputs, timing_ctx)
+
+    def _call_hf_processor(
+        self,
+        prompt: str,
+        mm_data: Mapping[str, object],
+        mm_kwargs: Mapping[str, object],
+        tok_kwargs: Mapping[str, object],
+    ) -> BatchFeature:
+        tokenizer = self.info.get_tokenizer()
+        encode = getattr(tokenizer, "encode", None)
+        if encode is None:
+            raise RuntimeError("Vision-Exp processor requires tokenizer.encode")
+        prompt_ids = list(encode(prompt, add_special_tokens=False))
+        images = _collect_images(mm_data)
+        if not images:
+            return BatchFeature(
+                data={"input_ids": torch.tensor([prompt_ids], dtype=torch.long)}
+            )
+
+        args = vision_args_from_config(self.info.get_hf_config())
+        image_token_id = _image_token_id(tokenizer)
+        max_tokens = args.vision_max_n_token
+        pixel_rows: list[torch.Tensor] = []
+        n_vit_h_rows: list[int] = []
+        n_vit_w_rows: list[int] = []
+        types_rows: list[torch.Tensor] = []
+        perm_rows: list[torch.Tensor] = []
+        num_token_rows: list[int] = []
+        max_patches = 1
+
+        expanded_len = 0
+        image_iter = iter(images)
+        for tok in prompt_ids:
+            if tok != image_token_id:
+                expanded_len += 1
+                continue
+            pil = as_pil(next(image_iter))
+            patches, n_vit_h, n_vit_w, n_llm_h, n_llm_w = pil_to_patches(pil, args)
+            types, perm = build_image_block(n_llm_h, n_llm_w, expanded_len)
+            if types.numel() > max_tokens:
+                raise ValueError(
+                    f"Image token block length {types.numel()} exceeds "
+                    f"vision_max_n_token={max_tokens}"
+                )
+            pixel_rows.append(patches)
+            n_vit_h_rows.append(int(n_vit_h))
+            n_vit_w_rows.append(int(n_vit_w))
+            types_rows.append(types)
+            perm_rows.append(perm)
+            num_token_rows.append(int(types.numel()))
+            max_patches = max(max_patches, int(patches.shape[0]))
+            expanded_len += int(types.numel())
+
+        leftover = list(image_iter)
+        if leftover:
+            raise ValueError(
+                f"Found {len(images)} images but only "
+                f"{len(images) - len(leftover)} {IMAGE_PLACEHOLDER!r} placeholders"
+            )
+        if len(pixel_rows) != prompt_ids.count(image_token_id):
+            raise ValueError(
+                f"Found {prompt_ids.count(image_token_id)} image placeholders "
+                f"but processed {len(pixel_rows)} images"
+            )
+
+        padded_pixels = []
+        for patches in pixel_rows:
+            if patches.shape[0] < max_patches:
+                pad = patches.new_zeros(
+                    max_patches - patches.shape[0], *patches.shape[1:]
+                )
+                patches = torch.cat([patches, pad], dim=0)
+            padded_pixels.append(patches.to(torch.bfloat16))
+
+        def _pad_1d(rows: list[torch.Tensor], fill: int, width: int) -> torch.Tensor:
+            out = []
+            for row in rows:
+                if row.numel() < width:
+                    row = torch.cat(
+                        [
+                            row,
+                            torch.full((width - row.numel(),), fill, dtype=row.dtype),
+                        ]
+                    )
+                out.append(row[:width])
+            return torch.stack(out, dim=0)
+
+        return BatchFeature(
+            data={
+                "input_ids": torch.tensor([prompt_ids], dtype=torch.long),
+                "pixel_values": torch.stack(padded_pixels, dim=0),
+                "n_vit_h": torch.tensor(n_vit_h_rows, dtype=torch.int32),
+                "n_vit_w": torch.tensor(n_vit_w_rows, dtype=torch.int32),
+                "types": _pad_1d(types_rows, IMAGE_PAD, max_tokens),
+                "perm": _pad_1d(perm_rows, -1, max_tokens),
+                "num_tokens": torch.tensor(num_token_rows, dtype=torch.int32),
+            }
+        )
+
+    def _get_mm_fields_config(
+        self,
+        hf_inputs: BatchFeature,
+        hf_processor_mm_kwargs: Mapping[str, object],
+    ) -> Mapping[str, MultiModalFieldConfig]:
+        return dict(
+            pixel_values=MultiModalFieldConfig.batched("image"),
+            n_vit_h=MultiModalFieldConfig.batched("image"),
+            n_vit_w=MultiModalFieldConfig.batched("image"),
+            types=MultiModalFieldConfig.batched("image"),
+            perm=MultiModalFieldConfig.batched("image"),
+            num_tokens=MultiModalFieldConfig.batched("image"),
+        )
+
+    def _get_prompt_updates(
+        self,
+        mm_items,
+        hf_processor_mm_kwargs: Mapping[str, object],
+        out_mm_kwargs,
+    ) -> Sequence[PromptUpdate]:
+        image_token_id = _image_token_id(self.info.get_tokenizer())
+
+        def get_replacement(item_idx: int):
+            item = out_mm_kwargs["image"][item_idx]
+            raw = item["num_tokens"]
+            num_tokens = raw.data if hasattr(raw, "data") else raw
+            if hasattr(num_tokens, "reshape"):
+                num_tokens = int(num_tokens.reshape(-1)[0].item())
+            else:
+                num_tokens = int(num_tokens)
+            if num_tokens <= 0:
+                raise ValueError(f"Image {item_idx} produced 0 vision tokens")
+            return [image_token_id] * num_tokens
+
+        return [
+            PromptReplacement(
+                modality="image",
+                target=[image_token_id],
+                replacement=get_replacement,
+            )
+        ]
+
+
+def register_vision_exp_processor(model_cls):
+    return MULTIMODAL_REGISTRY.register_processor(
+        DeepseekV4VisionExpMultiModalProcessor,
+        info=DeepseekV4VisionExpProcessingInfo,
+        dummy_inputs=DeepseekV4VisionExpDummyInputsBuilder,
+    )(model_cls)

recipes/dsv4-vision-exp-vavrusa/mods/deepseek-v4-vision-exp/vision_exp/vision.py

@@ -0,0 +1,136 @@
+"""Official DeepSeek-V4-Flash-Vision-Exp ViT + Aligner (pure PyTorch).
+
+Adapted from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp ``inference/vision.py``
+(MIT) so it can load the checkpoint tensors on Anemll vLLM without the
+official custom kernels. RoPE tables are moved onto the activation device.
+"""
+from __future__ import annotations
+
+from functools import lru_cache
+from typing import Any
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+
+@lru_cache(8)
+def get_vision_cos_sin(n_h: int, n_w: int, dim: int, theta: float):
+    inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
+    hpos = torch.arange(n_h).unsqueeze(1).expand(n_h, n_w)
+    wpos = torch.arange(n_w).unsqueeze(0).expand(n_h, n_w)
+    freqs = torch.stack([hpos, wpos], dim=-1).reshape(-1, 2, 1).float() * inv_freq
+    freqs = freqs.flatten(1)
+    return freqs.cos().unsqueeze(1), freqs.sin().unsqueeze(1)
+
+
+def apply_rotary(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
+    dtype = x.dtype
+    x1, x2 = x.float().chunk(2, dim=-1)
+    return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1).to(dtype)
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-6):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim, dtype=torch.float32))
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        dtype = x.dtype
+        x = x.float()
+        x = x * torch.rsqrt(x.square().mean(-1, keepdim=True) + self.eps)
+        return (self.weight * x).to(dtype)
+
+
+class PatchEmbed(nn.Module):
+    def __init__(self, args: Any):
+        super().__init__()
+        self.proj = nn.Linear(3 * args.vision_patch_size**2, args.vision_dim)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return self.proj(x.flatten(1))
+
+
+class Attention(nn.Module):
+    def __init__(self, args: Any):
+        super().__init__()
+        self.n_heads = args.vision_n_heads
+        self.head_dim = args.vision_dim // args.vision_n_heads
+        self.wqkv = nn.Linear(args.vision_dim, 3 * args.vision_dim)
+        self.wo = nn.Linear(args.vision_dim, args.vision_dim)
+
+    def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
+        n = x.size(0)
+        q, k, v = (
+            t.view(n, self.n_heads, self.head_dim)
+            for t in self.wqkv(x).chunk(3, dim=-1)
+        )
+        q = apply_rotary(q, cos, sin)
+        k = apply_rotary(k, cos, sin)
+        o = F.scaled_dot_product_attention(
+            q.transpose(0, 1), k.transpose(0, 1), v.transpose(0, 1)
+        )
+        return self.wo(o.transpose(0, 1).reshape(n, -1))
+
+
+class MLP(nn.Module):
+    def __init__(self, args: Any):
+        super().__init__()
+        self.w1 = nn.Linear(args.vision_dim, 2 * args.vision_inter_dim, bias=False)
+        self.w2 = nn.Linear(args.vision_inter_dim, args.vision_dim, bias=False)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        gate, up = self.w1(x).chunk(2, dim=-1)
+        return self.w2(F.silu(gate) * up)
+
+
+class Block(nn.Module):
+    def __init__(self, args: Any):
+        super().__init__()
+        self.norm1 = RMSNorm(args.vision_dim)
+        self.attn = Attention(args)
+        self.norm2 = RMSNorm(args.vision_dim)
+        self.mlp = MLP(args)
+
+    def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
+        x = x + self.attn(self.norm1(x), cos, sin)
+        return x + self.mlp(self.norm2(x))
+
+
+class ViT(nn.Module):
+    """DeepSeek ViT: full bidirectional attention over one image with 2D RoPE."""
+
+    def __init__(self, args: Any):
+        super().__init__()
+        self.rope_dim = args.vision_dim // args.vision_n_heads // 2
+        self.rope_theta = args.vision_rope_theta
+        self.patch_embed = PatchEmbed(args)
+        self.blocks = nn.ModuleList([Block(args) for _ in range(args.vision_n_layers)])
+        self.norm = RMSNorm(args.vision_dim)
+
+    def forward(self, patches: torch.Tensor, n_h: int, n_w: int) -> torch.Tensor:
+        x = self.patch_embed(patches)
+        cos, sin = get_vision_cos_sin(n_h, n_w, self.rope_dim, self.rope_theta)
+        cos = cos.to(device=x.device, dtype=torch.float32)
+        sin = sin.to(device=x.device, dtype=torch.float32)
+        for block in self.blocks:
+            x = block(x, cos, sin)
+        return self.norm(x)
+
+
+class Aligner(nn.Module):
+    def __init__(self, args: Any):
+        super().__init__()
+        self.downsample_ratio = args.vision_downsample_ratio
+        in_dim = args.vision_dim * self.downsample_ratio**2
+        hidden = getattr(args, "dim", None) or args.hidden_size
+        self.w1 = nn.Linear(in_dim, hidden)
+        self.w2 = nn.Linear(hidden, hidden)
+
+    def forward(self, x: torch.Tensor, n_h: int, n_w: int) -> torch.Tensor:
+        r = self.downsample_ratio
+        x = x.view(n_h, n_w, -1).permute(2, 0, 1)
+        x = F.pad(x, (0, -n_w % r, 0, -n_h % r))
+        x = F.unfold(x.unsqueeze(0), r, stride=r).squeeze(0).transpose(0, 1)
+        return self.w2(F.gelu(self.w1(x)))

recipes/dsv4-vision-exp-vavrusa/mods/instanttensor-hybrid-draft-loader/README.md

@@ -0,0 +1,67 @@
+# InstantTensor hybrid speculative-draft loader
+
+This opt-in mod keeps InstantTensor for the primary model while allowing a
+speculative draft model to use lazy safetensors. It avoids a second
+InstantTensor GPU-streaming pass over a large checkpoint when an embedded MTP
+or other same-checkpoint draft needs only a small subset of its tensors.
+
+The mod patches vLLM's central `get_model()` entry point. It identifies a real
+speculative draft by comparing the `ModelConfig` object with
+`speculative_config.draft_model_config`; target and unrelated model loads are
+left unchanged.
+
+## Modes
+
+Set `INSTANTTENSOR_DRAFT_LOADER` on the containers:
+
+- `auto` (default): use lazy safetensors only when the draft and target have
+  the same model path and revision.
+- `safetensors`: use lazy safetensors for every speculative draft whose
+  effective loader would otherwise be InstantTensor.
+- `instanttensor`: preserve InstantTensor for drafts. This disables the hybrid
+  behavior without removing the mod.
+
+If the primary/effective loader is not InstantTensor, the mod does nothing.
+An explicitly configured non-InstantTensor draft loader is also preserved.
+
+## Usage
+
+The default `auto` mode is appropriate for embedded MTP weights:
+
+```bash
+./launch-cluster.sh \
+  --apply-mod mods/instanttensor-hybrid-draft-loader \
+  exec \
+  vllm serve /model \
+    --load-format instanttensor \
+    --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
+    ...
+```
+
+To force lazy safetensors for a standalone speculative draft too:
+
+```bash
+./launch-cluster.sh \
+  --apply-mod mods/instanttensor-hybrid-draft-loader \
+  -e INSTANTTENSOR_DRAFT_LOADER=safetensors \
+  exec \
+  vllm serve ...
+```
+
+At runtime, a switched draft logs:
+
+```text
+Hybrid draft loading: using lazy safetensors for speculative draft weights
+while preserving InstantTensor for the target model
+```
+
+## Scope and limitations
+
+This is a vLLM integration workaround, not a selective-loading implementation
+inside InstantTensor. Lazy safetensors still scans checkpoint metadata, but
+weights rejected by the draft model remain CPU memory-mapped views instead of
+being staged and cloned on the GPU.
+
+Remove the mod once vLLM exposes a supported per-draft load-format option or
+InstantTensor and vLLM can plan and selectively stream only requested tensor
+names.

recipes/dsv4-vision-exp-vavrusa/mods/instanttensor-hybrid-draft-loader/patch_model_loader.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""Add an opt-in hybrid loader policy for speculative draft models."""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import sys
+from pathlib import Path
+
+MARKER = "# spark-vllm mod: instanttensor-hybrid-draft-loader v1"
+
+GET_MODEL_ANCHOR = """
+def get_model(
+    *,
+    vllm_config: VllmConfig,
+    model_config: ModelConfig | None = None,
+    prefix: str = "",
+    load_config: LoadConfig | None = None,
+) -> nn.Module:
+    loader = get_model_loader(load_config or vllm_config.load_config)
+    if model_config is None:
+        model_config = vllm_config.model_config
+    return loader.load_model(
+        vllm_config=vllm_config, model_config=model_config, prefix=prefix
+    )
+"""
+
+GET_MODEL_REPLACEMENT = f"""
+{MARKER}
+def _instanttensor_draft_load_config(
+    vllm_config: VllmConfig,
+    model_config: ModelConfig,
+    load_config: LoadConfig | None,
+) -> LoadConfig:
+    \"\"\"Resolve the loader for one model without changing the target loader.\"\"\"
+    import os
+
+    from vllm.config import replace
+
+    mode = os.environ.get("INSTANTTENSOR_DRAFT_LOADER", "auto").strip().lower()
+    allowed_modes = ("auto", "safetensors", "instanttensor")
+    if mode not in allowed_modes:
+        raise ValueError(
+            "INSTANTTENSOR_DRAFT_LOADER must be one of "
+            f"{{', '.join(allowed_modes)}}; got {{mode!r}}"
+        )
+
+    effective = load_config or vllm_config.load_config
+    load_format = getattr(effective.load_format, "value", effective.load_format)
+    if mode == "instanttensor" or str(load_format).lower() != "instanttensor":
+        return effective
+
+    speculative_config = getattr(vllm_config, "speculative_config", None)
+    draft_model_config = getattr(speculative_config, "draft_model_config", None)
+    if draft_model_config is None or model_config is not draft_model_config:
+        return effective
+
+    if mode == "auto":
+        target_model_config = (
+            getattr(speculative_config, "target_model_config", None)
+            or vllm_config.model_config
+        )
+        draft_source = (
+            getattr(draft_model_config, "model", None),
+            getattr(draft_model_config, "revision", None),
+        )
+        target_source = (
+            getattr(target_model_config, "model", None),
+            getattr(target_model_config, "revision", None),
+        )
+        if draft_source != target_source:
+            return effective
+
+    logger.info_once(
+        "Hybrid draft loading: using lazy safetensors for speculative draft "
+        "weights while preserving InstantTensor for the target model "
+        "(INSTANTTENSOR_DRAFT_LOADER=%s).",
+        mode,
+    )
+    return replace(
+        effective,
+        load_format="safetensors",
+        safetensors_load_strategy="lazy",
+    )
+
+
+def get_model(
+    *,
+    vllm_config: VllmConfig,
+    model_config: ModelConfig | None = None,
+    prefix: str = "",
+    load_config: LoadConfig | None = None,
+) -> nn.Module:
+    if model_config is None:
+        model_config = vllm_config.model_config
+    resolved_load_config = _instanttensor_draft_load_config(
+        vllm_config, model_config, load_config
+    )
+    loader = get_model_loader(resolved_load_config)
+    return loader.load_model(
+        vllm_config=vllm_config, model_config=model_config, prefix=prefix
+    )
+"""
+
+
+def validate_shape(text: str, *, patched: bool) -> None:
+    tree = ast.parse(text)
+    functions = {
+        node.name
+        for node in tree.body
+        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+    }
+    required = {"get_model", "get_model_loader"}
+    if patched:
+        required.add("_instanttensor_draft_load_config")
+    missing = sorted(required - functions)
+    if missing:
+        raise ValueError(
+            "unexpected model-loader module; missing " + ", ".join(missing)
+        )
+
+
+def patched_text(text: str) -> str:
+    validate_shape(text, patched=MARKER in text)
+    if MARKER in text:
+        if text.count(MARKER) != 1:
+            raise ValueError("hybrid-loader marker occurs more than once")
+        if "resolved_load_config = _instanttensor_draft_load_config(" not in text:
+            raise ValueError("mod marker exists but get_model does not use the helper")
+        compile(text, "<patched model_loader/__init__.py>", "exec")
+        return text
+
+    count = text.count(GET_MODEL_ANCHOR)
+    if count != 1:
+        raise ValueError(
+            f"expected exactly one supported get_model function; found {count}"
+        )
+    patched = text.replace(GET_MODEL_ANCHOR, GET_MODEL_REPLACEMENT, 1)
+    validate_shape(patched, patched=True)
+    compile(patched, "<patched model_loader/__init__.py>", "exec")
+    return patched
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("target", type=Path)
+    parser.add_argument(
+        "--check", action="store_true", help="validate compatibility without writing"
+    )
+    args = parser.parse_args()
+
+    if not args.target.is_file():
+        print(
+            "[instanttensor-hybrid-draft-loader ERROR] "
+            f"target not found: {args.target}",
+            file=sys.stderr,
+        )
+        return 1
+
+    original = args.target.read_text()
+    try:
+        patched = patched_text(original)
+    except (SyntaxError, ValueError) as exc:
+        print(
+            "[instanttensor-hybrid-draft-loader ERROR] "
+            f"refusing to patch {args.target}: {exc}",
+            file=sys.stderr,
+        )
+        return 1
+
+    if args.check:
+        state = "already patched" if patched == original else "compatible"
+        print(f"[instanttensor-hybrid-draft-loader] {args.target} is {state}.")
+        return 0
+
+    if patched == original:
+        print(
+            "[instanttensor-hybrid-draft-loader] "
+            "Model loader is already patched; skipping."
+        )
+        return 0
+
+    temporary = args.target.with_suffix(
+        args.target.suffix + ".instanttensor-draft-mod.tmp"
+    )
+    temporary.write_text(patched)
+    temporary.replace(args.target)
+    print(f"[instanttensor-hybrid-draft-loader] Patched {args.target}.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

recipes/dsv4-vision-exp-vavrusa/mods/instanttensor-hybrid-draft-loader/run.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+set -euo pipefail
+
+PREFIX="[instanttensor-hybrid-draft-loader]"
+MOD_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+DEFAULT_PYTHON_ROOT="/usr/local/lib/python3.12/dist-packages"
+PYTHON_ROOT="${VLLM_SITE_PACKAGES:-${PYTHON_ROOT:-$DEFAULT_PYTHON_ROOT}}"
+VLLM_ROOT="$PYTHON_ROOT/vllm"
+TARGET="$VLLM_ROOT/model_executor/model_loader/__init__.py"
+PATCHER="$MOD_DIR/patch_model_loader.py"
+MODE="${INSTANTTENSOR_DRAFT_LOADER:-auto}"
+MODE_NORMALIZED="$(printf '%s' "$MODE" | tr '[:upper:]' '[:lower:]')"
+
+echo "=== InstantTensor hybrid speculative-draft loader mod ==="
+
+case "$MODE_NORMALIZED" in
+    auto|safetensors|instanttensor) ;;
+    *)
+        echo "$PREFIX INSTANTTENSOR_DRAFT_LOADER must be auto, safetensors," \
+            "or instanttensor; got '$MODE'." >&2
+        exit 1
+        ;;
+esac
+
+if [[ ! -d "$VLLM_ROOT" ]]; then
+    echo "$PREFIX vLLM package not found at $VLLM_ROOT" >&2
+    exit 1
+fi
+
+if [[ ! -f "$PATCHER" ]]; then
+    echo "$PREFIX patcher not found at $PATCHER" >&2
+    exit 1
+fi
+
+if [[ ! -f "$TARGET" ]]; then
+    echo "$PREFIX vLLM model-loader module not found at $TARGET" >&2
+    exit 1
+fi
+
+python3 "$PATCHER" --check "$TARGET"
+python3 "$PATCHER" "$TARGET"
+python3 "$PATCHER" --check "$TARGET"
+
+find "$(dirname "$TARGET")" -name "__pycache__" -type d \
+    -exec rm -rf {} + 2>/dev/null || true
+
+echo "$PREFIX Enabled with INSTANTTENSOR_DRAFT_LOADER=$MODE_NORMALIZED."
+echo "=== OK: target loads stay on InstantTensor; selected drafts use lazy safetensors ==="

recipes/dsv4-vision-exp-vavrusa/recipe.toml

@@ -0,0 +1,128 @@
+# DeepSeek-V4-Flash-Vision-Exp — vavrusa sparkrun lane ported to the launcher.
+#
+# Working OpenAI image_url multimodal serving on DGX Spark from
+# vavrusa/sparkrun-recipes `deepseek-v4-flash-vision-exp-eugr.yaml`
+# (ghcr.io/spark-arena/dgx-vllm-eugr-nightly-b12x:20260810 + two runtime
+# mods). This is the eugr A/B lane (todo #26) AND the image_url lane for
+# the vision checkpoint — compare against recipes/dsv4-flash-vision
+# (our plugin on dsv4-flash-vision:local).
+#
+# Mod provenance (vendored verbatim under mods/):
+#   * instanttensor-hybrid-draft-loader — keeps InstantTensor for the target
+#     while embedded-MTP drafts load via lazy safetensors.
+#   * deepseek-v4-vision-exp — stages vision_exp + injects a fail-closed
+#     import hook (port of MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark)
+#     into the Anemll fork's deepseek_v4/nvidia/model.py: ViT + aligner,
+#     weight mapping for vision./aligner./image_*/bias_vl, vLLM multimodal
+#     processor registration, embed_input_ids embedding scatter, DSpark
+#     draft ffn.gate.bias_vl remap.
+#
+# Checkpoint deltas vs text-only 0731 (from their README): vision keys;
+# num_nextn_predict_layers 1→3 (DSpark k must be a multiple of 3 → k=6);
+# rms_norm_eps 1e-6→1e-20 (mod overrides back to 1e-6 for the fused Gram
+# mHC kernel); ffn.gate +bias_vl (loaded, not applied at routing).
+# Images go in USER messages only; ≤384 image tokens per image.
+#
+# DEVIATIONS from the sparkrun recipe — all forced by our launcher:
+#   * no --host/--port (launcher fabric), TP size from recipe math,
+#   * --served-model-name pinned to our quant-agnostic lane alias
+#     `deepseek-v4-flash-vision` (HF_HUB_OFFLINE makes a bare label carry
+#     the cache path otherwise),
+#   * default-chat-template-kwargs thinking=false / effort=low to match the
+#     tool-eval c1 lane (their recipe ships thinking=true / high),
+#   * weights from per-node local NVMe copies (`~/dsv4vision-hf` on
+#     dgx3/dgx4, 167.8 GiB each). Fallback: swap the extra_mounts block to
+#     /mnt/nas/ai/huggingface (see recipes/dsv4-flash-vision).
+
+engine = "vllm"
+# eugr nightly image carries no ENTRYPOINT — required by the mods contract.
+entrypoint = ["vllm", "serve"]
+name = "DeepSeek V4 Flash Vision-Exp (eugr nightly + vavrusa vision mods)"
+docker_image = "ghcr.io/spark-arena/dgx-vllm-eugr-nightly-b12x:20260810"
+model = "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"
+required_nodes = 2
+tensor_parallel_size = 2
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+
+args = [
+    "--served-model-name", "deepseek-v4-flash-vision",
+    "--trust-remote-code",
+    "--tensor-parallel-size", "2",
+    "--kv-cache-dtype", "fp8_ds_mla",
+    # their picks: max-model-len auto (KV pool revalidated after first
+    # launch — the vision tower replicates per rank), 5 seqs / 4096 tokens.
+    "--max-model-len", "auto",
+    "--max-num-seqs", "5",
+    "--max-num-batched-tokens", "4096",
+    "--gpu-memory-utilization", "0.82",
+    "--block-size", "256",
+    "--enable-prefix-caching",
+    "--enable-chunked-prefill",
+    "--enable-prompt-tokens-details",
+    "--enable-flashinfer-autotune",
+    "--async-scheduling",
+    "--tokenizer-mode", "deepseek_v4",
+    "--tool-call-parser", "deepseek_v4",
+    "--enable-auto-tool-choice",
+    "--reasoning-parser", "deepseek_v4",
+    "--reasoning-config", """
+'{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}'""",
+    "--default-chat-template-kwargs", """
+'{"thinking":false,"reasoning_effort":"low"}'""",
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95}'""",
+    "--limit-mm-per-prompt", """
+'{"image":8}'""",
+    "--load-format", "instanttensor",
+    "--moe-backend", "b12x",
+    "--linear-backend", "b12x",
+    "--attention-backend", "B12X_MLA_SPARSE",
+    "--max-cudagraph-capture-size", "64",
+    "--compilation-config", """
+'{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'""",
+    # num_nextn_predict_layers=3 → k multiple of 3; k=6 (smallest ≥ dspark
+    # block size 5). Do NOT drop back to 5.
+    "--speculative-config", """
+'{"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"greedy","attention_backend":"B12X_MLA_SPARSE"}'""",
+]
+
+[env]
+PYTHONHASHSEED = "0"
+CUTE_DSL_ARCH = "sm_121a"
+VLLM_USE_AOT_COMPILE = "1"
+VLLM_USE_BREAKABLE_CUDAGRAPH = "0"
+VLLM_USE_MEGA_AOT_ARTIFACT = "-1"
+VLLM_MEMORY_PROFILE_INCLUDE_ATTN = "1"
+VLLM_USE_FLASHINFER_SAMPLER = "1"
+VLLM_USE_B12X_WO_PROJECTION = "1"
+VLLM_USE_B12X_MHC = "1"
+VLLM_USE_B12X_FP8_GEMM = "1"
+VLLM_USE_B12X_MOE = "1"
+VLLM_USE_B12X_SPARSE_INDEXER = "1"
+VLLM_MOE_SKIP_PADDING = "0"
+VLLM_USE_V2_MODEL_RUNNER = "1"
+B12X_MLA_SM120_UNIFIED = "1"
+B12X_MOE_FORCE_A8 = "1"
+VLLM_PREFIX_CACHE_RETENTION_INTERVAL = "4096"
+HF_HUB_OFFLINE = "1"
+TRANSFORMERS_OFFLINE = "1"
+VLLM_NO_USAGE_STATS = "1"
+
+# Weights from per-node local NVMe copies; JIT caches persist per node.
+# NOTE: extra_mounts stay AFTER args (TOML table headers would swallow the
+# args array otherwise).
+[[extra_mounts]]
+host = "~/dsv4vision-hf"
+container = "/opt/local-hf"
+env = "HF_HOME"
+
+# Both mods run inside the container before exec'ing vllm serve. The
+# vision mod stages to /opt/dspark-patches and rewrites site-packages —
+# the container fs stays writable (only /mods is :ro).
+[[mods.commands]]
+command = "bash /mods/instanttensor-hybrid-draft-loader/run.sh"
+
+[[mods.commands]]
+command = "bash /mods/deepseek-v4-vision-exp/run.sh"