--- bug_id: BUG-2026-08-03T204037 status: resolved severity: high scope: recipes title: "DeepSeek-V4-Flash recipe fails to update to aidendle94/sparkrun-vllm-ds4-gb10:production-3.8" --- # BUG-2026-08-03T204037: DeepSeek-V4-Flash recipe fails to update to production-3.8 ## Problem **Actual:** Updating the `deepseek-v4-flash` recipe + `deepseek-v4-flash-dspark` overlay from the working `production-3.75` image to the new `aidendle94/sparkrun-vllm-ds4-gb10:production-3.8` image fails to boot in three distinct, compounding ways: 1. **Does not run with the V2 model runner** (`VLLM_USE_V2_MODEL_RUNNER=1`, carried over from the 3.75 recipe). 2. **OOMs with the "tiny kernel"** — the opt-in DSPARK tiny-A8 decode kernel (`VLLM_B12X_TINY_DECODE=1`) exhausts GPU memory at load time. 3. **DeepGEMM fails to compile** when both v2 and tiny are deactivated, crashing with: ``` Assertion error (/tmp/dsv4-vllm-...-3dnaf_2y/sources/DeepGEMM/csrc/apis/../ jit_kernels/impls/../heuristics/../../utils/layout.hpp:39): t.dim() == N ``` **Expected:** The recipe should boot on `production-3.8` as it does on `production-3.75`, with DSpark speculative decoding (k=5) and the b12x MoE backend. **Context:** The cluster is serving live traffic at the time of writing, so no runtime/boot experiment could be performed. This investigation is based on non-invasive image inspection only (`docker image inspect`, `docker history`, `docker run` without `--gpus` running `pip show` / source `grep` — never importing vLLM or torch, never allocating GPU memory). All four physical DGX Spark boxes are busy serving `glm52-nf4-lloyd-experts-gptq` on the `4nodes-switch` cluster; the `2nodes-dac` cluster shares the same hardware (different data-plane IPs, same management IPs). **Failed attempts recovered from git:** - Commit `72f40b3` (reverted by `3a783ee`): FROM→3.8, patch path `/opt/venv`→`/opt/env`, dropped instanttensor, `--load-format auto`, removed `--attention-backend FLASHINFER_MLA_SPARSE_DSV4` (correct — does not exist in vLLM 0.21), set `VLLM_ATTENTION_BACKEND=B12X_MLA_SPARSE`, raised k 5→7, gpu_util 0.84→0.9, max-len 524288→393216, dotted kwargs→ single JSON arg, added `VLLM_B12X_TINY_DECODE=1/MAX_M=8/CHUNK=8`, dropped 5 stale env vars. - `stash@{0}`: same direction minus tiny-decode, k stays 5, gpu_util 0.84. Both kept `VLLM_USE_V2_MODEL_RUNNER=1` from the 3.75 recipe. ## Root Cause Analysis ### The two images are different forks, not version bumps `production-3.8` and the working `eugr/spark-vllm-b12x` are *fundamentally different b12x forks*, not a linear version bump from 3.75: | | **production-3.8** ("dsv4/apostolic" harness) | **eugr/spark-vllm-b12x** ("gilded-gnosis") | **production-3.75** (working) | |---|---|---|---| | vLLM | `0.21.1rc1.dev339+g1967a56` (2026-05-28) | `0.1.dev19023+g30038602b` (2026-08-02) | `0.11.2` eldritch | | torch | 2.11.0+cu130 | 2.12.0+cu130 | 2.11+cu130 | | flashinfer | 0.6.12 | 0.6.17 | — | | triton | 3.6.0 | 3.7.0 | — | | b12x | **standalone** `/opt/b12x` v0.15.3 | **integrated into vLLM** | standalone | | DeepGEMM | vendored src at `/opt/vllm-apostolic/.deps/deepgemm-src/` (commit `1f2f161`) | not visible | — | | base | micromamba/conda `/opt/env` | nvidia/cuda:13.0.2-devel-ubuntu24.04 | `/opt/venv` | | NCCL | 2.30.4 (conda) | 2.28.3 (apt) | 2.30.4 | | instanttensor | **absent** | 0.1.9 present | present | | launch | wrapper `/usr/local/bin/dsv4-vllm-entrypoint`; base `CMD sleep infinity` | `ENTRY /opt/nvidia/nvidia_entrypoint.sh` | direct `vllm` | **Provenance** (from `/opt/dsv4/provenance.json` in the 3.8 image): `production-3.8` is built by a "dsv4" harness (config sha `f9dcce6c...`, artifact `dsv4-vllm-sm120-20260528-arm64-nccl2304-linux-arm64-2a32b9cc4d9f`), base image `mambaorg/micromamba:2.3.3`, conda env `vllm` at `/opt/env`. The deepgemm source is a "sanitized DeepGEMM export via `DEEPGEMM_SRC_DIR` and `VLLM_DEEPGEMM_SRC`" (commit `1f2f161dba747b7c12671d017f7c88e1249c3d3e`). ### production-3.8 is a "clean-overlay" base meant to be launched via a wrapper The decisive structural difference: `production-3.8` is a clean base (`CMD ["sleep","infinity"]`, `ENTRY null`) designed to be launched via `/usr/local/bin/dsv4-vllm-entrypoint`, which: - prepends `/opt/env/bin` to `PATH`, - sets the JIT toolchain env (`DG_JIT_NVCC_COMPILER=/opt/env/bin/nvcc`, `FLASHINFER_NVCC`, `PYTORCH_NVCC`, `NVCC_PREPEND_FLAGS` with the CCCL include path, `CUDAHOSTCXX`), - sets the JIT cache dirs (`DG_JIT_CACHE_DIR=/cache/deep_gemm_jit`, `FLASHINFER_WORKSPACE_BASE`, `VLLM_CACHE_ROOT`, `TILELANG_CACHE_DIR`, `TRITON_CACHE_DIR`, `TORCHINDUCTOR_CACHE_DIR`, `TORCH_EXTENSIONS_DIR`), - `mkdir -p`s those dirs (they do not exist in the bare image), - then `exec vllm "$@"`. By keeping `entrypoint = ["vllm", "serve"]` (the 3.75 contract), the recipe **bypasses the wrapper**, so the image's intended launch contract is broken. nvcc discovery happens to still work (it is on PATH and `CUDA_HOME` is baked into the image env), but the cache-dir layout and the CCCL include paths are unset — making DeepGEMM JIT fragile. `production-3.75` worked because it was a self-contained serving image where vLLM was directly runnable with all env baked in. ### Symptom #3 — DeepGEMM `layout.hpp:39` `t.dim() == N` (the boot-blocker) The real failing file is `/opt/vllm-apostolic/.deps/deepgemm-src/csrc/utils/layout.hpp` (the stack path `apis/../jit_kernels/impls/../heuristics/../../utils/layout.hpp` resolves there). Line 39: ```cpp template static auto get_shape(const torch::Tensor& t) { DG_HOST_ASSERT(t.dim() == N); // ← line 39, N is a compile-time template arg ``` This is **not** a missing-nvcc / cache-dir problem. It is a **tensor-rank contract violation**: vLLM calls DeepGEMM's `transform_sf_into_required_layout` (bound in `csrc/apis/layout.hpp`, which calls `get_shape()`) passing a scale-factor tensor whose rank does not match the template specialization. The binding `check_sf_layout` asserts `sf.dim() == 2`; the grouped / mega-moe path expects rank-3 (`get_shape<3>`). A rank mismatch trips line 39. It fires during MoE weight-loading / scale-packing (`vllm/model_executor/layers/quantization/utils/fp8_utils.py:1048` and `.../fused_moe/oracle/mxfp4.py:758`), **before** any kernel runs. The recipe's `--moe-backend b12x` does *not* bypass it: `transform_sf_into_required_layout` is called during MoE weight prep regardless of the selected MoE compute backend. This is a **fork bug / version-skew** between this specific DeepGEMM fork (`1f2f161`) and what vLLM 0.21's `transform_sf_into_required_layout` callers feed it — not an env mistake, and the one that blocks boot entirely. ### Symptom #2 — "OOMs with tiny kernel" The opt-in tiny-decode kernel (`/opt/tinydec/tiny_hook.py`, `/opt/tinydec/tiny_moe.py`) is a runtime monkey-patch on `B12xExperts.apply`/`_get_or_prepare_fp4_moe_weights` that routes small-M MoE decode through a custom w4a8mx Triton kernel. Its own docstring states the OOM mechanism directly: > *"Chunking is what keeps the load-time repack off the OOM path that > crashed the node."* > *"the `.int()` expansion plus 9-dim reshape temporaries are what OOM'd > the head node when run unchunked at E=256."* The hook repacks b12x's `w4a16` packed tensors into tiny's `w4a8_mx` format **on GPU at load time**, per expert-tensor, caching the result. The failed attempt set `--gpu-memory-utilization 0.9` **together with** `VLLM_B12X_TINY_DECODE=1` — the repack needs GPU headroom that 0.9 does not leave. `VLLM_B12X_TINY_CHUNK=8` exists precisely to bound the peak, but does not help when the utilization ceiling leaves no room for the repack buffers. The image label confirms the gating contract: `org.tinydec.flag=VLLM_B12X_TINY_DECODE`, `org.tinydec.kernel=w4a8mx-tiny-decode`, `org.tinydec.gates="A:cos0.999999 b0:0.81x b:cos0.999990 M:99%"`. ### Symptom #1 — "does not run with v2" The V2 model runner is experimental in this fork. `gpu_worker.py:803` has a hard `assert not self.use_v2_model_runner` when **pipeline-parallel + sequence-parallel** is active (not the case here, TP-only), so the v2 failure here is elsewhere in the V2 warmup path (`warmup_kernels` runs a full `execute_model + sample_tokens` JIT compile). The known-working `glm52` recipe does **not** set `VLLM_USE_V2_MODEL_RUNNER`. Carrying `=1` over from the 3.75 recipe is a separate, independent source of fragility. ## Reproduce Could not be reproduced at boot time (cluster busy serving). The diagnosis is based on static inspection of both images on `dgx1.lan` (management IP `192.168.88.14`, the `2nodes-dac` cluster entry, same physical box as the serving `4nodes-switch` rank 0): ``` # image versions (metadata only, no CUDA import, no GPU allocation): docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \ -lc '/opt/env/bin/pip3 show vllm torch flashinfer-python triton' # the wrapper the recipe must use: docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \ -lc 'cat /usr/local/bin/dsv4-vllm-entrypoint' # the failing assertion source: docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \ -lc 'sed -n "30,45p" /opt/vllm-apostolic/.deps/deepgemm-src/csrc/utils/layout.hpp' ``` All inspection used `docker run` **without `--gpus`** (no GPU device passthrough → no competition with the live `glm52` serve), running only `pip show` / `find` / `grep` / `sed` — never importing vLLM or torch, never allocating GPU memory. The git-recovered failed attempts: ``` git show 72f40b3 # reverted 3.8 attempt + tiny-decode git stash show -p stash@{0} # 3.8 attempt, no tiny-decode ``` ## Fix direction Given all nodes are occupied (no runtime test possible), the safe path is: 1. **Use the `dsv4-vllm-entrypoint` wrapper** as the recipe entrypoint so the image's intended JIT env + cache dirs are active: ```toml entrypoint = ["/usr/local/bin/dsv4-vllm-entrypoint", "serve"] ``` 2. **Mirror / rely on the wrapper env** (`DG_JIT_NVCC_COMPILER`, `FLASHINFER_NVCC`, `DG_JIT_CACHE_DIR=/cache/deep_gemm_jit`, …) and add an `[[extra_mounts]]` for `/cache` so the JIT compile cache persists across restarts (the wrapper `mkdir -p`s these under `/cache`). 3. **Drop the V2 runner** (unset `VLLM_USE_V2_MODEL_RUNNER`) — experimental in this fork; the working `glm52` recipe does not use it. 4. **Drop tiny-decode** (unset `VLLM_B12X_TINY_DECODE`) for first bring-up — it is an opt-in *experimental* kernel with a load-time GPU repack that needs real headroom. Get a stable boot first, then re-enable with `--gpu-memory-utilization` ≤ 0.84 and `VLLM_B12X_TINY_CHUNK=8`. 5. **The DeepGEMM `t.dim() == N` crash needs a real fix, not env** — it fires during MoE scale packing on this fork. Options to investigate once a node frees up: - try `--moe-backend deep_gemm_mega_moe` or `--moe-backend auto` (instead of `b12x`) to see if a different backend avoids the rank-3 scale path; - check whether the model's quant config (NVFP4 vs MXFP4) selects a DeepGEMM template that mismatches — `mxfp4.py`'s oracle `_pack` calls `transform_sf_into_required_layout` with `recipe=(1,1,32)` and `num_groups=num_experts`, feeding a 3D tensor; - if it persists, this is a bug to report upstream against the `1f2f161` DeepGEMM fork — the assertion message is the fork's own contract. The `eugr/spark-vllm-b12x` image (recipe `deepseek-v4-flash-eugr`, commit `a8e4932`) sidesteps *all* of this because it is a different, newer fork (`gilded-gnosis`, 2026-08-02) where b12x is integrated and the DSV4 path is already wired — which is why that recipe boots cleanly. The two images are **not interchangeable** at the recipe level; a 3.8 recipe must respect the wrapper-entrypoint + clean-base contract. ## Acceptance Criteria (original) - [x] `deepseek-v4-flash-dspark38` boots on `production-3.8` via the `dsv4-vllm-entrypoint` wrapper. - [x] No `VLLM_USE_V2_MODEL_RUNNER`, no `VLLM_B12X_TINY_DECODE`. - [x] `/cache` JIT dirs persist via an `[[extra_mounts]]`. - [x] The DeepGEMM `t.dim() == N` crash is resolved (drafter wo_a scale reconstruction; see Resolution). - [ ] DSpark k=5 decode acceptance + throughput verified against the 3.75 baseline (boot + generation smoke-test pass; perf benchmark pending node availability). ## Resolution **Closed (fixed).** Boot-tested on the `ds4` cluster (dgx3+dgx4, idle). The `t.dim() == N` crash was **not** the MoE scale-packing path the original write-up hypothesized — the DSV4 target + drafter load cleanly via the b12x MoE backend (DeepGEMM MoE packing never fires). Two real, distinct fork bugs blocked boot, both fixed in the `deepseek-v4-flash-dspark38` recipe + local overlay image (`docker-images/deepseek-v4-flash-dspark38/`, build target `sparkrun-vllm-ds4-gb10:dspark38`): ### Bug A — b12x cutlass from_dlpack rejects grad-requiring tensors During KV-cache profiling the b12x attention `norm_weight` carries `requires_grad=True` under `torch.compile`; `b12x/integration/residual_kernels.py _to_kernel_tensor` -> `from_dlpack(tensor)` throws `BufferError: Can't export tensors that require gradient, use tensor.detach()`. Fixed by `.detach()` before export (no-op when requires_grad=False). ### Bug B — drafter wo_a feeds DeepGEMM fp8_einsum an incompatible scale The DSpark drafter `forward_dspark` calls `deepseek_v4_fp8_einsum("bhr,hdr->bhd", wo_a.weight, wo_a.weight_scale_inv, …)` with recipe `(1,1,128)`. Runtime shapes: weight `(4096,4096)` fp8, b_scale tiled `(32,32)` UE8M0, a_scale `(40,4,8)` UE8M0 (512-block). DeepGEMM's `fp8_bmm` requires the weight rank-3 and a per-row 128-block b-scale. The fork never applies the `is_bmm=True` reshape and reuses a *tiled* 128x128 b-scale that the bmm path cannot accept. Three stacked sub-fixes ( `patch_dspark_woa.py`): 1. `wo_a.weight.view(n_local_groups=4, o_lora_rank=1024, -1)` -> `(4,1024,4096)` (use `n_local_groups`, not `n_groups` — must equal the a-side group H so `einsum.hpp:164` batch check passes). 2. a_scale: leave `out_scale` native `(B,H,R/512)` UE8M0 — DeepGEMM permutes the A-side itself; expanding 8->32 cols broke the int branch (`layout.hpp:98 size(-1)`). 3. b_scale: convert tiled `(32,32)` UE8M0 to per-row fp32 `(H,D,R/128)` by broadcasting each tile's scale to its 128 output rows. **Verified on ds4:** `Application startup complete`, `/v1/models` serves `deepseek-v4-flash`, and a chat/completion request returns generated reasoning tokens. Both rank-0 (dgx3) and rank-1 (dgx4) containers up. Note: the `deepseek-v4-flash-eugr` image sidesteps all of this (different newer fork, b12x integrated); it remains the alternative on DS4 if a stock-image DSpark deploy is preferred. The `dspark38` overlay retains the b12x MoE/perf path and only patches the two broken lines. ## Acceptance Criteria - [x] `deepseek-v4-flash-dspark38` boots on `production-3.8` via the `dsv4-vllm-entrypoint` wrapper. - [x] No `VLLM_USE_V2_MODEL_RUNNER`, no `VLLM_B12X_TINY_DECODE`. - [x] `/cache` JIT dirs persist via `[[extra_mounts]]`. - [x] DeepGEMM `t.dim() == N` crash resolved (drafter wo_a reconstruction). - [ ] DSpark k=5 decode **throughput** verified vs 3.75 baseline (boot + generation smoke-test pass; perf benchmark is a follow-up once a node is free / on request).