launcher/specs/bugs/BUG-2026-08-03T204037-production-3.8-image-update-failures.md

---

bug_id: BUG-2026-08-03T204037 status: open severity: high scope: recipes

title: "DeepSeek-V4-Flash recipe fails to update to aidendle94/sparkrun-vllm-ds4-gb10:production-3.8"

BUG-2026-08-03T204037: DeepSeek-V4-Flash recipe fails to update to production-3.8

Problem

Actual: Updating the deepseek-v4-flash recipe + deepseek-v4-flash-dspark overlay from the working production-3.75 image to the new aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 image fails to boot in three distinct, compounding ways:

1. Does not run with the V2 model runner (VLLM_USE_V2_MODEL_RUNNER=1, carried over from the 3.75 recipe).
2. OOMs with the "tiny kernel" — the opt-in DSPARK tiny-A8 decode kernel (VLLM_B12X_TINY_DECODE=1) exhausts GPU memory at load time.

3. DeepGEMM fails to compile when both v2 and tiny are deactivated, crashing with:

   Assertion error (/tmp/dsv4-vllm-...-3dnaf_2y/sources/DeepGEMM/csrc/apis/../
   jit_kernels/impls/../heuristics/../../utils/layout.hpp:39): t.dim() == N
   

Expected: The recipe should boot on production-3.8 as it does on production-3.75, with DSpark speculative decoding (k=5) and the b12x MoE backend.

Context: The cluster is serving live traffic at the time of writing, so no runtime/boot experiment could be performed. This investigation is based on non-invasive image inspection only (docker image inspect, docker history, docker run without --gpus running pip show / source grep — never importing vLLM or torch, never allocating GPU memory). All four physical DGX Spark boxes are busy serving glm52-nf4-lloyd-experts-gptq on the 4nodes-switch cluster; the 2nodes-dac cluster shares the same hardware (different data-plane IPs, same management IPs).

Failed attempts recovered from git:

• Commit 72f40b3 (reverted by 3a783ee): FROM→3.8, patch path /opt/venv→/opt/env, dropped instanttensor, --load-format auto, removed --attention-backend FLASHINFER_MLA_SPARSE_DSV4 (correct — does not exist in vLLM 0.21), set VLLM_ATTENTION_BACKEND=B12X_MLA_SPARSE, raised k 5→7, gpu_util 0.84→0.9, max-len 524288→393216, dotted kwargs→ single JSON arg, added VLLM_B12X_TINY_DECODE=1/MAX_M=8/CHUNK=8, dropped 5 stale env vars.
• stash@{0}: same direction minus tiny-decode, k stays 5, gpu_util 0.84.

Both kept VLLM_USE_V2_MODEL_RUNNER=1 from the 3.75 recipe.

Root Cause Analysis

The two images are different forks, not version bumps

production-3.8 and the working eugr/spark-vllm-b12x are fundamentally different b12x forks, not a linear version bump from 3.75:

	production-3.8 ("dsv4/apostolic" harness)	eugr/spark-vllm-b12x ("gilded-gnosis")	production-3.75 (working)
vLLM	0.21.1rc1.dev339+g1967a56 (2026-05-28)	0.1.dev19023+g30038602b (2026-08-02)	0.11.2 eldritch
torch	2.11.0+cu130	2.12.0+cu130	2.11+cu130
flashinfer	0.6.12	0.6.17	—
triton	3.6.0	3.7.0	—
b12x	standalone /opt/b12x v0.15.3	integrated into vLLM	standalone
DeepGEMM	vendored src at /opt/vllm-apostolic/.deps/deepgemm-src/ (commit 1f2f161)	not visible	—
base	micromamba/conda /opt/env	nvidia/cuda:13.0.2-devel-ubuntu24.04	/opt/venv
NCCL	2.30.4 (conda)	2.28.3 (apt)	2.30.4
instanttensor	absent	0.1.9 present	present
launch	wrapper /usr/local/bin/dsv4-vllm-entrypoint; base CMD sleep infinity	ENTRY /opt/nvidia/nvidia_entrypoint.sh	direct vllm

Provenance (from /opt/dsv4/provenance.json in the 3.8 image): production-3.8 is built by a "dsv4" harness (config sha f9dcce6c..., artifact dsv4-vllm-sm120-20260528-arm64-nccl2304-linux-arm64-2a32b9cc4d9f), base image mambaorg/micromamba:2.3.3, conda env vllm at /opt/env. The deepgemm source is a "sanitized DeepGEMM export via DEEPGEMM_SRC_DIR and VLLM_DEEPGEMM_SRC" (commit 1f2f161dba747b7c12671d017f7c88e1249c3d3e).

production-3.8 is a "clean-overlay" base meant to be launched via a wrapper

The decisive structural difference: production-3.8 is a clean base (CMD ["sleep","infinity"], ENTRY null) designed to be launched via /usr/local/bin/dsv4-vllm-entrypoint, which:

• prepends /opt/env/bin to PATH,
• sets the JIT toolchain env (DG_JIT_NVCC_COMPILER=/opt/env/bin/nvcc, FLASHINFER_NVCC, PYTORCH_NVCC, NVCC_PREPEND_FLAGS with the CCCL include path, CUDAHOSTCXX),
• sets the JIT cache dirs (DG_JIT_CACHE_DIR=/cache/deep_gemm_jit, FLASHINFER_WORKSPACE_BASE, VLLM_CACHE_ROOT, TILELANG_CACHE_DIR, TRITON_CACHE_DIR, TORCHINDUCTOR_CACHE_DIR, TORCH_EXTENSIONS_DIR),
• mkdir -ps those dirs (they do not exist in the bare image),
• then exec vllm "$@".

By keeping entrypoint = ["vllm", "serve"] (the 3.75 contract), the recipe bypasses the wrapper, so the image's intended launch contract is broken. nvcc discovery happens to still work (it is on PATH and CUDA_HOME is baked into the image env), but the cache-dir layout and the CCCL include paths are unset — making DeepGEMM JIT fragile. production-3.75 worked because it was a self-contained serving image where vLLM was directly runnable with all env baked in.

Symptom #3 — DeepGEMM layout.hpp:39 t.dim() == N (the boot-blocker)

The real failing file is /opt/vllm-apostolic/.deps/deepgemm-src/csrc/utils/layout.hpp (the stack path apis/../jit_kernels/impls/../heuristics/../../utils/layout.hpp resolves there). Line 39:

template <int N>
static auto get_shape(const torch::Tensor& t) {
    DG_HOST_ASSERT(t.dim() == N);   // ← line 39, N is a compile-time template arg

This is not a missing-nvcc / cache-dir problem. It is a tensor-rank contract violation: vLLM calls DeepGEMM's transform_sf_into_required_layout (bound in csrc/apis/layout.hpp, which calls get_shape<N>()) passing a scale-factor tensor whose rank does not match the template specialization. The binding check_sf_layout asserts sf.dim() == 2; the grouped / mega-moe path expects rank-3 (get_shape<3>). A rank mismatch trips line 39.

It fires during MoE weight-loading / scale-packing (vllm/model_executor/layers/quantization/utils/fp8_utils.py:1048 and .../fused_moe/oracle/mxfp4.py:758), before any kernel runs. The recipe's --moe-backend b12x does not bypass it: transform_sf_into_required_layout is called during MoE weight prep regardless of the selected MoE compute backend.

This is a fork bug / version-skew between this specific DeepGEMM fork (1f2f161) and what vLLM 0.21's transform_sf_into_required_layout callers feed it — not an env mistake, and the one that blocks boot entirely.

Symptom #2 — "OOMs with tiny kernel"

The opt-in tiny-decode kernel (/opt/tinydec/tiny_hook.py, /opt/tinydec/tiny_moe.py) is a runtime monkey-patch on B12xExperts.apply/_get_or_prepare_fp4_moe_weights that routes small-M MoE decode through a custom w4a8mx Triton kernel. Its own docstring states the OOM mechanism directly:

"Chunking is what keeps the load-time repack off the OOM path that crashed the node." "the .int() expansion plus 9-dim reshape temporaries are what OOM'd the head node when run unchunked at E=256."

The hook repacks b12x's w4a16 packed tensors into tiny's w4a8_mx format on GPU at load time, per expert-tensor, caching the result. The failed attempt set --gpu-memory-utilization 0.9 together with VLLM_B12X_TINY_DECODE=1 — the repack needs GPU headroom that 0.9 does not leave. VLLM_B12X_TINY_CHUNK=8 exists precisely to bound the peak, but does not help when the utilization ceiling leaves no room for the repack buffers.

The image label confirms the gating contract: org.tinydec.flag=VLLM_B12X_TINY_DECODE, org.tinydec.kernel=w4a8mx-tiny-decode, org.tinydec.gates="A:cos0.999999 b0:0.81x b:cos0.999990 M:99%".

Symptom #1 — "does not run with v2"

The V2 model runner is experimental in this fork. gpu_worker.py:803 has a hard assert not self.use_v2_model_runner when pipeline-parallel + sequence-parallel is active (not the case here, TP-only), so the v2 failure here is elsewhere in the V2 warmup path (warmup_kernels runs a full execute_model + sample_tokens JIT compile). The known-working glm52 recipe does not set VLLM_USE_V2_MODEL_RUNNER. Carrying =1 over from the 3.75 recipe is a separate, independent source of fragility.

Reproduce

Could not be reproduced at boot time (cluster busy serving). The diagnosis is based on static inspection of both images on dgx1.lan (management IP 192.168.88.14, the 2nodes-dac cluster entry, same physical box as the serving 4nodes-switch rank 0):

# image versions (metadata only, no CUDA import, no GPU allocation):
docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
  -lc '/opt/env/bin/pip3 show vllm torch flashinfer-python triton'

# the wrapper the recipe must use:
docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
  -lc 'cat /usr/local/bin/dsv4-vllm-entrypoint'

# the failing assertion source:
docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
  -lc 'sed -n "30,45p" /opt/vllm-apostolic/.deps/deepgemm-src/csrc/utils/layout.hpp'

All inspection used docker run without --gpus (no GPU device passthrough → no competition with the live glm52 serve), running only pip show / find / grep / sed — never importing vLLM or torch, never allocating GPU memory.

The git-recovered failed attempts:

git show 72f40b3   # reverted 3.8 attempt + tiny-decode
git stash show -p stash@{0}   # 3.8 attempt, no tiny-decode

Fix direction

Given all nodes are occupied (no runtime test possible), the safe path is:

1. Use the dsv4-vllm-entrypoint wrapper as the recipe entrypoint so the image's intended JIT env + cache dirs are active:

   entrypoint = ["/usr/local/bin/dsv4-vllm-entrypoint", "serve"]
   

2. Mirror / rely on the wrapper env (DG_JIT_NVCC_COMPILER, FLASHINFER_NVCC, DG_JIT_CACHE_DIR=/cache/deep_gemm_jit, …) and add an [[extra_mounts]] for /cache so the JIT compile cache persists across restarts (the wrapper mkdir -ps these under /cache).

3. Drop the V2 runner (unset VLLM_USE_V2_MODEL_RUNNER) — experimental in this fork; the working glm52 recipe does not use it.

4. Drop tiny-decode (unset VLLM_B12X_TINY_DECODE) for first bring-up — it is an opt-in experimental kernel with a load-time GPU repack that needs real headroom. Get a stable boot first, then re-enable with --gpu-memory-utilization ≤ 0.84 and VLLM_B12X_TINY_CHUNK=8.

5. The DeepGEMM t.dim() == N crash needs a real fix, not env — it fires during MoE scale packing on this fork. Options to investigate once a node frees up:

   • try --moe-backend deep_gemm_mega_moe or --moe-backend auto (instead of b12x) to see if a different backend avoids the rank-3 scale path;
   • check whether the model's quant config (NVFP4 vs MXFP4) selects a DeepGEMM template that mismatches — mxfp4.py's oracle _pack calls transform_sf_into_required_layout with recipe=(1,1,32) and num_groups=num_experts, feeding a 3D tensor;
   • if it persists, this is a bug to report upstream against the 1f2f161 DeepGEMM fork — the assertion message is the fork's own contract.

The eugr/spark-vllm-b12x image (recipe deepseek-v4-flash-eugr, commit a8e4932) sidesteps all of this because it is a different, newer fork (gilded-gnosis, 2026-08-02) where b12x is integrated and the DSV4 path is already wired — which is why that recipe boots cleanly. The two images are not interchangeable at the recipe level; a 3.8 recipe must respect the wrapper-entrypoint + clean-base contract.

Acceptance Criteria

• deepseek-v4-flash (or a new -dspark38 variant) boots on production-3.8 via the dsv4-vllm-entrypoint wrapper.
• No VLLM_USE_V2_MODEL_RUNNER, no VLLM_B12X_TINY_DECODE in the first stable 3.8 recipe.
• /cache JIT dirs persist via an [[extra_mounts]] (restarts reuse compiled kernels).
• The DeepGEMM t.dim() == N crash is either resolved (backend/quant config change) or confirmed as an upstream fork bug with a workaround.
• DSpark k=5 decode acceptance + throughput verified against the 3.75 baseline once a node is free.

Resolution

Open. A deepseek-v4-flash-dspark38 recipe + overlay applying points 1–4 is the recommended next step (drafted but not yet written, pending node availability for a boot test). Point 5 (DeepGEMM rank contract) is the remaining unknown and the most likely blocker.