Inicio Explorar Axuda
Iniciar sesión
sm121
/
launcher
2
0
Fork 0
Ficheiros Incidencias 0 Pull Requests 0 Wiki
Árbore: 198e6137dd
Ramas Etiquetas
launcher
/
specs
/
bugs
/
BUG-2026-08-03T204037-production-3.8-image-update-failures.md

BUG-2026-08-03T204037-production-3.8-image-update-failures.md 15 KB
Histórico Raw

bug_id: BUG-2026-08-03T204037 status: resolved severity: high scope: recipes

title: "DeepSeek-V4-Flash recipe fails to update to aidendle94/sparkrun-vllm-ds4-gb10:production-3.8"

BUG-2026-08-03T204037: DeepSeek-V4-Flash recipe fails to update to production-3.8

Problem

Actual: Updating the deepseek-v4-flash recipe + deepseek-v4-flash-dspark overlay from the working production-3.75 image to the new aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 image fails to boot in three distinct, compounding ways:

  1. Does not run with the V2 model runner (VLLM_USE_V2_MODEL_RUNNER=1, carried over from the 3.75 recipe).
  2. OOMs with the "tiny kernel" — the opt-in DSPARK tiny-A8 decode kernel (VLLM_B12X_TINY_DECODE=1) exhausts GPU memory at load time.

  3. DeepGEMM fails to compile when both v2 and tiny are deactivated, crashing with:

    Assertion error (/tmp/dsv4-vllm-...-3dnaf_2y/sources/DeepGEMM/csrc/apis/../
jit_kernels/impls/../heuristics/../../utils/layout.hpp:39): t.dim() == N

Expected: The recipe should boot on production-3.8 as it does on production-3.75, with DSpark speculative decoding (k=5) and the b12x MoE backend.

Context: The cluster is serving live traffic at the time of writing, so no runtime/boot experiment could be performed. This investigation is based on non-invasive image inspection only (docker image inspect, docker history, docker run without --gpus running pip show / source grep — never importing vLLM or torch, never allocating GPU memory). All four physical DGX Spark boxes are busy serving glm52-nf4-lloyd-experts-gptq on the 4nodes-switch cluster; the 2nodes-dac cluster shares the same hardware (different data-plane IPs, same management IPs).

Failed attempts recovered from git:

  • Commit 72f40b3 (reverted by 3a783ee): FROM→3.8, patch path /opt/venv→/opt/env, dropped instanttensor, --load-format auto, removed --attention-backend FLASHINFER_MLA_SPARSE_DSV4 (correct — does not exist in vLLM 0.21), set VLLM_ATTENTION_BACKEND=B12X_MLA_SPARSE, raised k 5→7, gpu_util 0.84→0.9, max-len 524288→393216, dotted kwargs→ single JSON arg, added VLLM_B12X_TINY_DECODE=1/MAX_M=8/CHUNK=8, dropped 5 stale env vars.
  • stash@{0}: same direction minus tiny-decode, k stays 5, gpu_util 0.84.

Both kept VLLM_USE_V2_MODEL_RUNNER=1 from the 3.75 recipe.

Root Cause Analysis

The two images are different forks, not version bumps

production-3.8 and the working eugr/spark-vllm-b12x are fundamentally different b12x forks, not a linear version bump from 3.75:

production-3.8 ("dsv4/apostolic" harness) eugr/spark-vllm-b12x ("gilded-gnosis") production-3.75 (working)
vLLM 0.21.1rc1.dev339+g1967a56 (2026-05-28) 0.1.dev19023+g30038602b (2026-08-02) 0.11.2 eldritch
torch 2.11.0+cu130 2.12.0+cu130 2.11+cu130
flashinfer 0.6.12 0.6.17 —
triton 3.6.0 3.7.0 —
b12x standalone /opt/b12x v0.15.3 integrated into vLLM standalone
DeepGEMM vendored src at /opt/vllm-apostolic/.deps/deepgemm-src/ (commit 1f2f161) not visible —
base micromamba/conda /opt/env nvidia/cuda:13.0.2-devel-ubuntu24.04 /opt/venv
NCCL 2.30.4 (conda) 2.28.3 (apt) 2.30.4
instanttensor absent 0.1.9 present present
launch wrapper /usr/local/bin/dsv4-vllm-entrypoint; base CMD sleep infinity ENTRY /opt/nvidia/nvidia_entrypoint.sh direct vllm

Provenance (from /opt/dsv4/provenance.json in the 3.8 image): production-3.8 is built by a "dsv4" harness (config sha f9dcce6c..., artifact dsv4-vllm-sm120-20260528-arm64-nccl2304-linux-arm64-2a32b9cc4d9f), base image mambaorg/micromamba:2.3.3, conda env vllm at /opt/env. The deepgemm source is a "sanitized DeepGEMM export via DEEPGEMM_SRC_DIR and VLLM_DEEPGEMM_SRC" (commit 1f2f161dba747b7c12671d017f7c88e1249c3d3e).

production-3.8 is a "clean-overlay" base meant to be launched via a wrapper

The decisive structural difference: production-3.8 is a clean base (CMD ["sleep","infinity"], ENTRY null) designed to be launched via /usr/local/bin/dsv4-vllm-entrypoint, which:

  • prepends /opt/env/bin to PATH,
  • sets the JIT toolchain env (DG_JIT_NVCC_COMPILER=/opt/env/bin/nvcc, FLASHINFER_NVCC, PYTORCH_NVCC, NVCC_PREPEND_FLAGS with the CCCL include path, CUDAHOSTCXX),
  • sets the JIT cache dirs (DG_JIT_CACHE_DIR=/cache/deep_gemm_jit, FLASHINFER_WORKSPACE_BASE, VLLM_CACHE_ROOT, TILELANG_CACHE_DIR, TRITON_CACHE_DIR, TORCHINDUCTOR_CACHE_DIR, TORCH_EXTENSIONS_DIR),
  • mkdir -ps those dirs (they do not exist in the bare image),
  • then exec vllm "$@".

By keeping entrypoint = ["vllm", "serve"] (the 3.75 contract), the recipe bypasses the wrapper, so the image's intended launch contract is broken. nvcc discovery happens to still work (it is on PATH and CUDA_HOME is baked into the image env), but the cache-dir layout and the CCCL include paths are unset — making DeepGEMM JIT fragile. production-3.75 worked because it was a self-contained serving image where vLLM was directly runnable with all env baked in.

Symptom #3 — DeepGEMM layout.hpp:39 t.dim() == N (the boot-blocker)

The real failing file is /opt/vllm-apostolic/.deps/deepgemm-src/csrc/utils/layout.hpp (the stack path apis/../jit_kernels/impls/../heuristics/../../utils/layout.hpp resolves there). Line 39:

template <int N>
static auto get_shape(const torch::Tensor& t) {
    DG_HOST_ASSERT(t.dim() == N);   // ← line 39, N is a compile-time template arg

This is not a missing-nvcc / cache-dir problem. It is a tensor-rank contract violation: vLLM calls DeepGEMM's transform_sf_into_required_layout (bound in csrc/apis/layout.hpp, which calls get_shape<N>()) passing a scale-factor tensor whose rank does not match the template specialization. The binding check_sf_layout asserts sf.dim() == 2; the grouped / mega-moe path expects rank-3 (get_shape<3>). A rank mismatch trips line 39.

It fires during MoE weight-loading / scale-packing (vllm/model_executor/layers/quantization/utils/fp8_utils.py:1048 and .../fused_moe/oracle/mxfp4.py:758), before any kernel runs. The recipe's --moe-backend b12x does not bypass it: transform_sf_into_required_layout is called during MoE weight prep regardless of the selected MoE compute backend.

This is a fork bug / version-skew between this specific DeepGEMM fork (1f2f161) and what vLLM 0.21's transform_sf_into_required_layout callers feed it — not an env mistake, and the one that blocks boot entirely.

Symptom #2 — "OOMs with tiny kernel"

The opt-in tiny-decode kernel (/opt/tinydec/tiny_hook.py, /opt/tinydec/tiny_moe.py) is a runtime monkey-patch on B12xExperts.apply/_get_or_prepare_fp4_moe_weights that routes small-M MoE decode through a custom w4a8mx Triton kernel. Its own docstring states the OOM mechanism directly:

"Chunking is what keeps the load-time repack off the OOM path that crashed the node." "the .int() expansion plus 9-dim reshape temporaries are what OOM'd the head node when run unchunked at E=256."

The hook repacks b12x's w4a16 packed tensors into tiny's w4a8_mx format on GPU at load time, per expert-tensor, caching the result. The failed attempt set --gpu-memory-utilization 0.9 together with VLLM_B12X_TINY_DECODE=1 — the repack needs GPU headroom that 0.9 does not leave. VLLM_B12X_TINY_CHUNK=8 exists precisely to bound the peak, but does not help when the utilization ceiling leaves no room for the repack buffers.

The image label confirms the gating contract: org.tinydec.flag=VLLM_B12X_TINY_DECODE, org.tinydec.kernel=w4a8mx-tiny-decode, org.tinydec.gates="A:cos0.999999 b0:0.81x b:cos0.999990 M:99%".

Symptom #1 — "does not run with v2"

The V2 model runner is experimental in this fork. gpu_worker.py:803 has a hard assert not self.use_v2_model_runner when pipeline-parallel + sequence-parallel is active (not the case here, TP-only), so the v2 failure here is elsewhere in the V2 warmup path (warmup_kernels runs a full execute_model + sample_tokens JIT compile). The known-working glm52 recipe does not set VLLM_USE_V2_MODEL_RUNNER. Carrying =1 over from the 3.75 recipe is a separate, independent source of fragility.

Reproduce

Could not be reproduced at boot time (cluster busy serving). The diagnosis is based on static inspection of both images on dgx1.lan (management IP 192.168.88.14, the 2nodes-dac cluster entry, same physical box as the serving 4nodes-switch rank 0):

# image versions (metadata only, no CUDA import, no GPU allocation):
docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
  -lc '/opt/env/bin/pip3 show vllm torch flashinfer-python triton'

# the wrapper the recipe must use:
docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
  -lc 'cat /usr/local/bin/dsv4-vllm-entrypoint'

# the failing assertion source:
docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
  -lc 'sed -n "30,45p" /opt/vllm-apostolic/.deps/deepgemm-src/csrc/utils/layout.hpp'

All inspection used docker run without --gpus (no GPU device passthrough → no competition with the live glm52 serve), running only pip show / find / grep / sed — never importing vLLM or torch, never allocating GPU memory.

The git-recovered failed attempts:

git show 72f40b3   # reverted 3.8 attempt + tiny-decode
git stash show -p stash@{0}   # 3.8 attempt, no tiny-decode

Fix direction

Given all nodes are occupied (no runtime test possible), the safe path is:

  1. Use the dsv4-vllm-entrypoint wrapper as the recipe entrypoint so the image's intended JIT env + cache dirs are active:

    entrypoint = ["/usr/local/bin/dsv4-vllm-entrypoint", "serve"]

  2. Mirror / rely on the wrapper env (DG_JIT_NVCC_COMPILER, FLASHINFER_NVCC, DG_JIT_CACHE_DIR=/cache/deep_gemm_jit, …) and add an [[extra_mounts]] for /cache so the JIT compile cache persists across restarts (the wrapper mkdir -ps these under /cache).

  3. Drop the V2 runner (unset VLLM_USE_V2_MODEL_RUNNER) — experimental in this fork; the working glm52 recipe does not use it.

  4. Drop tiny-decode (unset VLLM_B12X_TINY_DECODE) for first bring-up — it is an opt-in experimental kernel with a load-time GPU repack that needs real headroom. Get a stable boot first, then re-enable with --gpu-memory-utilization ≤ 0.84 and VLLM_B12X_TINY_CHUNK=8.

  5. The DeepGEMM t.dim() == N crash needs a real fix, not env — it fires during MoE scale packing on this fork. Options to investigate once a node frees up:

    • try --moe-backend deep_gemm_mega_moe or --moe-backend auto (instead of b12x) to see if a different backend avoids the rank-3 scale path;
    • check whether the model's quant config (NVFP4 vs MXFP4) selects a DeepGEMM template that mismatches — mxfp4.py's oracle _pack calls transform_sf_into_required_layout with recipe=(1,1,32) and num_groups=num_experts, feeding a 3D tensor;
    • if it persists, this is a bug to report upstream against the 1f2f161 DeepGEMM fork — the assertion message is the fork's own contract.

The eugr/spark-vllm-b12x image (recipe deepseek-v4-flash-eugr, commit a8e4932) sidesteps all of this because it is a different, newer fork (gilded-gnosis, 2026-08-02) where b12x is integrated and the DSV4 path is already wired — which is why that recipe boots cleanly. The two images are not interchangeable at the recipe level; a 3.8 recipe must respect the wrapper-entrypoint + clean-base contract.

Acceptance Criteria (original)

  • deepseek-v4-flash-dspark38 boots on production-3.8 via the dsv4-vllm-entrypoint wrapper.
  • No VLLM_USE_V2_MODEL_RUNNER, no VLLM_B12X_TINY_DECODE.
  • /cache JIT dirs persist via an [[extra_mounts]].
  • The DeepGEMM t.dim() == N crash is resolved (drafter wo_a scale reconstruction; see Resolution).
  • DSpark k=5 decode acceptance + throughput verified against the 3.75 baseline (boot + generation smoke-test pass; perf benchmark pending node availability).

Resolution

Closed (fixed). Boot-tested on the ds4 cluster (dgx3+dgx4, idle).

The t.dim() == N crash was not the MoE scale-packing path the original write-up hypothesized — the DSV4 target + drafter load cleanly via the b12x MoE backend (DeepGEMM MoE packing never fires). Two real, distinct fork bugs blocked boot, both fixed in the deepseek-v4-flash-dspark38 recipe + local overlay image (docker-images/deepseek-v4-flash-dspark38/, build target sparkrun-vllm-ds4-gb10:dspark38):

Bug A — b12x cutlass from_dlpack rejects grad-requiring tensors

During KV-cache profiling the b12x attention norm_weight carries requires_grad=True under torch.compile; b12x/integration/residual_kernels.py _to_kernel_tensor -> from_dlpack(tensor) throws BufferError: Can't export tensors that require gradient, use tensor.detach(). Fixed by .detach() before export (no-op when requires_grad=False).

Bug B — drafter wo_a feeds DeepGEMM fp8_einsum an incompatible scale

The DSpark drafter forward_dspark calls deepseek_v4_fp8_einsum("bhr,hdr->bhd", wo_a.weight, wo_a.weight_scale_inv, …) with recipe (1,1,128). Runtime shapes: weight (4096,4096) fp8, b_scale tiled (32,32) UE8M0, a_scale (40,4,8) UE8M0 (512-block). DeepGEMM's fp8_bmm requires the weight rank-3 and a per-row 128-block b-scale. The fork never applies the is_bmm=True reshape and reuses a tiled 128x128 b-scale that the bmm path cannot accept. Three stacked sub-fixes ( patch_dspark_woa.py):

  1. wo_a.weight.view(n_local_groups=4, o_lora_rank=1024, -1) -> (4,1024,4096) (use n_local_groups, not n_groups — must equal the a-side group H so einsum.hpp:164 batch check passes).
  2. a_scale: leave out_scale native (B,H,R/512) UE8M0 — DeepGEMM permutes the A-side itself; expanding 8->32 cols broke the int branch (layout.hpp:98 size(-1)).
  3. b_scale: convert tiled (32,32) UE8M0 to per-row fp32 (H,D,R/128) by broadcasting each tile's scale to its 128 output rows.

Verified on ds4: Application startup complete, /v1/models serves deepseek-v4-flash, and a chat/completion request returns generated reasoning tokens. Both rank-0 (dgx3) and rank-1 (dgx4) containers up.

Note: the deepseek-v4-flash-eugr image sidesteps all of this (different newer fork, b12x integrated); it remains the alternative on DS4 if a stock-image DSpark deploy is preferred. The dspark38 overlay retains the b12x MoE/perf path and only patches the two broken lines.

Acceptance Criteria

  • deepseek-v4-flash-dspark38 boots on production-3.8 via the dsv4-vllm-entrypoint wrapper.
  • No VLLM_USE_V2_MODEL_RUNNER, no VLLM_B12X_TINY_DECODE.
  • /cache JIT dirs persist via [[extra_mounts]].
  • DeepGEMM t.dim() == N crash resolved (drafter wo_a reconstruction).
  • DSpark k=5 decode throughput verified vs 3.75 baseline (boot + generation smoke-test pass; perf benchmark is a follow-up once a node is free / on request).