docs(specs): investigation — production-3.8 image update failures

Records the diagnosis of why the deepseek-v4-flash recipe fails to move
from production-3.75 to production-3.8 (three compounding symptoms: V2
runner, tiny-decode OOM, DeepGEMM layout.hpp:39 rank assertion).

Root cause: production-3.8 is a clean-overlay base meant to be launched
via /usr/local/bin/dsv4-vllm-entrypoint (sets JIT toolchain env + cache
dirs, execs vllm) — keeping entrypoint=["vllm","serve"] bypasses it.
The DeepGEMM t.dim()==N crash is a deeper fork-skew bug in the 1f2f161
DeepGEMM vendored source, not an env mistake. Investigation was static
inspection only (no GPU allocated — cluster was serving live traffic).

Registered as BUG-2026-08-03T204037 in specs/bugs/registry.yaml.

specs/bugs/BUG-2026-08-03T204037-production-3.8-image-update-failures.md

@@ -0,0 +1,261 @@
+---
+bug_id: BUG-2026-08-03T204037
+status: open
+severity: high
+scope: recipes
+title: "DeepSeek-V4-Flash recipe fails to update to aidendle94/sparkrun-vllm-ds4-gb10:production-3.8"
+---
+
+# BUG-2026-08-03T204037: DeepSeek-V4-Flash recipe fails to update to production-3.8
+
+## Problem
+
+**Actual:** Updating the `deepseek-v4-flash` recipe + `deepseek-v4-flash-dspark`
+overlay from the working `production-3.75` image to the new
+`aidendle94/sparkrun-vllm-ds4-gb10:production-3.8` image fails to boot in three
+distinct, compounding ways:
+
+1. **Does not run with the V2 model runner** (`VLLM_USE_V2_MODEL_RUNNER=1`,
+   carried over from the 3.75 recipe).
+2. **OOMs with the "tiny kernel"** — the opt-in DSPARK tiny-A8 decode kernel
+   (`VLLM_B12X_TINY_DECODE=1`) exhausts GPU memory at load time.
+3. **DeepGEMM fails to compile** when both v2 and tiny are deactivated,
+   crashing with:
+   ```
+   Assertion error (/tmp/dsv4-vllm-...-3dnaf_2y/sources/DeepGEMM/csrc/apis/../
+   jit_kernels/impls/../heuristics/../../utils/layout.hpp:39): t.dim() == N
+   ```
+
+**Expected:** The recipe should boot on `production-3.8` as it does on
+`production-3.75`, with DSpark speculative decoding (k=5) and the b12x MoE
+backend.
+
+**Context:** The cluster is serving live traffic at the time of writing, so
+no runtime/boot experiment could be performed. This investigation is based on
+non-invasive image inspection only (`docker image inspect`, `docker history`,
+`docker run` without `--gpus` running `pip show` / source `grep` — never
+importing vLLM or torch, never allocating GPU memory). All four physical DGX
+Spark boxes are busy serving `glm52-nf4-lloyd-experts-gptq` on the
+`4nodes-switch` cluster; the `2nodes-dac` cluster shares the same hardware
+(different data-plane IPs, same management IPs).
+
+**Failed attempts recovered from git:**
+
+- Commit `72f40b3` (reverted by `3a783ee`): FROM→3.8, patch path
+  `/opt/venv`→`/opt/env`, dropped instanttensor, `--load-format auto`,
+  removed `--attention-backend FLASHINFER_MLA_SPARSE_DSV4` (correct — does
+  not exist in vLLM 0.21), set `VLLM_ATTENTION_BACKEND=B12X_MLA_SPARSE`,
+  raised k 5→7, gpu_util 0.84→0.9, max-len 524288→393216, dotted kwargs→
+  single JSON arg, added `VLLM_B12X_TINY_DECODE=1/MAX_M=8/CHUNK=8`, dropped
+  5 stale env vars.
+- `stash@{0}`: same direction minus tiny-decode, k stays 5, gpu_util 0.84.
+
+Both kept `VLLM_USE_V2_MODEL_RUNNER=1` from the 3.75 recipe.
+
+## Root Cause Analysis
+
+### The two images are different forks, not version bumps
+
+`production-3.8` and the working `eugr/spark-vllm-b12x` are *fundamentally
+different b12x forks*, not a linear version bump from 3.75:
+
+| | **production-3.8** ("dsv4/apostolic" harness) | **eugr/spark-vllm-b12x** ("gilded-gnosis") | **production-3.75** (working) |
+|---|---|---|---|
+| vLLM | `0.21.1rc1.dev339+g1967a56` (2026-05-28) | `0.1.dev19023+g30038602b` (2026-08-02) | `0.11.2` eldritch |
+| torch | 2.11.0+cu130 | 2.12.0+cu130 | 2.11+cu130 |
+| flashinfer | 0.6.12 | 0.6.17 | — |
+| triton | 3.6.0 | 3.7.0 | — |
+| b12x | **standalone** `/opt/b12x` v0.15.3 | **integrated into vLLM** | standalone |
+| DeepGEMM | vendored src at `/opt/vllm-apostolic/.deps/deepgemm-src/` (commit `1f2f161`) | not visible | — |
+| base | micromamba/conda `/opt/env` | nvidia/cuda:13.0.2-devel-ubuntu24.04 | `/opt/venv` |
+| NCCL | 2.30.4 (conda) | 2.28.3 (apt) | 2.30.4 |
+| instanttensor | **absent** | 0.1.9 present | present |
+| launch | wrapper `/usr/local/bin/dsv4-vllm-entrypoint`; base `CMD sleep infinity` | `ENTRY /opt/nvidia/nvidia_entrypoint.sh` | direct `vllm` |
+
+**Provenance** (from `/opt/dsv4/provenance.json` in the 3.8 image):
+`production-3.8` is built by a "dsv4" harness (config sha
+`f9dcce6c...`, artifact `dsv4-vllm-sm120-20260528-arm64-nccl2304-linux-arm64-2a32b9cc4d9f`),
+base image `mambaorg/micromamba:2.3.3`, conda env `vllm` at `/opt/env`. The
+deepgemm source is a "sanitized DeepGEMM export via `DEEPGEMM_SRC_DIR` and
+`VLLM_DEEPGEMM_SRC`" (commit `1f2f161dba747b7c12671d017f7c88e1249c3d3e`).
+
+### production-3.8 is a "clean-overlay" base meant to be launched via a wrapper
+
+The decisive structural difference: `production-3.8` is a clean base
+(`CMD ["sleep","infinity"]`, `ENTRY null`) designed to be launched via
+`/usr/local/bin/dsv4-vllm-entrypoint`, which:
+
+- prepends `/opt/env/bin` to `PATH`,
+- sets the JIT toolchain env (`DG_JIT_NVCC_COMPILER=/opt/env/bin/nvcc`,
+  `FLASHINFER_NVCC`, `PYTORCH_NVCC`, `NVCC_PREPEND_FLAGS` with the CCCL
+  include path, `CUDAHOSTCXX`),
+- sets the JIT cache dirs (`DG_JIT_CACHE_DIR=/cache/deep_gemm_jit`,
+  `FLASHINFER_WORKSPACE_BASE`, `VLLM_CACHE_ROOT`, `TILELANG_CACHE_DIR`,
+  `TRITON_CACHE_DIR`, `TORCHINDUCTOR_CACHE_DIR`, `TORCH_EXTENSIONS_DIR`),
+- `mkdir -p`s those dirs (they do not exist in the bare image),
+- then `exec vllm "$@"`.
+
+By keeping `entrypoint = ["vllm", "serve"]` (the 3.75 contract), the recipe
+**bypasses the wrapper**, so the image's intended launch contract is broken.
+nvcc discovery happens to still work (it is on PATH and `CUDA_HOME` is baked
+into the image env), but the cache-dir layout and the CCCL include paths are
+unset — making DeepGEMM JIT fragile. `production-3.75` worked because it was a
+self-contained serving image where vLLM was directly runnable with all env
+baked in.
+
+### Symptom #3 — DeepGEMM `layout.hpp:39` `t.dim() == N` (the boot-blocker)
+
+The real failing file is
+`/opt/vllm-apostolic/.deps/deepgemm-src/csrc/utils/layout.hpp` (the stack path
+`apis/../jit_kernels/impls/../heuristics/../../utils/layout.hpp` resolves
+there). Line 39:
+
+```cpp
+template <int N>
+static auto get_shape(const torch::Tensor& t) {
+    DG_HOST_ASSERT(t.dim() == N);   // ← line 39, N is a compile-time template arg
+```
+
+This is **not** a missing-nvcc / cache-dir problem. It is a **tensor-rank
+contract violation**: vLLM calls DeepGEMM's `transform_sf_into_required_layout`
+(bound in `csrc/apis/layout.hpp`, which calls `get_shape<N>()`) passing a
+scale-factor tensor whose rank does not match the template specialization. The
+binding `check_sf_layout` asserts `sf.dim() == 2`; the grouped / mega-moe path
+expects rank-3 (`get_shape<3>`). A rank mismatch trips line 39.
+
+It fires during MoE weight-loading / scale-packing
+(`vllm/model_executor/layers/quantization/utils/fp8_utils.py:1048` and
+`.../fused_moe/oracle/mxfp4.py:758`), **before** any kernel runs. The recipe's
+`--moe-backend b12x` does *not* bypass it: `transform_sf_into_required_layout`
+is called during MoE weight prep regardless of the selected MoE compute
+backend.
+
+This is a **fork bug / version-skew** between this specific DeepGEMM fork
+(`1f2f161`) and what vLLM 0.21's `transform_sf_into_required_layout` callers
+feed it — not an env mistake, and the one that blocks boot entirely.
+
+### Symptom #2 — "OOMs with tiny kernel"
+
+The opt-in tiny-decode kernel (`/opt/tinydec/tiny_hook.py`,
+`/opt/tinydec/tiny_moe.py`) is a runtime monkey-patch on
+`B12xExperts.apply`/`_get_or_prepare_fp4_moe_weights` that routes small-M
+MoE decode through a custom w4a8mx Triton kernel. Its own docstring states
+the OOM mechanism directly:
+
+> *"Chunking is what keeps the load-time repack off the OOM path that
+> crashed the node."*
+> *"the `.int()` expansion plus 9-dim reshape temporaries are what OOM'd
+> the head node when run unchunked at E=256."*
+
+The hook repacks b12x's `w4a16` packed tensors into tiny's `w4a8_mx` format
+**on GPU at load time**, per expert-tensor, caching the result. The failed
+attempt set `--gpu-memory-utilization 0.9` **together with**
+`VLLM_B12X_TINY_DECODE=1` — the repack needs GPU headroom that 0.9 does not
+leave. `VLLM_B12X_TINY_CHUNK=8` exists precisely to bound the peak, but does
+not help when the utilization ceiling leaves no room for the repack buffers.
+
+The image label confirms the gating contract:
+`org.tinydec.flag=VLLM_B12X_TINY_DECODE`,
+`org.tinydec.kernel=w4a8mx-tiny-decode`,
+`org.tinydec.gates="A:cos0.999999 b0:0.81x b:cos0.999990 M:99%"`.
+
+### Symptom #1 — "does not run with v2"
+
+The V2 model runner is experimental in this fork. `gpu_worker.py:803` has a
+hard `assert not self.use_v2_model_runner` when **pipeline-parallel +
+sequence-parallel** is active (not the case here, TP-only), so the v2 failure
+here is elsewhere in the V2 warmup path (`warmup_kernels` runs a full
+`execute_model + sample_tokens` JIT compile). The known-working `glm52`
+recipe does **not** set `VLLM_USE_V2_MODEL_RUNNER`. Carrying `=1` over from
+the 3.75 recipe is a separate, independent source of fragility.
+
+## Reproduce
+
+Could not be reproduced at boot time (cluster busy serving). The diagnosis is
+based on static inspection of both images on `dgx1.lan` (management IP
+`192.168.88.14`, the `2nodes-dac` cluster entry, same physical box as the
+serving `4nodes-switch` rank 0):
+
+```
+# image versions (metadata only, no CUDA import, no GPU allocation):
+docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
+  -lc '/opt/env/bin/pip3 show vllm torch flashinfer-python triton'
+
+# the wrapper the recipe must use:
+docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
+  -lc 'cat /usr/local/bin/dsv4-vllm-entrypoint'
+
+# the failing assertion source:
+docker run --rm --entrypoint bash aidendle94/sparkrun-vllm-ds4-gb10:production-3.8 \
+  -lc 'sed -n "30,45p" /opt/vllm-apostolic/.deps/deepgemm-src/csrc/utils/layout.hpp'
+```
+
+All inspection used `docker run` **without `--gpus`** (no GPU device
+passthrough → no competition with the live `glm52` serve), running only
+`pip show` / `find` / `grep` / `sed` — never importing vLLM or torch, never
+allocating GPU memory.
+
+The git-recovered failed attempts:
+
+```
+git show 72f40b3   # reverted 3.8 attempt + tiny-decode
+git stash show -p stash@{0}   # 3.8 attempt, no tiny-decode
+```
+
+## Fix direction
+
+Given all nodes are occupied (no runtime test possible), the safe path is:
+
+1. **Use the `dsv4-vllm-entrypoint` wrapper** as the recipe entrypoint so the
+   image's intended JIT env + cache dirs are active:
+   ```toml
+   entrypoint = ["/usr/local/bin/dsv4-vllm-entrypoint", "serve"]
+   ```
+2. **Mirror / rely on the wrapper env** (`DG_JIT_NVCC_COMPILER`,
+   `FLASHINFER_NVCC`, `DG_JIT_CACHE_DIR=/cache/deep_gemm_jit`, …) and add an
+   `[[extra_mounts]]` for `/cache` so the JIT compile cache persists across
+   restarts (the wrapper `mkdir -p`s these under `/cache`).
+3. **Drop the V2 runner** (unset `VLLM_USE_V2_MODEL_RUNNER`) — experimental in
+   this fork; the working `glm52` recipe does not use it.
+4. **Drop tiny-decode** (unset `VLLM_B12X_TINY_DECODE`) for first bring-up —
+   it is an opt-in *experimental* kernel with a load-time GPU repack that
+   needs real headroom. Get a stable boot first, then re-enable with
+   `--gpu-memory-utilization` ≤ 0.84 and `VLLM_B12X_TINY_CHUNK=8`.
+5. **The DeepGEMM `t.dim() == N` crash needs a real fix, not env** — it fires
+   during MoE scale packing on this fork. Options to investigate once a node
+   frees up:
+   - try `--moe-backend deep_gemm_mega_moe` or `--moe-backend auto` (instead
+     of `b12x`) to see if a different backend avoids the rank-3 scale path;
+   - check whether the model's quant config (NVFP4 vs MXFP4) selects a
+     DeepGEMM template that mismatches — `mxfp4.py`'s oracle `_pack` calls
+     `transform_sf_into_required_layout` with `recipe=(1,1,32)` and
+     `num_groups=num_experts`, feeding a 3D tensor;
+   - if it persists, this is a bug to report upstream against the `1f2f161`
+     DeepGEMM fork — the assertion message is the fork's own contract.
+
+The `eugr/spark-vllm-b12x` image (recipe `deepseek-v4-flash-eugr`,
+commit `a8e4932`) sidesteps *all* of this because it is a different, newer
+fork (`gilded-gnosis`, 2026-08-02) where b12x is integrated and the DSV4
+path is already wired — which is why that recipe boots cleanly. The two
+images are **not interchangeable** at the recipe level; a 3.8 recipe must
+respect the wrapper-entrypoint + clean-base contract.
+
+## Acceptance Criteria
+
+- [ ] `deepseek-v4-flash` (or a new `-dspark38` variant) boots on
+      `production-3.8` via the `dsv4-vllm-entrypoint` wrapper.
+- [ ] No `VLLM_USE_V2_MODEL_RUNNER`, no `VLLM_B12X_TINY_DECODE` in the first
+      stable 3.8 recipe.
+- [ ] `/cache` JIT dirs persist via an `[[extra_mounts]]` (restarts reuse
+      compiled kernels).
+- [ ] The DeepGEMM `t.dim() == N` crash is either resolved (backend/quant
+      config change) or confirmed as an upstream fork bug with a workaround.
+- [ ] DSpark k=5 decode acceptance + throughput verified against the 3.75
+      baseline once a node is free.
+
+## Resolution
+
+Open. A `deepseek-v4-flash-dspark38` recipe + overlay applying points 1–4 is
+the recommended next step (drafted but not yet written, pending node
+availability for a boot test). Point 5 (DeepGEMM rank contract) is the
+remaining unknown and the most likely blocker.

specs/bugs/registry.yaml

@@ -9,3 +9,12 @@
   summary: "Spawned tasks fall back to English (thread-local set_locale)"
   file: specs/bugs/BUG-2026-07-27T120657-spawned-tasks-english.md
   status: open
+
+- bug_id: BUG-2026-08-03T204037
+  date: 2026-08-03
+  severity: high
+  priority: normal
+  scope: recipes
+  summary: "DeepSeek-V4-Flash recipe fails to update to production-3.8 (v2/tiny-kernel/DeepGEMM layout.hpp:39)"
+  file: specs/bugs/BUG-2026-08-03T204037-production-3.8-image-update-failures.md
+  status: open