BUG-2026-09-14-r37-ds41-text-garbling.md 5.4 KB
BUG-2026-09-14: r37 tree garbles DeepSeek-V4.1-Flash text on 4x GB10
Status: bisect A-C complete (flags excluded) + A/B on the lab's newer 09-13 build complete (2026-09-18 night): the garbling is a lab-tree trait, not an r37 regression. Boot D (engram off) is infeasible on 4x — see Implication.
Symptom
On jovian-r37-ds41:local (vLLM c687594b8a8, tree 3fb88fdf, image label
9350429b...), DeepSeek-V4.1-Flash (deepseek-ai/DeepSeek-V4.1-Flash, 552B MoE,
4x DGX Spark GB10, TP4) generates text with a high rate of single-letter
corruptions — the model seems to miss the vocabulary:
- "данних, Етот цыфровой оазис среди вечной мерзлоти, навикации, иследований, планети, критically важной точкой опори, вахтовим, халодом, видлели колосальное количество енергии" (300-word Russian generation, temp 0.7)
- code comments: "возврощает", "валідный" (Ukrainian і), "должин быть словарём"
- second sample: "метал, процесоров, информатсии, системи, мошной, постувает, эфективная енергия"
- third sample (prod contract, below): "индекаторами, температу́рами, данние, халодный, масивов, потерьял, бистро"
The corruption hits exactly the high-frequency, closely-spaced vocabulary; structure, identifiers, and long tokens stay intact (5/5 near-identical config keys emitted correctly in a code test).
Control (clean)
The same checkpoint + same 4 nodes + same weights served by the
tonyd2wild-overlay image (dsv41-flash-b12x:local: vllm/vllm-openai
digest-pinned arm64 nightly + dsv41-feat@e47aa780 + 7 boot-10 patches)
produces clean text (production of record, ~4 weeks of agent traffic).
Bisect (r37 image, one change per boot, same test prompt)
| Boot | Change vs r37 contract | Result |
|---|---|---|
| A | remove --linear-backend b12x |
garbled |
| B | prod spec: K5 probabilistic/block, adaptive off (was K7 greedy/standard/adaptive) | garbled |
| C | FULL prod contract: GMU 0.85, seqs 8, tokens 8192, block 128, K5-tuned capture sizes, no B12X attention/linear backends, no instanttensor | garbled |
| D | C + engram disabled (no --engram-config) |
infeasible on 4x — without disk mode + lazy load the full 510 GB (incl. the 406 GB engram table) loads eagerly into DRAM: 127.6 GB/node > 121.6 GiB, all 4 nodes OOM'd (2026-09-18 14:37 UTC, power-cycled + recovered). The engram table is part of the model weights, not a toggle — the off-state only exists at 8x (64 GB/node). |
Excluded along the way: runtime LM-head MXFP8 quantization
(VLLM_MXFP8_LM_HEAD=0 verified in-container, no "Quantizing LM head" log
line, KV pool shrank 10.39M -> 10.19M tokens proving the BF16 head) — the
typos persist with the BF16 head.
A/B: the lab's newer build (2026-09-18 night)
eugr/spark-vllm-b12x:nightly-20260913 (v0.1.dev20759+gb40673cd0.d20260913
— the lab's dev/infernal-invocation branch, newer than r37's 3fb88fdf)
running the same measured contract (GMU 0.88, 512K DCP1, block 256, B12X
trio, K7 greedy/adaptive, engram disk, instanttensor/lazy, VLLM_MXFP8_LM_HEAD=0)
on the same 4 nodes / same NAS snapshot:
- 300-word Russian probe: garbled, same pattern ("абслуживание", "парамитры", "мадель", "тензары", "матречная аперация", "маштабировать", "гаризантально", "упровляемой", "полнастью", "интелекта" — the close-vocabulary single-letter corruption, ~30 errors in 2.4 KB).
- Identifier probe: 6/6 exact (structure/code clean, as on r37).
Both independent 09-13 lab builds (r37 vllmc687594b + 0913 b40673cd0) garble;
the tonyd2wild prod chain (vllm/vllm-openai nightly + dsv41-feat@e47aa780 +
boot-10 patches) is clean. The shared factor is the lab's DS4.1 model-code
line as of 09-13 — not our build (aarch64/sm_121a deviation), not the
contract, not the MXFP8 LM head (opted out in all runs).
Implication
The lab's DS4.1 line is not quality-qualified for agent text on this stack
(as of the 09-13 builds). The defect sits in code common to both lab builds:
the deepseek_v32/DS4.1 model code, the b12x MoE kernels, or the engram
disk-read path. DS4.1 stays on the tonyd2wild prod chain of record; the lab
line remains usable only where it is proven (the full GLM-5.3 lane, where
r37/0913 output is clean — see the 2026-09-18 night P0: full GLM + DSpark
K8, quality gate passed).
Related (same tree, same week, already documented elsewhere)
- DCP>1 sparse-decode crash (
deepseek_v32/attention.py:596, both r35 and r37) — blocks 512K on the full GLM-5.3 model. FlashInferMxFp4LinearKernel.is_supportedaccepts sm_121 while cute-dslmm_fp4rejects it.- b12x mxfp4-linear warmup rank-3-vs-4 crash on RedHatAI GLM-5.3-MXFP4 shapes.
Repro assets
- Recipe:
recipes/dsv41-flash-jovian-r37-4x/recipe.toml(bisect log in the args comments); A/B reciperecipes/dsv41-flash-eugr-0913-4x/recipe.toml. - Test: greedy 300-word Russian generation, 3 samples above (temp 0.7/0.0)
- the 6-string TOML identifier probe (pass on all garbled builds).
- Boot logs: node
/var/lib/docker/containers/*(kept via--keepuntil the outage); 0913 boot logs on dgx1, containersm121_dsv41-flash-eugr-0913-4x_0.