BUG-2026-09-14-r37-ds41-text-garbling.md 3.4 KB
BUG-2026-09-14: r37 tree garbles DeepSeek-V4.1-Flash text on 4x GB10
Status: bisect A-C complete (flags excluded), D pending (cluster outage).
Symptom
On jovian-r37-ds41:local (vLLM c687594b8a8, tree 3fb88fdf, image label
9350429b...), DeepSeek-V4.1-Flash (deepseek-ai/DeepSeek-V4.1-Flash, 552B MoE,
4x DGX Spark GB10, TP4) generates text with a high rate of single-letter
corruptions — the model seems to miss the vocabulary:
- "данних, Етот цыфровой оазис среди вечной мерзлоти, навикации, иследований, планети, критically важной точкой опори, вахтовим, халодом, видлели колосальное количество енергии" (300-word Russian generation, temp 0.7)
- code comments: "возврощает", "валідный" (Ukrainian і), "должин быть словарём"
- second sample: "метал, процесоров, информатсии, системи, мошной, постувает, эфективная енергия"
- third sample (prod contract, below): "индекаторами, температу́рами, данние, халодный, масивов, потерьял, бистро"
The corruption hits exactly the high-frequency, closely-spaced vocabulary; structure, identifiers, and long tokens stay intact (5/5 near-identical config keys emitted correctly in a code test).
Control (clean)
The same checkpoint + same 4 nodes + same weights served by the
tonyd2wild-overlay image (dsv41-flash-b12x:local: vllm/vllm-openai
digest-pinned arm64 nightly + dsv41-feat@e47aa780 + 7 boot-10 patches)
produces clean text (production of record, ~4 weeks of agent traffic).
Bisect (r37 image, one change per boot, same test prompt)
| Boot | Change vs r37 contract | Result |
|---|---|---|
| A | remove --linear-backend b12x |
garbled |
| B | prod spec: K5 probabilistic/block, adaptive off (was K7 greedy/standard/adaptive) | garbled |
| C | FULL prod contract: GMU 0.85, seqs 8, tokens 8192, block 128, K5-tuned capture sizes, no B12X attention/linear backends, no instanttensor | garbled |
| D | C + engram disabled (no --engram-config) |
pending (cluster outage mid-boot, 2026-09-14 ~07:27 UTC) |
Excluded along the way: runtime LM-head MXFP8 quantization
(VLLM_MXFP8_LM_HEAD=0 verified in-container, no "Quantizing LM head" log
line, KV pool shrank 10.39M -> 10.19M tokens proving the BF16 head) — the
typos persist with the BF16 head.
Implication
The defect is in the r37 tree itself (not the contract flags): the default
attention/dense path, the deepseek_v32 model code, the b12x MoE kernels, or
the engram disk-read implementation — or a combination. Boot D separates the
engram hypothesis from the rest.
Related (same tree, same week, already documented elsewhere)
- DCP>1 sparse-decode crash (
deepseek_v32/attention.py:596, both r35 and r37) — blocks 512K on the full GLM-5.3 model. FlashInferMxFp4LinearKernel.is_supportedaccepts sm_121 while cute-dslmm_fp4rejects it.- b12x mxfp4-linear warmup rank-3-vs-4 crash on RedHatAI GLM-5.3-MXFP4 shapes.
Repro assets
- Recipe:
recipes/dsv41-flash-jovian-r37-4x/recipe.toml(bisect log in the args comments). - Test: greedy 300-word Russian generation, 3 samples above (temp 0.7/0.0).
- Boot logs: node
/var/lib/docker/containers/*(kept via--keepuntil the outage).