# BUG-2026-09-14: r37 tree garbles DeepSeek-V4.1-Flash text on 4x GB10

Status: bisect A-C complete (flags excluded) + **A/B on the lab's newer
09-13 build complete (2026-09-18 night): the garbling is a lab-tree trait,
not an r37 regression**. Boot D (engram off) is infeasible on 4x — see
Implication.

## Symptom

On `jovian-r37-ds41:local` (vLLM `c687594b8a8`, tree `3fb88fdf`, image label
`9350429b...`), DeepSeek-V4.1-Flash (deepseek-ai/DeepSeek-V4.1-Flash, 552B MoE,
4x DGX Spark GB10, TP4) generates text with a high rate of single-letter
corruptions — the model seems to miss the vocabulary:

- "данних, Етот цыфровой оазис среди вечной мерзлоти, навикации, иследований,
  планети, критically важной точкой опори, вахтовим, халодом, видлели
  колосальное количество енергии" (300-word Russian generation, temp 0.7)
- code comments: "возврощает", "валідный" (Ukrainian і), "должин быть словарём"
- second sample: "метал, процесоров, информатсии, системи, мошной, постувает,
  эфективная енергия"
- third sample (prod contract, below): "индекаторами, температу́рами, данние,
  халодный, масивов, потерьял, бистро"

The corruption hits exactly the high-frequency, closely-spaced vocabulary;
structure, identifiers, and long tokens stay intact (5/5 near-identical
config keys emitted correctly in a code test).

## Control (clean)

The same checkpoint + same 4 nodes + same weights served by the
tonyd2wild-overlay image (`dsv41-flash-b12x:local`: `vllm/vllm-openai`
digest-pinned arm64 nightly + dsv41-feat@e47aa780 + 7 boot-10 patches)
produces clean text (production of record, ~4 weeks of agent traffic).

## Bisect (r37 image, one change per boot, same test prompt)

| Boot | Change vs r37 contract | Result |
|------|------------------------|--------|
| A    | remove `--linear-backend b12x` | garbled |
| B    | prod spec: K5 probabilistic/block, adaptive off (was K7 greedy/standard/adaptive) | garbled |
| C    | FULL prod contract: GMU 0.85, seqs 8, tokens 8192, block 128, K5-tuned capture sizes, no B12X attention/linear backends, no instanttensor | garbled |
| D    | C + engram disabled (no `--engram-config`) | **infeasible on 4x** — without disk mode + lazy load the full 510 GB (incl. the 406 GB engram table) loads eagerly into DRAM: 127.6 GB/node > 121.6 GiB, all 4 nodes OOM'd (2026-09-18 14:37 UTC, power-cycled + recovered). The engram table is part of the model weights, not a toggle — the off-state only exists at 8x (64 GB/node). |

Excluded along the way: runtime LM-head MXFP8 quantization
(`VLLM_MXFP8_LM_HEAD=0` verified in-container, no "Quantizing LM head" log
line, KV pool shrank 10.39M -> 10.19M tokens proving the BF16 head) — the
typos persist with the BF16 head.

## A/B: the lab's newer build (2026-09-18 night)

`eugr/spark-vllm-b12x:nightly-20260913` (`v0.1.dev20759+gb40673cd0.d20260913`
— the lab's `dev/infernal-invocation` branch, newer than r37's `3fb88fdf`)
running the same measured contract (GMU 0.88, 512K DCP1, block 256, B12X
trio, K7 greedy/adaptive, engram disk, instanttensor/lazy, `VLLM_MXFP8_LM_HEAD=0`)
on the same 4 nodes / same NAS snapshot:

- 300-word Russian probe: **garbled, same pattern** ("абслуживание",
  "парамитры", "мадель", "тензары", "матречная аперация", "маштабировать",
  "гаризантально", "упровляемой", "полнастью", "интелекта" — the
  close-vocabulary single-letter corruption, ~30 errors in 2.4 KB).
- Identifier probe: 6/6 exact (structure/code clean, as on r37).

Both independent 09-13 lab builds (r37 vllmc687594b + 0913 b40673cd0) garble;
the tonyd2wild prod chain (vllm/vllm-openai nightly + dsv41-feat@e47aa780 +
boot-10 patches) is clean. The shared factor is the lab's DS4.1 model-code
line as of 09-13 — not our build (aarch64/sm_121a deviation), not the
contract, not the MXFP8 LM head (opted out in all runs).

## Implication

The lab's DS4.1 line is not quality-qualified for agent text on this stack
(as of the 09-13 builds). The defect sits in code common to both lab builds:
the `deepseek_v32`/DS4.1 model code, the b12x MoE kernels, or the engram
disk-read path. DS4.1 stays on the tonyd2wild prod chain of record; the lab
line remains usable only where it is proven (the full GLM-5.3 lane, where
r37/0913 output is clean — see the 2026-09-18 night P0: full GLM + DSpark
K8, quality gate passed).

## Related (same tree, same week, already documented elsewhere)

- DCP>1 sparse-decode crash (`deepseek_v32/attention.py:596`, both r35 and
  r37) — blocks 512K on the full GLM-5.3 model.
- `FlashInferMxFp4LinearKernel.is_supported` accepts sm_121 while cute-dsl
  `mm_fp4` rejects it.
- b12x mxfp4-linear warmup rank-3-vs-4 crash on RedHatAI GLM-5.3-MXFP4 shapes.

## Repro assets

- Recipe: `recipes/dsv41-flash-jovian-r37-4x/recipe.toml` (bisect log in the
  args comments); A/B recipe `recipes/dsv41-flash-eugr-0913-4x/recipe.toml`.
- Test: greedy 300-word Russian generation, 3 samples above (temp 0.7/0.0)
  + the 6-string TOML identifier probe (pass on all garbled builds).
- Boot logs: node `/var/lib/docker/containers/*` (kept via `--keep` until the
  outage); 0913 boot logs on dgx1, container
  `sm121_dsv41-flash-eugr-0913-4x_0`.