|
|
@@ -0,0 +1,67 @@
|
|
|
+# BUG-2026-09-14: r37 tree garbles DeepSeek-V4.1-Flash text on 4x GB10
|
|
|
+
|
|
|
+Status: bisect A-C complete (flags excluded), D pending (cluster outage).
|
|
|
+
|
|
|
+## Symptom
|
|
|
+
|
|
|
+On `jovian-r37-ds41:local` (vLLM `c687594b8a8`, tree `3fb88fdf`, image label
|
|
|
+`9350429b...`), DeepSeek-V4.1-Flash (deepseek-ai/DeepSeek-V4.1-Flash, 552B MoE,
|
|
|
+4x DGX Spark GB10, TP4) generates text with a high rate of single-letter
|
|
|
+corruptions — the model seems to miss the vocabulary:
|
|
|
+
|
|
|
+- "данних, Етот цыфровой оазис среди вечной мерзлоти, навикации, иследований,
|
|
|
+ планети, критically важной точкой опори, вахтовим, халодом, видлели
|
|
|
+ колосальное количество енергии" (300-word Russian generation, temp 0.7)
|
|
|
+- code comments: "возврощает", "валідный" (Ukrainian і), "должин быть словарём"
|
|
|
+- second sample: "метал, процесоров, информатсии, системи, мошной, постувает,
|
|
|
+ эфективная енергия"
|
|
|
+- third sample (prod contract, below): "индекаторами, температу́рами, данние,
|
|
|
+ халодный, масивов, потерьял, бистро"
|
|
|
+
|
|
|
+The corruption hits exactly the high-frequency, closely-spaced vocabulary;
|
|
|
+structure, identifiers, and long tokens stay intact (5/5 near-identical
|
|
|
+config keys emitted correctly in a code test).
|
|
|
+
|
|
|
+## Control (clean)
|
|
|
+
|
|
|
+The same checkpoint + same 4 nodes + same weights served by the
|
|
|
+tonyd2wild-overlay image (`dsv41-flash-b12x:local`: `vllm/vllm-openai`
|
|
|
+digest-pinned arm64 nightly + dsv41-feat@e47aa780 + 7 boot-10 patches)
|
|
|
+produces clean text (production of record, ~4 weeks of agent traffic).
|
|
|
+
|
|
|
+## Bisect (r37 image, one change per boot, same test prompt)
|
|
|
+
|
|
|
+| Boot | Change vs r37 contract | Result |
|
|
|
+|------|------------------------|--------|
|
|
|
+| A | remove `--linear-backend b12x` | garbled |
|
|
|
+| B | prod spec: K5 probabilistic/block, adaptive off (was K7 greedy/standard/adaptive) | garbled |
|
|
|
+| C | FULL prod contract: GMU 0.85, seqs 8, tokens 8192, block 128, K5-tuned capture sizes, no B12X attention/linear backends, no instanttensor | garbled |
|
|
|
+| D | C + engram disabled (no `--engram-config`) | **pending** (cluster outage mid-boot, 2026-09-14 ~07:27 UTC) |
|
|
|
+
|
|
|
+Excluded along the way: runtime LM-head MXFP8 quantization
|
|
|
+(`VLLM_MXFP8_LM_HEAD=0` verified in-container, no "Quantizing LM head" log
|
|
|
+line, KV pool shrank 10.39M -> 10.19M tokens proving the BF16 head) — the
|
|
|
+typos persist with the BF16 head.
|
|
|
+
|
|
|
+## Implication
|
|
|
+
|
|
|
+The defect is in the r37 tree itself (not the contract flags): the default
|
|
|
+attention/dense path, the `deepseek_v32` model code, the b12x MoE kernels, or
|
|
|
+the engram disk-read implementation — or a combination. Boot D separates the
|
|
|
+engram hypothesis from the rest.
|
|
|
+
|
|
|
+## Related (same tree, same week, already documented elsewhere)
|
|
|
+
|
|
|
+- DCP>1 sparse-decode crash (`deepseek_v32/attention.py:596`, both r35 and
|
|
|
+ r37) — blocks 512K on the full GLM-5.3 model.
|
|
|
+- `FlashInferMxFp4LinearKernel.is_supported` accepts sm_121 while cute-dsl
|
|
|
+ `mm_fp4` rejects it.
|
|
|
+- b12x mxfp4-linear warmup rank-3-vs-4 crash on RedHatAI GLM-5.3-MXFP4 shapes.
|
|
|
+
|
|
|
+## Repro assets
|
|
|
+
|
|
|
+- Recipe: `recipes/dsv41-flash-jovian-r37-4x/recipe.toml` (bisect log in the
|
|
|
+ args comments).
|
|
|
+- Test: greedy 300-word Russian generation, 3 samples above (temp 0.7/0.0).
|
|
|
+- Boot logs: node `/var/lib/docker/containers/*` (kept via `--keep` until the
|
|
|
+ outage).
|
