P1 verdict: lab-tree garbling trait (0913 A/B garbled like r37); bug report finalized; dsv41-flash-8x recipe (in-memory engram, GMU 0.91)

history/night-shifts/2026-09-18.md

@@ -179,8 +179,8 @@ Boot log (6 boots, 12:21 → 13:34):
 5. Teardown: stop, then the P1/P2 sequence.
 
 ### P1 — r37 typo verdict (closes the morning's bisect)
-STATUS: Boot D INFEASIBLE on 4x (OOM incident 14:37 UTC, nodes recovered);
-P1.2 eugr 0913 A/B running since 14:59 UTC.
+DONE (2026-09-18 night): **verdict = the garbling is a lab-tree trait.**
+Boot D INFEASIBLE on 4x (OOM incident 14:37 UTC, nodes recovered).
 1. **Boot D — INFEASIBLE ON 4X, OOM incident 14:37 UTC.** The r37 recipe
    state has no `--engram-config` (the DS4.1 engram table is part of the
    model weights, not a toggle) and no instanttensor/lazy → the full 510 GB
@@ -191,16 +191,23 @@ P1.2 eugr 0913 A/B running since 14:59 UTC.
    The engram-off hypothesis can only be tested on 8x (510/8 = 64 GB/node
    fits) — P2.3 is the natural test. Morning A/B/C (same tree, engram disk
    mode + lazy load) booted fine — the OOM is specific to the D state.
-2. **eugr b12x nightly-20260913 A/B (running since 14:59 UTC):** recipe
+2. **eugr b12x nightly-20260913 A/B (done 15:45 UTC):** recipe
    `dsv41-flash-eugr-0913-4x` = the measured r37 contract (GMU 0.88, 512K
    DCP1, block 256, B12X trio, K7 greedy/adaptive, engram disk,
    instanttensor/lazy, MXFP8_LM_HEAD=0) on the eugr 0913 image (the lab's
-   dev/infernal-invocation, newer than r37's 3fb88fdf tree; pre-pulled to
-   dgx1-4 during the P0 bench). Garbled → lab-tree trait (file with the
-   lab; r37 is not DS4.1 qualified); clean → r37-specific regression
-   (exact bisect for the lab).
-3. If both still garble: DS4.1 stays on tonyd2wild prod; R37 = GLM-lane
-   only (where it's proven).
+   dev/infernal-invocation, `b40673cd0`, newer than r37's 3fb88fdf tree).
+   Result: **GARBLED, same pattern** (300-word RU: ~30 close-vocabulary
+   corruptions in 2.4 KB — "абслуживание", "парамитры", "мадель", "тензары",
+   "маштабировать", "упровляемой"...; identifiers 6/6 PASS, structure clean).
+   Both independent 09-13 lab builds garble, the tonyd2wild prod chain is
+   clean, MXFP8-LM-head opted out in all runs → the defect is in code common
+   to the lab's DS4.1 line as of 09-13 (`deepseek_v32`/DS4.1 model code,
+   b12x MoE, or the engram disk-read path). DS4.1 stays on tonyd2wild prod;
+   the lab line stays proven only on the full GLM-5.3 lane (P0 quality gate
+   passed there). Bug report updated with the A/B section.
+   Note: the launcher's 1800 s readiness wait is too short for a cold NFS
+   load of 510 GB at ~130 MB/s (boot = ~40 min); the serve failed the wait
+   while the engine kept booting — probes went straight at the API.
 
 ### P2 — 8x trials (cables on site; nodes provisioned, cluster configured)
 Gate: CLEARED — all 8 nodes up + validated, `8nodes-switch` in config,

recipes/dsv41-flash-8x/recipe.toml

@@ -0,0 +1,91 @@
+# DeepSeek-V4.1-Flash (552B MoE, MXFP4 experts) — 8x DGX Spark, in-memory Engram.
+#
+# P2.3 of the 2026-09-18 night: the first 8x trial. Operator's design: at 8x
+# the Engram table fits in DRAM (406 GB / 8 = 50.7 GB/node) — no disk mode,
+# no per-node sparse staging, no /engram-local mounts. The engram rows load
+# from the NAS checkpoint into DRAM once at boot (one-off NFS cost), decode
+# reads them from memory (no per-token disk reads).
+#
+# Memory math per node (121.6 GiB): weights 307.3 GB / 8 = 38.4 GB + engram
+# rows 50.7 GB + process ~12 GB = ~101 GB. GMU 0.91 (the lab DS4.1 policy,
+# lil-catalog 09-15) leaves a ~9.5 GiB KV pool ≈ 1M tokens = ~1.9x 512K.
+#
+# Image: dsv41-flash:local (the tonyd2wild prod chain — engram.py patch
+# baked in; DSV41_ENGRAM_DISK simply NOT set here = memory mode). The
+# launcher syncs the image to dgx6-8 at serve time (dgx1-5 have it).
+#
+# This is also the natural "Boot D" test the 4x r37 bisect could not run:
+# engram WITHOUT the disk-read path. If the text is clean here while the r37
+# 4x disk-mode run garbles, the bisect narrows to the lab tree's engram
+# disk-read path.
+engine = "vllm"
+name = "DeepSeek-V4.1-Flash MXFP4 (dsv41 img, 8x, DSpark k=5, Engram in DRAM)"
+docker_image = "dsv41-flash:local"
+docker_image_dir = "dsv41-flash"
+model = "deepseek-ai/DeepSeek-V4.1-Flash"
+required_nodes = 8
+tensor_parallel_size = 8
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+# NO recipe entrypoint: the base image bakes ENTRYPOINT ["vllm","serve"]
+# (vllm/vllm-openai), so the launcher's CMD (model + flags) completes it.
+
+args = [
+    # Stable gateway alias — clients/gateways keep the DeepSeek-V4-Flash
+    # model name; no per-deployment config churn.
+    "--served-model-name", "deepseek-v4-flash",
+    "--trust-remote-code",
+    "--distributed-executor-backend", "mp",
+    "--gpu-memory-utilization", "0.91",
+    "--max-model-len", "524288",
+    "--max-num-seqs", "8",
+    "--max-num-batched-tokens", "8192",
+    # boot-4 fix: the V4 indexer backend refuses the smallest-listed block
+    "--block-size", "128",
+    # boot-10: exact capture sizes for DSpark k=5, SEQS=8 (multiples of 5
+    # and 6 up to 48 — every decode batch gets an exact FULL graph, no padding)
+    "--compilation-config", """
+'{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,10,12,15,18,20,24,25,30,35,36,40,42,48]}'""",
+    "--speculative-config", """
+'{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'""",
+    "--tool-call-parser", "deepseek_v41",
+    "--enable-auto-tool-choice",
+    "--reasoning-parser", "deepseek_v41",
+    "--default-chat-template-kwargs", """
+'{"thinking": true, "reasoning_effort": "max"}'""",
+]
+
+[env]
+HF_HUB_OFFLINE = "1"
+# Engram: MEMORY mode — DSV41_ENGRAM_DISK deliberately NOT set (the baked
+# engram.py patch defaults to "0" = in-DRAM table). At 8x the table fits:
+# 406 GB / 8 = 50.7 GB/node.
+# CUDA graphs: the Engram rows are staged in prepare_inputs (baked
+# model_state.py prestage patch) so the whole decode step is capturable
+VLLM_USE_BREAKABLE_CUDAGRAPH = "1"
+VLLM_HAS_FLASHINFER_CUBIN = "1"
+VLLM_USE_RUST_FRONTEND = "0"
+# runtime JIT wedge countermeasure (their boot-3 lesson): kernels are
+# prebuilt in the image, but any residual JIT must not OOM the box
+MAX_JOBS = "2"
+VLLM_ENGINE_READY_TIMEOUT_S = "3600"
+PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+FLASHINFER_CUDA_ARCH_LIST = "12.1a"
+OMP_NUM_THREADS = "2"
+
+# JIT caches persist per-node (self-namespace by engine version key; the
+# image prebakes the flashinfer 0.7.0rc1 SM120 kernels, first boots re-JIT
+# the rest)
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/flashinfer"
+container = "/root/.cache/flashinfer"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/torchinductor"
+container = "/root/.cache/torchinductor"
+
+[[extra_mounts]]
+host = "~/sm121-cache/jit-121f/vllm"
+container = "/root/.cache/vllm"

specs/bugs/BUG-2026-09-14-r37-ds41-text-garbling.md

@@ -1,6 +1,9 @@
 # BUG-2026-09-14: r37 tree garbles DeepSeek-V4.1-Flash text on 4x GB10
 
-Status: bisect A-C complete (flags excluded), D pending (cluster outage).
+Status: bisect A-C complete (flags excluded) + **A/B on the lab's newer
+09-13 build complete (2026-09-18 night): the garbling is a lab-tree trait,
+not an r37 regression**. Boot D (engram off) is infeasible on 4x — see
+Implication.
 
 ## Symptom
 
@@ -36,19 +39,42 @@ produces clean text (production of record, ~4 weeks of agent traffic).
 | A    | remove `--linear-backend b12x` | garbled |
 | B    | prod spec: K5 probabilistic/block, adaptive off (was K7 greedy/standard/adaptive) | garbled |
 | C    | FULL prod contract: GMU 0.85, seqs 8, tokens 8192, block 128, K5-tuned capture sizes, no B12X attention/linear backends, no instanttensor | garbled |
-| D    | C + engram disabled (no `--engram-config`) | **pending** (cluster outage mid-boot, 2026-09-14 ~07:27 UTC) |
+| D    | C + engram disabled (no `--engram-config`) | **infeasible on 4x** — without disk mode + lazy load the full 510 GB (incl. the 406 GB engram table) loads eagerly into DRAM: 127.6 GB/node > 121.6 GiB, all 4 nodes OOM'd (2026-09-18 14:37 UTC, power-cycled + recovered). The engram table is part of the model weights, not a toggle — the off-state only exists at 8x (64 GB/node). |
 
 Excluded along the way: runtime LM-head MXFP8 quantization
 (`VLLM_MXFP8_LM_HEAD=0` verified in-container, no "Quantizing LM head" log
 line, KV pool shrank 10.39M -> 10.19M tokens proving the BF16 head) — the
 typos persist with the BF16 head.
 
+## A/B: the lab's newer build (2026-09-18 night)
+
+`eugr/spark-vllm-b12x:nightly-20260913` (`v0.1.dev20759+gb40673cd0.d20260913`
+— the lab's `dev/infernal-invocation` branch, newer than r37's `3fb88fdf`)
+running the same measured contract (GMU 0.88, 512K DCP1, block 256, B12X
+trio, K7 greedy/adaptive, engram disk, instanttensor/lazy, `VLLM_MXFP8_LM_HEAD=0`)
+on the same 4 nodes / same NAS snapshot:
+
+- 300-word Russian probe: **garbled, same pattern** ("абслуживание",
+  "парамитры", "мадель", "тензары", "матречная аперация", "маштабировать",
+  "гаризантально", "упровляемой", "полнастью", "интелекта" — the
+  close-vocabulary single-letter corruption, ~30 errors in 2.4 KB).
+- Identifier probe: 6/6 exact (structure/code clean, as on r37).
+
+Both independent 09-13 lab builds (r37 vllmc687594b + 0913 b40673cd0) garble;
+the tonyd2wild prod chain (vllm/vllm-openai nightly + dsv41-feat@e47aa780 +
+boot-10 patches) is clean. The shared factor is the lab's DS4.1 model-code
+line as of 09-13 — not our build (aarch64/sm_121a deviation), not the
+contract, not the MXFP8 LM head (opted out in all runs).
+
 ## Implication
 
-The defect is in the r37 tree itself (not the contract flags): the default
-attention/dense path, the `deepseek_v32` model code, the b12x MoE kernels, or
-the engram disk-read implementation — or a combination. Boot D separates the
-engram hypothesis from the rest.
+The lab's DS4.1 line is not quality-qualified for agent text on this stack
+(as of the 09-13 builds). The defect sits in code common to both lab builds:
+the `deepseek_v32`/DS4.1 model code, the b12x MoE kernels, or the engram
+disk-read path. DS4.1 stays on the tonyd2wild prod chain of record; the lab
+line remains usable only where it is proven (the full GLM-5.3 lane, where
+r37/0913 output is clean — see the 2026-09-18 night P0: full GLM + DSpark
+K8, quality gate passed).
 
 ## Related (same tree, same week, already documented elsewhere)
 
@@ -61,7 +87,9 @@ engram hypothesis from the rest.
 ## Repro assets
 
 - Recipe: `recipes/dsv41-flash-jovian-r37-4x/recipe.toml` (bisect log in the
-  args comments).
-- Test: greedy 300-word Russian generation, 3 samples above (temp 0.7/0.0).
+  args comments); A/B recipe `recipes/dsv41-flash-eugr-0913-4x/recipe.toml`.
+- Test: greedy 300-word Russian generation, 3 samples above (temp 0.7/0.0)
+  + the 6-string TOML identifier probe (pass on all garbled builds).
 - Boot logs: node `/var/lib/docker/containers/*` (kept via `--keep` until the
-  outage).
+  outage); 0913 boot logs on dgx1, container
+  `sm121_dsv41-flash-eugr-0913-4x_0`.