shift notes: qwen garble root cause — unpinned b12x image (nightly-20260925), forum 382733 #195-198; pinned to nightly-20260913

history/night-shifts/2026-09-26.md

@@ -252,3 +252,36 @@ reads it over SSH after the hooks and appends it as read-only binds. The
 container sees regular files only → O_NOFOLLOW is satisfied.
 Result: EXL3 2x boots (shards stream from the NAS, ~15 min), API live on
 dgx1:30000 as `deepseek-v41-flash-exl3`, smoke 127×89 + Paris correct.
+
+## Qwen garble hunt — root cause (2026-09-26 late)
+
+The intermittent Qwen garble (empty replies, degenerate text, special-token
+leakage in tool-call markup) was tracked all day. The trail:
+
+1. Caught live on the arithmetic monitor (fresh-context probes, 2-min cadence)
+   twice; caught in the user's pi session (80K+ tool-heavy context) three
+   times — `stopReason: stop` at 36-229 tokens (the model stopped itself
+   mid-degeneration). Engine logs at those moments: no warnings, no Xid, no
+   OOM.
+2. A/B matrix (all red herrings): marlin atomic-add 1 vs 0 (perf-neutral,
+   garble on both), MTP k=4 vs k=3 (garble on both), min config (no MTP, no
+   prefix cache — temporarily clean), GDM/display stack (was up on 88.43,
+   absent on 88.50 — garble on both), both boxes (garble on both).
+3. Label-inversion trap: the pi provider named "dgx5" pointed at dgx3.lan
+   (88.50) — the box actually serving the user's agent — while all my
+   testing/A/Bs went to the other box (88.43). Provider list now has
+   unambiguous `qwen-min` / `qwen-prod` entries.
+4. Root cause (forum 382733 posts 195-198, 0rand + azampatti): the same
+   corruption on an UNPINNED b12x image, fixed by the pinned digest. Our
+   recipe ran floating `eugr/spark-vllm-b12x:latest`, which we had pulled as
+   `nightly-20260925` (digest c9e22735). The validated pin
+   (azampatti repo config.env) is `nightly-20260913` (8e7e0621). So every
+   nightly from 09-14..09-25 is suspect; the 165/176 bench ran on the
+   suspect image (the 88-scenario pool just doesn't hit the degenerate tail).
+
+Fix: both Qwen boxes now serve the pinned digest (pulled node-side, ~25 GB
+each), recipe `docker_image` pinned with a comment (no floating tags),
+bench baseline params restored (MTP k=4, atomic-add 1). Launcher fix:
+`get_image_digest` now accepts digest refs (`docker image ls` silently
+returns nothing for `@sha256:` refs — would have made every pinned prebuilt
+recipe fail). Re-bench pending (label int4-auto-1x-pinned).