|
|
@@ -252,3 +252,36 @@ reads it over SSH after the hooks and appends it as read-only binds. The
|
|
|
container sees regular files only → O_NOFOLLOW is satisfied.
|
|
|
Result: EXL3 2x boots (shards stream from the NAS, ~15 min), API live on
|
|
|
dgx1:30000 as `deepseek-v41-flash-exl3`, smoke 127×89 + Paris correct.
|
|
|
+
|
|
|
+## Qwen garble hunt — root cause (2026-09-26 late)
|
|
|
+
|
|
|
+The intermittent Qwen garble (empty replies, degenerate text, special-token
|
|
|
+leakage in tool-call markup) was tracked all day. The trail:
|
|
|
+
|
|
|
+1. Caught live on the arithmetic monitor (fresh-context probes, 2-min cadence)
|
|
|
+ twice; caught in the user's pi session (80K+ tool-heavy context) three
|
|
|
+ times — `stopReason: stop` at 36-229 tokens (the model stopped itself
|
|
|
+ mid-degeneration). Engine logs at those moments: no warnings, no Xid, no
|
|
|
+ OOM.
|
|
|
+2. A/B matrix (all red herrings): marlin atomic-add 1 vs 0 (perf-neutral,
|
|
|
+ garble on both), MTP k=4 vs k=3 (garble on both), min config (no MTP, no
|
|
|
+ prefix cache — temporarily clean), GDM/display stack (was up on 88.43,
|
|
|
+ absent on 88.50 — garble on both), both boxes (garble on both).
|
|
|
+3. Label-inversion trap: the pi provider named "dgx5" pointed at dgx3.lan
|
|
|
+ (88.50) — the box actually serving the user's agent — while all my
|
|
|
+ testing/A/Bs went to the other box (88.43). Provider list now has
|
|
|
+ unambiguous `qwen-min` / `qwen-prod` entries.
|
|
|
+4. Root cause (forum 382733 posts 195-198, 0rand + azampatti): the same
|
|
|
+ corruption on an UNPINNED b12x image, fixed by the pinned digest. Our
|
|
|
+ recipe ran floating `eugr/spark-vllm-b12x:latest`, which we had pulled as
|
|
|
+ `nightly-20260925` (digest c9e22735). The validated pin
|
|
|
+ (azampatti repo config.env) is `nightly-20260913` (8e7e0621). So every
|
|
|
+ nightly from 09-14..09-25 is suspect; the 165/176 bench ran on the
|
|
|
+ suspect image (the 88-scenario pool just doesn't hit the degenerate tail).
|
|
|
+
|
|
|
+Fix: both Qwen boxes now serve the pinned digest (pulled node-side, ~25 GB
|
|
|
+each), recipe `docker_image` pinned with a comment (no floating tags),
|
|
|
+bench baseline params restored (MTP k=4, atomic-add 1). Launcher fix:
|
|
|
+`get_image_digest` now accepts digest refs (`docker image ls` silently
|
|
|
+returns nothing for `@sha256:` refs — would have made every pinned prebuilt
|
|
|
+recipe fail). Re-bench pending (label int4-auto-1x-pinned).
|
