|
|
@@ -190,3 +190,48 @@ recipe scripts).
|
|
|
restore back).
|
|
|
- In flight at this point: model download ~55/636 GB on the NAS, image
|
|
|
pull on dgx1.
|
|
|
+
|
|
|
+## Day shift (26 Sept, ~12:00–16:00 UTC+2) — Qwen garble investigation + EXL3
|
|
|
+
|
|
|
+### Qwen (`qwen38-flash-next-int4`, dgx5, port 30000, `qwen3.8`)
|
|
|
+User report: model works for a while, then emits random numbers/garbage;
|
|
|
+"appears around 30% of the context"; still happening at report time.
|
|
|
+
|
|
|
+Investigation (controlled repro attempts, all via API on 192.168.88.43:30000):
|
|
|
+- Short/medium context: clean (120-req stress 1/120 = 1 client timeout, 0 garble).
|
|
|
+- 60K multi-turn (12 blocks ~6K each, prefix-cached): 1 true garble in the
|
|
|
+ whole campaign (digit truncation 726→"72" etc.) — clustered in the first
|
|
|
+ 30 min of the 09:24 instance (10:35–10:37 UTC); **0 true garbles in
|
|
|
+ 400+ requests since** (both atomic-flag states, temp 0/0.7, streaming,
|
|
|
+ concurrency, tools, 123K session growth = 0/72 true fails).
|
|
|
+- `content=null` cases in early probes were my own artifact: `max_tokens`
|
|
|
+ (50–80) truncated the reasoning channel (105–145 tokens) before content.
|
|
|
+- Engine logs clean (no NaN/OOM/warnings; spec-decode acceptance normal
|
|
|
+ 4.3/4). dmesg: 3× NVRM NV_ERR_NO_MEMORY — all at container startups
|
|
|
+ (08:52, 09:24), none during operation; no Xid/ECC.
|
|
|
+- Hypothesis tested: `VLLM_MARLIN_USE_ATOMIC_ADD=1` (experimental Marlin
|
|
|
+ atomic-add path, n<2048/k≥2048 layers) — kernel analysis: slice-0 zeroes
|
|
|
+ the `torch::empty` output then sets a lock, other slices atomicAdd after
|
|
|
+ a lock wait; zeroing→lock write lacks a GPU memory fence → potential
|
|
|
+ race. A/B (restart each way): perf within noise (decode 47.5 vs 51.9
|
|
|
+ tok/s, conc1 339 vs 303, conc4 253 vs 195), quality 0/40 both states →
|
|
|
+ **not conclusively the cause**; flag left as the recipe shipped (1).
|
|
|
+- Hardware cross-check: Qwen replicated on **dgx3** (model pulled from
|
|
|
+ dgx5 over the fabric, image pushed node-to-node): 0/40 garble there too.
|
|
|
+ Inconclusive (bug too intermittent to catch in controlled runs).
|
|
|
+- Monitor running: `/tmp/qwen_monitor.py` (probe every 2 min →
|
|
|
+ /tmp/qwen-monitor.log) as a tripwire.
|
|
|
+
|
|
|
+Standing: if garble recurs, need **exact time + a sample** to correlate
|
|
|
+with engine log / dmesg. dgx3 kept up as hot spare (was off before).
|
|
|
+
|
|
|
+### Launcher bug found (not fixed)
|
|
|
+Prebuilt external image missing on the target node → `docker run` silently
|
|
|
+pulls from Docker Hub over the internet → the launcher's SSH command timeout
|
|
|
+kills the pull ("context canceled" in dockerd) but the serve still logs
|
|
|
+"✅ VLLM started" and waits 1800 s for an API that never comes. Needs a
|
|
|
+pre-run image presence check (or node-to-node sync) + honest error.
|
|
|
+
|
|
|
+### Config fix
|
|
|
+`config.toml` 1node-4 node entity was `switch.dgx3` (copy-paste) → fixed to
|
|
|
+`switch.dgx4` (the 1node-3 node keeps `switch.dgx3`).
|
