Home Explore Help
Sign In
sm121
/
launcher
2
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Tree: 4d7b927931
Branches Tags
launcher
/
history
/
night-shifts
/
2026-09-26.md

2026-09-26.md 17 KB
History Raw

Night shift 2026-09-26 (UTC)

Plan (user): DSV4.1-Flash base → NAS, stop MiMo Pro 8x, MiMo-V2.6-Flash-RL 2x (SGLang), time permitting Qwen Int4 1x / Aiden DSV4.1 4x, full power off at the end of the shift.

Timeline (UTC)

  • ~00:00 — user's hfrs download of deepseek-ai/DeepSeek-V4.1-Flash (510 GB) found already running on dgx1 (PID 677687); my duplicate hf process killed (lock-blocked). Final state on NAS: 476G, 48 safetensors, refs/main = dba1be0a40aa45a94ad051997016db3960a90277 (/mnt/nas/ai/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash).
  • MiMo Pro 8x stopped (sm121 stop --recipe mimo26-pro-rl-8x --cluster 8nodes-switch); nodes freed.
  • MiMo-V2.6-Flash-RL 2x (dgx1+2, mimo26-flash-rl-2x, mimo26-spark:a14, model staged on dgx1+2 local, 166 GB each): first boot failed ("SG17 RoCEnante requires the TP8 Gloo exchange group") → removed SGLANG8_ROCE_ALLREDUCE from the recipe env. Boot OK: KV 340,736 tokens, ctx 327680. Smoke OK (math, tools, needle 132K ×2, code C1 71.7-78.7 t/s vs vLLM ref 70.5). Bench: 161/176 (91/100, ★★★★★) — TC-74 (safety-critical) failed. Report recipes/mimo26-flash-rl-2x/reports/2026-09-25_231142_sglang-2x-first.*. Committed+pushed (5bd465a). Stopped to free dgx1+2.
  • Aiden DSV4.1-Flash 4x (dgx1-4, dsv41-flash-4x-sglang-aiden16): image aidendle94/sparkrun-sglang-dsv41-gb10:production-1.6 (33.3 GB) pulled on dgx4 → synced to dgx1-3. Full flag/env set extracted from the aidenle.com JS bundle.

Aiden 4x — boot saga (three launcher-side traps found)

  1. NGC entrypoint: image ENTRYPOINT = /opt/nvidia/nvidia_entrypoint.sh (banner + exec "$@") — args pass through, not the culprit.
  2. TOML ordering trap: my [[extra_mounts]] block landed before args = [...]; the bare args key silently parsed into the table and was dropped (the recipe served with no flags — server_args showed all defaults: ep_size 1, mem_fraction 0.885, no speculative). Fix: [[…]] headers after all bare top-level keys. (Already documented in AGENTS.md "TOML section order".)
  3. Engram per-layer row counts: the overlay requires the ranks' SPARK_ENGRAM_ROWS ranges to tile [0, N) exactly per layer — layer 1 has 384,006,168 rows, layer 14 has 384,016,682 (384016682/4 is not integer). The wrapper (repo: recipes/dsv41-flash-4x-sglang-aiden16/aiden4x-launch.sh, deployed to /home/kilork/aiden4x-launch.sh on dgx1-4) reads both N from the checkpoint's safetensors headers at launch and tiles floor/ceil.

Booted: 524K ctx, EP4+TP4, dsv4 attention, DSpark block 5, fp8 KV, KV pool 9.5M full tokens/node (Aiden's fleet: ~4.2M). Smoke: count-20 exact, tools OK, 155K-token needle OK.

Aiden 4x bench + the perf anomaly (important)

  • Quality: 162/176 (92/100, ★★★★★) — best DSV4.1 score on this cluster (vLLM r36/r37 4x: ~155-159). Fails TC-68/74/88; partials TC-03/14/50/51/ 57/58/62/81.
  • Perf table looked broken: llama-benchy pp 154-359 t/s, tg 8.6-21.8 t/s, TTFT 10-51 s — while manual requests on the same server ran 2,400-3,000 t/s prefill. Investigation:
    • return_token_ids=true + stream=true → HTTP 400 on this SGLang build (sgl-project/sglang#30917). tool-eval-bench already works around it for sglang backends (--extra-body return_token_ids=false), so the bench's requests were valid.
    • A/B: first request of a session TTFT 10.4 s; repeats of the same prefix 0.3 s; every fresh (non-prefix-cached) context 2-10 s slower with a decaying tail (5.0 → 3.6 → 2.6 → 2.1 s for identical 2.7K fresh prompts).
    • Container engram stats during the fresh prompts: 100-140 lookups, mean gather 29-35 ms, page cache "peak 4.00 GB … 1 releases (4.0 GB released)" — the engram n-gram rows (layers 1+14, ~384M rows each) are served from the NAS shards via libengram_rows.so; a fresh context touches thousands of new rows, each a random NFS read. Prefix-cached repeats touch cached rows → fast.
    • Control: the Qwen Int4 bench (same harness, no engram model) measured normal numbers (pp 1,886-2,103 t/s, tg 66-100 t/s) — the harness and the Mac side are not the problem.
  • Conclusion: not an engine bug. The engram host-table design pays a fresh-context penalty of ~2-10 s when the rows live on the NAS (our fallback; Aiden's production uses 47.2 GiB node-local sparse copies via the private engram_local.py). Agent traffic (short, repeated system prompts) sits in the cached regime and is fast; long fresh-context loads are slow. The committed perf table for this lane reflects the fresh-context penalty. If the lane becomes production, either accept the penalty or obtain Aiden's local-copy setup.
  • Perf-only re-run (--perf-only, label aiden16-perf-rerun): 189 t/s d0 c1 (same pattern), confirms it's systematic, not a one-off.

Qwen3.8-Flash-Next-Int4-AutoRound 1x (dgx5, qwen38-flash-next-int4)

  • Model: azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound (122 GB, 123 files) downloaded on dgx5 (the node's hf CLI was missing → uv tool run hf — first attempt lost, hf: command not found).
  • Legacy cache layout: the hf download on this node wrote /home/kilork/hf-local/models--… (top level, no hub/). The mod expects hub/models--…/refs/main → added a relative symlink hub/models--… → ../models--…. An absolute symlink dangles inside the container (the mount root differs: host /home/kilork/hf-local vs container /root/.cache/huggingface).
  • Mod vendored from azampatti's repo (recipes/qwen38-flash-next-int4/mods/ flashnext-int4-b12x/: run.sh, build_views.py, patch_b12x.py, vllm_fp8_hybrid.py) — launcher uploads it per node and mounts /mods:ro; it checks the snapshot, builds /workspace/flashnext/{model,draft-k10} (symlink views + small JSON, zero copy) and patches the container's vLLM (fp8-hybrid, int4-head, index-filter, draft-scale, fla fixes).
  • Two-positional-model trap: the launcher emits the recipe model as the positional arg; with model = "<repo id>" + args[0] = "/workspace/…" vLLM got two positionals (unrecognized arguments: /workspace/…). Fix: model = "" (the launcher skips the positional) + the view in args.
  • Serve: 1x Spark (cluster 1node-5), eugr/spark-vllm-b12x, MEDIUM-thinking chat template (his default; the speculative head is trained on it).
  • Smoke: math OK, tools OK, 45K-token needle OK, decode C1 75.9 t/s (his claim 60-70), prefill 34K/18 s (~1.9K t/s).
  • Bench: 165/176 (94/100, ★★★★★) — best score of the night (single node!). Perf: pp 1,886-2,103 t/s, tg 66-100 t/s (c1/c2), median turn 2.2 s. 2 safety-critical flags: TC-51 (batched send_email with create_calendar_event instead of waiting), TC-82 (sent the checklist without a confirmed recipient). The MEDIUM vs xhigh thinking level is the known A/B fairness caveat (his quant defaults to MEDIUM; the cluster anchors run xhigh).

State at shift end

  • DSV4.1-Flash base on the NAS: done (476G, 48 shards, refs/main dba1be0a).
  • Aiden 4x + Qwen 1x benches landed in their recipe reports/.
  • EXL3 2x (task 33, coolbho3k 3bpw + the 2 GB headless-RAM unlock): NOT done — out of time for the shift; the base model on the NAS unblocks it (the quant is smaller than 510 GB).
  • All serves stopped, cluster powered off fully (user's explicit instruction for the end of the shift).

Ops notes (new)

  • uv tool run hf on a node without the hf CLI works (installs huggingface_hub in an ephemeral env); a plain hf in a nohup bash -c is not on PATH → "command not found", and the download "completes" instantly with 0 bytes (check the log, not just the .done file — this one wrote the .done from a failed wrapper).
  • hf download --cache-dir <dir> on this node's huggingface_hub version creates a legacy layout (no hub/ subfolder) — plan the symlink accordingly; keep container-visible symlinks relative.
  • SGLang build in Aiden production-1.6: return_token_ids + streaming is a hard 400 (was ignored before sgl-project/sglang#30917); tool-eval-bench v2.7.0 handles it.
  • A new llama-benchy perf table that is 10×+ below the rest of the fleet's numbers: suspect a per-request cold path on the server (check prefix-cache behavior + server-side stats), not the harness.

Second half of the shift (post power-off, ~06:00+)

User re-engaged: finish the two recipes (Aiden + Qwen) with complete results, then the new EXL3 2x task (temporary 2 GB headless-RAM unlock via recipe scripts).

  • Aiden recipe TOML fixed — the header-comment edit had dropped the # on line 29 → tomllib Expected '=' after a key. Repaired, validates (38 args / 40 env).
  • Perf table missing from .md reports root-caused: tool-eval-bench v2.7.0 --json-file implies --json → _run_json dispatch path drops throughput_samples → the "## Throughput Metrics" section is absent from the md (and the perf data from the JSON); the table survives only in the .log. Fixed launcher-side: src/bench_report.rs (ensure_throughput_section) re-attaches the llama-benchy table parsed from the bench .log into the copied .md (harness pin untouched). Retroactively applied to the Aiden …_015216 and Qwen …_022931 reports. Commit 4546237.
  • EXL3 2x recipe (dsv41-flash-exl3-2x, coolbho3k rc2 image ghcr.io/coolbho3k/deepseek-v4.1-flash-exl3-3bpw-2x-dgx-spark:20260917-rc2):
    • User started the model download on dgx5 local — 636 GB does not fit (251 G free) → killed, redirected to the NAS (/mnt/nas/ai/huggingface/hub/…, hub layout). The 2x pair serves from the NAS; drafter overlay coolbho3k/DeepSeek-V4.1-Flash-DSpark-EXL3-3bpw downloaded too (small, done first).
    • Runtime requirements extracted from the pinned repo (commit 878e0ee): model must be mounted at exactly /model (drafter in /model/draft, engrams in /model/engrams), drafter overlay at DS41_DRAFT_EXL3_PATH=/draft-exl3, engine call serve.py serve /model --config conservative.yaml --worker-cls combined_worker.CombinedWorker --disable-custom-all-reduce …, hard serving cgroup guard (9 GiB / no-swap / ≤6 CPU), 2 GB display reserve (nvidia_drm modeset=1 fbdev=0) for the 1.75 GiB display-pool KV.
    • Launcher extensions (all committed): per-recipe devices = [...] (--device d:d), memory_limit/cpus (→ --memory/--memory-swap/--cpus), built-in scripts unlock_display_ram / restore_display_ram (temporary per-boot remod, no UEFI/modprobe.d/initramfs changes; restore on stop), --cluster auto now honors recipe.cluster on the docker path (tmux path already did — parity fix).
    • Recipe: NAS hub cache + per-node before_serve symlinks (/home/kilork/exl3-model → current refs/main snapshot) bound ro at /model / /draft-exl3; 41 DS41/NCCL envs; 200G pair dgx1+dgx2 (powered on for this), 1M ctx, GMU 0.92, C2.
    • Unlock/restore scripts validated live on dgx2 (default state was modeset=N fbdev=Y → remod → modeset=Y fbdev=N + /dev/dri/card0 → restore back).
    • In flight at this point: model download ~55/636 GB on the NAS, image pull on dgx1.

Day shift (26 Sept, ~12:00–16:00 UTC+2) — Qwen garble investigation + EXL3

Qwen (qwen38-flash-next-int4, dgx5, port 30000, qwen3.8)

User report: model works for a while, then emits random numbers/garbage; "appears around 30% of the context"; still happening at report time.

Investigation (controlled repro attempts, all via API on 192.168.88.43:30000):

  • Short/medium context: clean (120-req stress 1/120 = 1 client timeout, 0 garble).
  • 60K multi-turn (12 blocks ~6K each, prefix-cached): 1 true garble in the whole campaign (digit truncation 726→"72" etc.) — clustered in the first 30 min of the 09:24 instance (10:35–10:37 UTC); 0 true garbles in 400+ requests since (both atomic-flag states, temp 0/0.7, streaming, concurrency, tools, 123K session growth = 0/72 true fails).
  • content=null cases in early probes were my own artifact: max_tokens (50–80) truncated the reasoning channel (105–145 tokens) before content.
  • Engine logs clean (no NaN/OOM/warnings; spec-decode acceptance normal 4.3/4). dmesg: 3× NVRM NV_ERR_NO_MEMORY — all at container startups (08:52, 09:24), none during operation; no Xid/ECC.
  • Hypothesis tested: VLLM_MARLIN_USE_ATOMIC_ADD=1 (experimental Marlin atomic-add path, n<2048/k≥2048 layers) — kernel analysis: slice-0 zeroes the torch::empty output then sets a lock, other slices atomicAdd after a lock wait; zeroing→lock write lacks a GPU memory fence → potential race. A/B (restart each way): perf within noise (decode 47.5 vs 51.9 tok/s, conc1 339 vs 303, conc4 253 vs 195), quality 0/40 both states → not conclusively the cause; flag left as the recipe shipped (1).
  • Hardware cross-check: Qwen replicated on dgx3 (model pulled from dgx5 over the fabric, image pushed node-to-node): 0/40 garble there too. Inconclusive (bug too intermittent to catch in controlled runs).
  • Monitor running: /tmp/qwen_monitor.py (probe every 2 min → /tmp/qwen-monitor.log) as a tripwire.

Standing: if garble recurs, need exact time + a sample to correlate with engine log / dmesg. dgx3 kept up as hot spare (was off before).

Launcher bug found (not fixed)

Prebuilt external image missing on the target node → docker run silently pulls from Docker Hub over the internet → the launcher's SSH command timeout kills the pull ("context canceled" in dockerd) but the serve still logs "✅ VLLM started" and waits 1800 s for an API that never comes. Needs a pre-run image presence check (or node-to-node sync) + honest error.

Config fix

config.toml 1node-4 node entity was switch.dgx3 (copy-paste) → fixed to switch.dgx4 (the 1node-3 node keeps switch.dgx3).

EXL3 2x — ELOOP root cause + fix (serving)

Root cause (after ~10 red herrings: NFS dentry races, concurrency, cgroups, launcher-flag diffs — all disproven by flag-for-flag replicas): the DS41 runtime opens the shard files with O_NOFOLLOW (ssd_vocab_rows.py:68 os.open(self.path, O_RDONLY|O_CLOEXEC|O_NOFOLLOW)). With symlink-based staging, the final path component IS a symlink → open(2) returns ELOOP ("Too many levels of symbolic links") deterministically — the kernel's ELOOP for O_NOFOLLOW, not a loop at all. Fix: no symlinks in the container. New recipe field extra_mounts_manifest = "/tmp/sm121-exl3-mounts.txt": the before-serve hook generates a per-node host<TAB>container list (one bind per real file, 161 model + 10 draft entries, resolved to the final hub blob), the launcher reads it over SSH after the hooks and appends it as read-only binds. The container sees regular files only → O_NOFOLLOW is satisfied. Result: EXL3 2x boots (shards stream from the NAS, ~15 min), API live on dgx1:30000 as deepseek-v41-flash-exl3, smoke 127×89 + Paris correct.

Qwen garble hunt — root cause (2026-09-26 late)

The intermittent Qwen garble (empty replies, degenerate text, special-token leakage in tool-call markup) was tracked all day. The trail:

  1. Caught live on the arithmetic monitor (fresh-context probes, 2-min cadence) twice; caught in the user's pi session (80K+ tool-heavy context) three times — stopReason: stop at 36-229 tokens (the model stopped itself mid-degeneration). Engine logs at those moments: no warnings, no Xid, no OOM.
  2. A/B matrix (all red herrings): marlin atomic-add 1 vs 0 (perf-neutral, garble on both), MTP k=4 vs k=3 (garble on both), min config (no MTP, no prefix cache — temporarily clean), GDM/display stack (was up on 88.43, absent on 88.50 — garble on both), both boxes (garble on both).
  3. Label-inversion trap: the pi provider named "dgx5" pointed at dgx3.lan (88.50) — the box actually serving the user's agent — while all my testing/A/Bs went to the other box (88.43). Provider list now has unambiguous qwen-min / qwen-prod entries.
  4. Root cause (forum 382733 posts 195-198, 0rand + azampatti): the same corruption on an UNPINNED b12x image, fixed by the pinned digest. Our recipe ran floating eugr/spark-vllm-b12x:latest, which we had pulled as nightly-20260925 (digest c9e22735). The validated pin (azampatti repo config.env) is nightly-20260913 (8e7e0621). So every nightly from 09-14..09-25 is suspect; the 165/176 bench ran on the suspect image (the 88-scenario pool just doesn't hit the degenerate tail).

Fix: both Qwen boxes now serve the pinned digest (pulled node-side, ~25 GB each), recipe docker_image pinned with a comment (no floating tags), bench baseline params restored (MTP k=4, atomic-add 1). Launcher fix: get_image_digest now accepts digest refs (docker image ls silently returns nothing for @sha256: refs — would have made every pinned prebuilt recipe fail). Re-bench pending (label int4-auto-1x-pinned).