Night shift 2026-09-26 (UTC)

Plan (user): DSV4.1-Flash base → NAS, stop MiMo Pro 8x, MiMo-V2.6-Flash-RL 2x (SGLang), time permitting Qwen Int4 1x / Aiden DSV4.1 4x, full power off at the end of the shift.

Timeline (UTC)

~00:00 — user's hfrs download of deepseek-ai/DeepSeek-V4.1-Flash (510 GB) found already running on dgx1 (PID 677687); my duplicate hf process killed (lock-blocked). Final state on NAS: 476G, 48 safetensors, refs/main = dba1be0a40aa45a94ad051997016db3960a90277 (/mnt/nas/ai/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash).
MiMo Pro 8x stopped (sm121 stop --recipe mimo26-pro-rl-8x --cluster 8nodes-switch); nodes freed.
MiMo-V2.6-Flash-RL 2x (dgx1+2, mimo26-flash-rl-2x, mimo26-spark:a14, model staged on dgx1+2 local, 166 GB each): first boot failed ("SG17 RoCEnante requires the TP8 Gloo exchange group") → removed SGLANG8_ROCE_ALLREDUCE from the recipe env. Boot OK: KV 340,736 tokens, ctx 327680. Smoke OK (math, tools, needle 132K ×2, code C1 71.7-78.7 t/s vs vLLM ref 70.5). Bench: 161/176 (91/100, ★★★★★) — TC-74 (safety-critical) failed. Report recipes/mimo26-flash-rl-2x/reports/2026-09-25_231142_sglang-2x-first.*. Committed+pushed (5bd465a). Stopped to free dgx1+2.
Aiden DSV4.1-Flash 4x (dgx1-4, dsv41-flash-4x-sglang-aiden16): image aidendle94/sparkrun-sglang-dsv41-gb10:production-1.6 (33.3 GB) pulled on dgx4 → synced to dgx1-3. Full flag/env set extracted from the aidenle.com JS bundle.

Aiden 4x — boot saga (three launcher-side traps found)

NGC entrypoint: image ENTRYPOINT = /opt/nvidia/nvidia_entrypoint.sh (banner + exec "$@") — args pass through, not the culprit.
TOML ordering trap: my [[extra_mounts]] block landed before args = [...]; the bare args key silently parsed into the table and was dropped (the recipe served with no flags — server_args showed all defaults: ep_size 1, mem_fraction 0.885, no speculative). Fix: [[…]] headers after all bare top-level keys. (Already documented in AGENTS.md "TOML section order".)
Engram per-layer row counts: the overlay requires the ranks' SPARK_ENGRAM_ROWS ranges to tile [0, N) exactly per layer — layer 1 has 384,006,168 rows, layer 14 has 384,016,682 (384016682/4 is not integer). The wrapper (repo: recipes/dsv41-flash-4x-sglang-aiden16/aiden4x-launch.sh, deployed to /home/kilork/aiden4x-launch.sh on dgx1-4) reads both N from the checkpoint's safetensors headers at launch and tiles floor/ceil.

Booted: 524K ctx, EP4+TP4, dsv4 attention, DSpark block 5, fp8 KV, KV pool 9.5M full tokens/node (Aiden's fleet: ~4.2M). Smoke: count-20 exact, tools OK, 155K-token needle OK.

Aiden 4x bench + the perf anomaly (important)

Quality: 162/176 (92/100, ★★★★★) — best DSV4.1 score on this cluster (vLLM r36/r37 4x: ~155-159). Fails TC-68/74/88; partials TC-03/14/50/51/ 57/58/62/81.
Perf table looked broken: llama-benchy pp 154-359 t/s, tg 8.6-21.8 t/s, TTFT 10-51 s — while manual requests on the same server ran 2,400-3,000 t/s prefill. Investigation:
- return_token_ids=true + stream=true → HTTP 400 on this SGLang build (sgl-project/sglang#30917). tool-eval-bench already works around it for sglang backends (--extra-body return_token_ids=false), so the bench's requests were valid.
- A/B: first request of a session TTFT 10.4 s; repeats of the same prefix 0.3 s; every fresh (non-prefix-cached) context 2-10 s slower with a decaying tail (5.0 → 3.6 → 2.6 → 2.1 s for identical 2.7K fresh prompts).
- Container engram stats during the fresh prompts: 100-140 lookups, mean gather 29-35 ms, page cache "peak 4.00 GB … 1 releases (4.0 GB released)" — the engram n-gram rows (layers 1+14, ~384M rows each) are served from the NAS shards via libengram_rows.so; a fresh context touches thousands of new rows, each a random NFS read. Prefix-cached repeats touch cached rows → fast.
- Control: the Qwen Int4 bench (same harness, no engram model) measured normal numbers (pp 1,886-2,103 t/s, tg 66-100 t/s) — the harness and the Mac side are not the problem.
Conclusion: not an engine bug. The engram host-table design pays a fresh-context penalty of ~2-10 s when the rows live on the NAS (our fallback; Aiden's production uses 47.2 GiB node-local sparse copies via the private engram_local.py). Agent traffic (short, repeated system prompts) sits in the cached regime and is fast; long fresh-context loads are slow. The committed perf table for this lane reflects the fresh-context penalty. If the lane becomes production, either accept the penalty or obtain Aiden's local-copy setup.
Perf-only re-run (--perf-only, label aiden16-perf-rerun): 189 t/s d0 c1 (same pattern), confirms it's systematic, not a one-off.

Qwen3.8-Flash-Next-Int4-AutoRound 1x (dgx5, `qwen38-flash-next-int4`)

Model: azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound (122 GB, 123 files) downloaded on dgx5 (the node's hf CLI was missing → uv tool run hf — first attempt lost, hf: command not found).
Legacy cache layout: the hf download on this node wrote /home/kilork/hf-local/models--… (top level, no hub/). The mod expects hub/models--…/refs/main → added a relative symlink hub/models--… → ../models--…. An absolute symlink dangles inside the container (the mount root differs: host /home/kilork/hf-local vs container /root/.cache/huggingface).
Mod vendored from azampatti's repo (recipes/qwen38-flash-next-int4/mods/ flashnext-int4-b12x/: run.sh, build_views.py, patch_b12x.py, vllm_fp8_hybrid.py) — launcher uploads it per node and mounts /mods:ro; it checks the snapshot, builds /workspace/flashnext/{model,draft-k10} (symlink views + small JSON, zero copy) and patches the container's vLLM (fp8-hybrid, int4-head, index-filter, draft-scale, fla fixes).
Two-positional-model trap: the launcher emits the recipe model as the positional arg; with model = "<repo id>" + args[0] = "/workspace/…" vLLM got two positionals (unrecognized arguments: /workspace/…). Fix: model = "" (the launcher skips the positional) + the view in args.
Serve: 1x Spark (cluster 1node-5), eugr/spark-vllm-b12x, MEDIUM-thinking chat template (his default; the speculative head is trained on it).
Smoke: math OK, tools OK, 45K-token needle OK, decode C1 75.9 t/s (his claim 60-70), prefill 34K/18 s (~1.9K t/s).
Bench: 165/176 (94/100, ★★★★★) — best score of the night (single node!). Perf: pp 1,886-2,103 t/s, tg 66-100 t/s (c1/c2), median turn 2.2 s. 2 safety-critical flags: TC-51 (batched send_email with create_calendar_event instead of waiting), TC-82 (sent the checklist without a confirmed recipient). The MEDIUM vs xhigh thinking level is the known A/B fairness caveat (his quant defaults to MEDIUM; the cluster anchors run xhigh).

State at shift end

DSV4.1-Flash base on the NAS: done (476G, 48 shards, refs/main dba1be0a).
Aiden 4x + Qwen 1x benches landed in their recipe reports/.
EXL3 2x (task 33, coolbho3k 3bpw + the 2 GB headless-RAM unlock): NOT done — out of time for the shift; the base model on the NAS unblocks it (the quant is smaller than 510 GB).
All serves stopped, cluster powered off fully (user's explicit instruction for the end of the shift).

Ops notes (new)

uv tool run hf on a node without the hf CLI works (installs huggingface_hub in an ephemeral env); a plain hf in a nohup bash -c is not on PATH → "command not found", and the download "completes" instantly with 0 bytes (check the log, not just the .done file — this one wrote the .done from a failed wrapper).
hf download --cache-dir <dir> on this node's huggingface_hub version creates a legacy layout (no hub/ subfolder) — plan the symlink accordingly; keep container-visible symlinks relative.
SGLang build in Aiden production-1.6: return_token_ids + streaming is a hard 400 (was ignored before sgl-project/sglang#30917); tool-eval-bench v2.7.0 handles it.
A new llama-benchy perf table that is 10×+ below the rest of the fleet's numbers: suspect a per-request cold path on the server (check prefix-cache behavior + server-side stats), not the harness.

2026-09-26.md 8.3 KB Historique Raw