# Night shift 2026-09-26 (UTC)

Plan (user): DSV4.1-Flash base → NAS, stop MiMo Pro 8x, MiMo-V2.6-Flash-RL 2x
(SGLang), time permitting Qwen Int4 1x / Aiden DSV4.1 4x, **full power off at
the end of the shift**.

## Timeline (UTC)

- ~00:00 — user's `hfrs download` of `deepseek-ai/DeepSeek-V4.1-Flash` (510 GB)
  found already running on dgx1 (PID 677687); my duplicate `hf` process killed
  (lock-blocked). Final state on NAS: 476G, 48 safetensors,
  `refs/main = dba1be0a40aa45a94ad051997016db3960a90277`
  (`/mnt/nas/ai/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash`).
- MiMo Pro 8x stopped (`sm121 stop --recipe mimo26-pro-rl-8x --cluster
  8nodes-switch`); nodes freed.
- MiMo-V2.6-Flash-RL 2x (dgx1+2, `mimo26-flash-rl-2x`, `mimo26-spark:a14`,
  model staged on dgx1+2 local, 166 GB each): first boot failed
  ("SG17 RoCEnante requires the TP8 Gloo exchange group") → removed
  `SGLANG8_ROCE_ALLREDUCE` from the recipe env. Boot OK: KV 340,736 tokens,
  ctx 327680. Smoke OK (math, tools, needle 132K ×2, code C1 71.7-78.7 t/s vs
  vLLM ref 70.5). **Bench: 161/176 (91/100, ★★★★★)** — TC-74 (safety-critical)
  failed. Report `recipes/mimo26-flash-rl-2x/reports/2026-09-25_231142_sglang-2x-first.*`.
  Committed+pushed (5bd465a). Stopped to free dgx1+2.
- Aiden DSV4.1-Flash 4x (dgx1-4, `dsv41-flash-4x-sglang-aiden16`): image
  `aidendle94/sparkrun-sglang-dsv41-gb10:production-1.6` (33.3 GB) pulled on
  dgx4 → synced to dgx1-3. Full flag/env set extracted from the aidenle.com JS
  bundle.

## Aiden 4x — boot saga (three launcher-side traps found)

1. **NGC entrypoint**: image ENTRYPOINT = `/opt/nvidia/nvidia_entrypoint.sh`
   (banner + `exec "$@"`) — args pass through, not the culprit.
2. **TOML ordering trap**: my `[[extra_mounts]]` block landed *before*
   `args = [...]`; the bare `args` key silently parsed into the table and was
   dropped (the recipe served with no flags — server_args showed all
   defaults: ep_size 1, mem_fraction 0.885, no speculative). Fix: `[[…]]`
   headers after all bare top-level keys. (Already documented in AGENTS.md
   "TOML section order".)
3. **Engram per-layer row counts**: the overlay requires the ranks'
   `SPARK_ENGRAM_ROWS` ranges to tile `[0, N)` exactly per layer — layer 1 has
   384,006,168 rows, layer 14 has 384,016,682 (384016682/4 is not integer).
   The wrapper (repo: `recipes/dsv41-flash-4x-sglang-aiden16/aiden4x-launch.sh`,
   deployed to `/home/kilork/aiden4x-launch.sh` on dgx1-4) reads both N from
   the checkpoint's safetensors headers at launch and tiles floor/ceil.

Booted: 524K ctx, EP4+TP4, dsv4 attention, DSpark block 5, fp8 KV, KV pool
**9.5M full tokens/node** (Aiden's fleet: ~4.2M). Smoke: count-20 exact, tools
OK, 155K-token needle OK.

## Aiden 4x bench + the perf anomaly (important)

- **Quality: 162/176 (92/100, ★★★★★)** — best DSV4.1 score on this cluster
  (vLLM r36/r37 4x: ~155-159). Fails TC-68/74/88; partials TC-03/14/50/51/
  57/58/62/81.
- **Perf table looked broken**: llama-benchy pp 154-359 t/s, tg 8.6-21.8 t/s,
  TTFT 10-51 s — while manual requests on the same server ran 2,400-3,000 t/s
  prefill. Investigation:
  - `return_token_ids=true` + `stream=true` → HTTP 400 on this SGLang build
    (sgl-project/sglang#30917). tool-eval-bench already works around it for
    sglang backends (`--extra-body return_token_ids=false`), so the bench's
    requests were valid.
  - A/B: first request of a session TTFT 10.4 s; repeats of the same prefix
    0.3 s; **every fresh (non-prefix-cached) context 2-10 s slower** with a
    decaying tail (5.0 → 3.6 → 2.6 → 2.1 s for identical 2.7K fresh prompts).
  - Container engram stats during the fresh prompts: 100-140 lookups,
    **mean gather 29-35 ms**, page cache "peak 4.00 GB … 1 releases (4.0 GB
    released)" — the engram n-gram rows (layers 1+14, ~384M rows each) are
    served from the NAS shards via `libengram_rows.so`; a fresh context
    touches thousands of new rows, each a random NFS read. Prefix-cached
    repeats touch cached rows → fast.
  - Control: the Qwen Int4 bench (same harness, no engram model) measured
    normal numbers (pp 1,886-2,103 t/s, tg 66-100 t/s) — the harness and the
    Mac side are not the problem.
- **Conclusion**: not an engine bug. The engram host-table design pays a
  fresh-context penalty of ~2-10 s when the rows live on the NAS (our
  fallback; Aiden's production uses 47.2 GiB node-local sparse copies via the
  private `engram_local.py`). Agent traffic (short, repeated system prompts)
  sits in the cached regime and is fast; long fresh-context loads are slow.
  The committed perf table for this lane reflects the fresh-context penalty.
  If the lane becomes production, either accept the penalty or obtain
  Aiden's local-copy setup.
- Perf-only re-run (`--perf-only`, label `aiden16-perf-rerun`): 189 t/s d0
  c1 (same pattern), confirms it's systematic, not a one-off.

## Qwen3.8-Flash-Next-Int4-AutoRound 1x (dgx5, `qwen38-flash-next-int4`)

- Model: `azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound` (122 GB,
  123 files) downloaded on dgx5 (the node's `hf` CLI was missing →
  `uv tool run hf` — first attempt lost, `hf: command not found`).
- **Legacy cache layout**: the `hf` download on this node wrote
  `/home/kilork/hf-local/models--…` (top level, no `hub/`). The mod expects
  `hub/models--…/refs/main` → added a **relative** symlink
  `hub/models--… → ../models--…`. An absolute symlink dangles inside the
  container (the mount root differs: host `/home/kilork/hf-local` vs
  container `/root/.cache/huggingface`).
- Mod vendored from azampatti's repo (`recipes/qwen38-flash-next-int4/mods/
  flashnext-int4-b12x/`: run.sh, build_views.py, patch_b12x.py,
  vllm_fp8_hybrid.py) — launcher uploads it per node and mounts `/mods:ro`;
  it checks the snapshot, builds `/workspace/flashnext/{model,draft-k10}`
  (symlink views + small JSON, zero copy) and patches the container's vLLM
  (fp8-hybrid, int4-head, index-filter, draft-scale, fla fixes).
- **Two-positional-model trap**: the launcher emits the recipe `model` as the
  positional arg; with `model = "<repo id>"` + `args[0] = "/workspace/…"`
  vLLM got two positionals (`unrecognized arguments: /workspace/…`). Fix:
  `model = ""` (the launcher skips the positional) + the view in args.
- Serve: 1x Spark (cluster 1node-5), eugr/spark-vllm-b12x, MEDIUM-thinking
  chat template (his default; the speculative head is trained on it).
- Smoke: math OK, tools OK, 45K-token needle OK, **decode C1 75.9 t/s**
  (his claim 60-70), prefill 34K/18 s (~1.9K t/s).
- **Bench: 165/176 (94/100, ★★★★★)** — best score of the night (single node!).
  Perf: pp 1,886-2,103 t/s, tg 66-100 t/s (c1/c2), median turn 2.2 s.
  **2 safety-critical flags**: TC-51 (batched send_email with
  create_calendar_event instead of waiting), TC-82 (sent the checklist
  without a confirmed recipient). The MEDIUM vs xhigh thinking level is the
  known A/B fairness caveat (his quant defaults to MEDIUM; the cluster
  anchors run xhigh).

## State at shift end

- DSV4.1-Flash base on the NAS: done (476G, 48 shards, refs/main dba1be0a).
- Aiden 4x + Qwen 1x benches landed in their recipe `reports/`.
- EXL3 2x (task 33, coolbho3k 3bpw + the 2 GB headless-RAM unlock): NOT
  done — out of time for the shift; the base model on the NAS unblocks it
  (the quant is smaller than 510 GB).
- All serves stopped, cluster powered off fully (user's explicit instruction
  for the end of the shift).

## Ops notes (new)

- `uv tool run hf` on a node without the `hf` CLI works (installs
  huggingface_hub in an ephemeral env); a plain `hf` in a nohup bash -c is
  not on PATH → "command not found", and the download "completes" instantly
  with 0 bytes (check the log, not just the .done file — this one wrote the
  .done from a failed wrapper).
- `hf download --cache-dir <dir>` on this node's huggingface_hub version
  creates a **legacy** layout (no `hub/` subfolder) — plan the symlink
  accordingly; keep container-visible symlinks relative.
- SGLang build in Aiden production-1.6: `return_token_ids` + streaming is a
  hard 400 (was ignored before sgl-project/sglang#30917); tool-eval-bench
  v2.7.0 handles it.
- A new llama-benchy perf table that is 10×+ below the rest of the fleet's
  numbers: suspect a per-request cold path on the server (check
  prefix-cache behavior + server-side stats), not the harness.