# Night shift 2026-09-26 (UTC)

Plan (user): DSV4.1-Flash base → NAS, stop MiMo Pro 8x, MiMo-V2.6-Flash-RL 2x
(SGLang), time permitting Qwen Int4 1x / Aiden DSV4.1 4x, **full power off at
the end of the shift**.

## Timeline (UTC)

- ~00:00 — user's `hfrs download` of `deepseek-ai/DeepSeek-V4.1-Flash` (510 GB)
  found already running on dgx1 (PID 677687); my duplicate `hf` process killed
  (lock-blocked). Final state on NAS: 476G, 48 safetensors,
  `refs/main = dba1be0a40aa45a94ad051997016db3960a90277`
  (`/mnt/nas/ai/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash`).
- MiMo Pro 8x stopped (`sm121 stop --recipe mimo26-pro-rl-8x --cluster
  8nodes-switch`); nodes freed.
- MiMo-V2.6-Flash-RL 2x (dgx1+2, `mimo26-flash-rl-2x`, `mimo26-spark:a14`,
  model staged on dgx1+2 local, 166 GB each): first boot failed
  ("SG17 RoCEnante requires the TP8 Gloo exchange group") → removed
  `SGLANG8_ROCE_ALLREDUCE` from the recipe env. Boot OK: KV 340,736 tokens,
  ctx 327680. Smoke OK (math, tools, needle 132K ×2, code C1 71.7-78.7 t/s vs
  vLLM ref 70.5). **Bench: 161/176 (91/100, ★★★★★)** — TC-74 (safety-critical)
  failed. Report `recipes/mimo26-flash-rl-2x/reports/2026-09-25_231142_sglang-2x-first.*`.
  Committed+pushed (5bd465a). Stopped to free dgx1+2.
- Aiden DSV4.1-Flash 4x (dgx1-4, `dsv41-flash-4x-sglang-aiden16`): image
  `aidendle94/sparkrun-sglang-dsv41-gb10:production-1.6` (33.3 GB) pulled on
  dgx4 → synced to dgx1-3. Full flag/env set extracted from the aidenle.com JS
  bundle.

## Aiden 4x — boot saga (three launcher-side traps found)

1. **NGC entrypoint**: image ENTRYPOINT = `/opt/nvidia/nvidia_entrypoint.sh`
   (banner + `exec "$@"`) — args pass through, not the culprit.
2. **TOML ordering trap**: my `[[extra_mounts]]` block landed *before*
   `args = [...]`; the bare `args` key silently parsed into the table and was
   dropped (the recipe served with no flags — server_args showed all
   defaults: ep_size 1, mem_fraction 0.885, no speculative). Fix: `[[…]]`
   headers after all bare top-level keys. (Already documented in AGENTS.md
   "TOML section order".)
3. **Engram per-layer row counts**: the overlay requires the ranks'
   `SPARK_ENGRAM_ROWS` ranges to tile `[0, N)` exactly per layer — layer 1 has
   384,006,168 rows, layer 14 has 384,016,682 (384016682/4 is not integer).
   The wrapper (repo: `recipes/dsv41-flash-4x-sglang-aiden16/aiden4x-launch.sh`,
   deployed to `/home/kilork/aiden4x-launch.sh` on dgx1-4) reads both N from
   the checkpoint's safetensors headers at launch and tiles floor/ceil.

Booted: 524K ctx, EP4+TP4, dsv4 attention, DSpark block 5, fp8 KV, KV pool
**9.5M full tokens/node** (Aiden's fleet: ~4.2M). Smoke: count-20 exact, tools
OK, 155K-token needle OK.

## Aiden 4x bench + the perf anomaly (important)

- **Quality: 162/176 (92/100, ★★★★★)** — best DSV4.1 score on this cluster
  (vLLM r36/r37 4x: ~155-159). Fails TC-68/74/88; partials TC-03/14/50/51/
  57/58/62/81.
- **Perf table looked broken**: llama-benchy pp 154-359 t/s, tg 8.6-21.8 t/s,
  TTFT 10-51 s — while manual requests on the same server ran 2,400-3,000 t/s
  prefill. Investigation:
  - `return_token_ids=true` + `stream=true` → HTTP 400 on this SGLang build
    (sgl-project/sglang#30917). tool-eval-bench already works around it for
    sglang backends (`--extra-body return_token_ids=false`), so the bench's
    requests were valid.
  - A/B: first request of a session TTFT 10.4 s; repeats of the same prefix
    0.3 s; **every fresh (non-prefix-cached) context 2-10 s slower** with a
    decaying tail (5.0 → 3.6 → 2.6 → 2.1 s for identical 2.7K fresh prompts).
  - Container engram stats during the fresh prompts: 100-140 lookups,
    **mean gather 29-35 ms**, page cache "peak 4.00 GB … 1 releases (4.0 GB
    released)" — the engram n-gram rows (layers 1+14, ~384M rows each) are
    served from the NAS shards via `libengram_rows.so`; a fresh context
    touches thousands of new rows, each a random NFS read. Prefix-cached
    repeats touch cached rows → fast.
  - Control: the Qwen Int4 bench (same harness, no engram model) measured
    normal numbers (pp 1,886-2,103 t/s, tg 66-100 t/s) — the harness and the
    Mac side are not the problem.
- **Conclusion**: not an engine bug. The engram host-table design pays a
  fresh-context penalty of ~2-10 s when the rows live on the NAS (our
  fallback; Aiden's production uses 47.2 GiB node-local sparse copies via the
  private `engram_local.py`). Agent traffic (short, repeated system prompts)
  sits in the cached regime and is fast; long fresh-context loads are slow.
  The committed perf table for this lane reflects the fresh-context penalty.
  If the lane becomes production, either accept the penalty or obtain
  Aiden's local-copy setup.
- Perf-only re-run (`--perf-only`, label `aiden16-perf-rerun`): 189 t/s d0
  c1 (same pattern), confirms it's systematic, not a one-off.

## Qwen3.8-Flash-Next-Int4-AutoRound 1x (dgx5, `qwen38-flash-next-int4`)

- Model: `azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound` (122 GB,
  123 files) downloaded on dgx5 (the node's `hf` CLI was missing →
  `uv tool run hf` — first attempt lost, `hf: command not found`).
- **Legacy cache layout**: the `hf` download on this node wrote
  `/home/kilork/hf-local/models--…` (top level, no `hub/`). The mod expects
  `hub/models--…/refs/main` → added a **relative** symlink
  `hub/models--… → ../models--…`. An absolute symlink dangles inside the
  container (the mount root differs: host `/home/kilork/hf-local` vs
  container `/root/.cache/huggingface`).
- Mod vendored from azampatti's repo (`recipes/qwen38-flash-next-int4/mods/
  flashnext-int4-b12x/`: run.sh, build_views.py, patch_b12x.py,
  vllm_fp8_hybrid.py) — launcher uploads it per node and mounts `/mods:ro`;
  it checks the snapshot, builds `/workspace/flashnext/{model,draft-k10}`
  (symlink views + small JSON, zero copy) and patches the container's vLLM
  (fp8-hybrid, int4-head, index-filter, draft-scale, fla fixes).
- **Two-positional-model trap**: the launcher emits the recipe `model` as the
  positional arg; with `model = "<repo id>"` + `args[0] = "/workspace/…"`
  vLLM got two positionals (`unrecognized arguments: /workspace/…`). Fix:
  `model = ""` (the launcher skips the positional) + the view in args.
- Serve: 1x Spark (cluster 1node-5), eugr/spark-vllm-b12x, MEDIUM-thinking
  chat template (his default; the speculative head is trained on it).
- Smoke: math OK, tools OK, 45K-token needle OK, **decode C1 75.9 t/s**
  (his claim 60-70), prefill 34K/18 s (~1.9K t/s).
- **Bench: 165/176 (94/100, ★★★★★)** — best score of the night (single node!).
  Perf: pp 1,886-2,103 t/s, tg 66-100 t/s (c1/c2), median turn 2.2 s.
  **2 safety-critical flags**: TC-51 (batched send_email with
  create_calendar_event instead of waiting), TC-82 (sent the checklist
  without a confirmed recipient). The MEDIUM vs xhigh thinking level is the
  known A/B fairness caveat (his quant defaults to MEDIUM; the cluster
  anchors run xhigh).

## State at shift end

- DSV4.1-Flash base on the NAS: done (476G, 48 shards, refs/main dba1be0a).
- Aiden 4x + Qwen 1x benches landed in their recipe `reports/`.
- EXL3 2x (task 33, coolbho3k 3bpw + the 2 GB headless-RAM unlock): NOT
  done — out of time for the shift; the base model on the NAS unblocks it
  (the quant is smaller than 510 GB).
- All serves stopped, cluster powered off fully (user's explicit instruction
  for the end of the shift).

## Ops notes (new)

- `uv tool run hf` on a node without the `hf` CLI works (installs
  huggingface_hub in an ephemeral env); a plain `hf` in a nohup bash -c is
  not on PATH → "command not found", and the download "completes" instantly
  with 0 bytes (check the log, not just the .done file — this one wrote the
  .done from a failed wrapper).
- `hf download --cache-dir <dir>` on this node's huggingface_hub version
  creates a **legacy** layout (no `hub/` subfolder) — plan the symlink
  accordingly; keep container-visible symlinks relative.
- SGLang build in Aiden production-1.6: `return_token_ids` + streaming is a
  hard 400 (was ignored before sgl-project/sglang#30917); tool-eval-bench
  v2.7.0 handles it.
- A new llama-benchy perf table that is 10×+ below the rest of the fleet's
  numbers: suspect a per-request cold path on the server (check
  prefix-cache behavior + server-side stats), not the harness.

## Second half of the shift (post power-off, ~06:00+)

User re-engaged: finish the two recipes (Aiden + Qwen) with complete results,
then the new **EXL3 2x** task (temporary 2 GB headless-RAM unlock via
recipe scripts).

- **Aiden recipe TOML fixed** — the header-comment edit had dropped the `#`
  on line 29 → tomllib `Expected '=' after a key`. Repaired, validates
  (38 args / 40 env).
- **Perf table missing from .md reports root-caused**: tool-eval-bench v2.7.0
  `--json-file` implies `--json` → `_run_json` dispatch path drops
  `throughput_samples` → the "## Throughput Metrics" section is absent from
  the md (and the perf data from the JSON); the table survives only in the
  `.log`. Fixed launcher-side: `src/bench_report.rs`
  (`ensure_throughput_section`) re-attaches the llama-benchy table parsed
  from the bench `.log` into the copied `.md` (harness pin untouched).
  Retroactively applied to the Aiden `…_015216` and Qwen `…_022931` reports.
  Commit 4546237.
- **EXL3 2x recipe** (`dsv41-flash-exl3-2x`, coolbho3k rc2 image
  `ghcr.io/coolbho3k/deepseek-v4.1-flash-exl3-3bpw-2x-dgx-spark:20260917-rc2`):
  - User started the model download on dgx5 **local** — 636 GB does not fit
    (251 G free) → killed, redirected to the NAS
    (`/mnt/nas/ai/huggingface/hub/…`, hub layout). The 2x pair serves from
    the NAS; drafter overlay `coolbho3k/DeepSeek-V4.1-Flash-DSpark-EXL3-3bpw`
    downloaded too (small, done first).
  - Runtime requirements extracted from the pinned repo (commit 878e0ee):
    model must be mounted at exactly `/model` (drafter in `/model/draft`,
    engrams in `/model/engrams`), drafter overlay at
    `DS41_DRAFT_EXL3_PATH=/draft-exl3`, engine call
    `serve.py serve /model --config conservative.yaml --worker-cls
    combined_worker.CombinedWorker --disable-custom-all-reduce …`, hard
    serving cgroup guard (9 GiB / no-swap / ≤6 CPU), 2 GB display reserve
    (nvidia_drm modeset=1 fbdev=0) for the 1.75 GiB display-pool KV.
  - Launcher extensions (all committed): per-recipe `devices = [...]`
    (`--device d:d`), `memory_limit`/`cpus` (→ `--memory/--memory-swap/--cpus`),
    built-in scripts `unlock_display_ram` / `restore_display_ram` (temporary
    per-boot remod, no UEFI/modprobe.d/initramfs changes; restore on stop),
    `--cluster auto` now honors `recipe.cluster` on the docker path (tmux
    path already did — parity fix).
  - Recipe: NAS hub cache + per-node before_serve symlinks
    (`/home/kilork/exl3-model` → current refs/main snapshot) bound ro at
    `/model` / `/draft-exl3`; 41 DS41/NCCL envs; 200G pair dgx1+dgx2
    (powered on for this), 1M ctx, GMU 0.92, C2.
  - Unlock/restore scripts validated live on dgx2 (default state was
    modeset=N fbdev=Y → remod → modeset=Y fbdev=N + /dev/dri/card0 →
    restore back).
  - In flight at this point: model download ~55/636 GB on the NAS, image
    pull on dgx1.

## Day shift (26 Sept, ~12:00–16:00 UTC+2) — Qwen garble investigation + EXL3

### Qwen (`qwen38-flash-next-int4`, dgx5, port 30000, `qwen3.8`)
User report: model works for a while, then emits random numbers/garbage;
"appears around 30% of the context"; still happening at report time.

Investigation (controlled repro attempts, all via API on 192.168.88.43:30000):
- Short/medium context: clean (120-req stress 1/120 = 1 client timeout, 0 garble).
- 60K multi-turn (12 blocks ~6K each, prefix-cached): 1 true garble in the
  whole campaign (digit truncation 726→"72" etc.) — clustered in the first
  30 min of the 09:24 instance (10:35–10:37 UTC); **0 true garbles in
  400+ requests since** (both atomic-flag states, temp 0/0.7, streaming,
  concurrency, tools, 123K session growth = 0/72 true fails).
- `content=null` cases in early probes were my own artifact: `max_tokens`
  (50–80) truncated the reasoning channel (105–145 tokens) before content.
- Engine logs clean (no NaN/OOM/warnings; spec-decode acceptance normal
  4.3/4). dmesg: 3× NVRM NV_ERR_NO_MEMORY — all at container startups
  (08:52, 09:24), none during operation; no Xid/ECC.
- Hypothesis tested: `VLLM_MARLIN_USE_ATOMIC_ADD=1` (experimental Marlin
  atomic-add path, n<2048/k≥2048 layers) — kernel analysis: slice-0 zeroes
  the `torch::empty` output then sets a lock, other slices atomicAdd after
  a lock wait; zeroing→lock write lacks a GPU memory fence → potential
  race. A/B (restart each way): perf within noise (decode 47.5 vs 51.9
  tok/s, conc1 339 vs 303, conc4 253 vs 195), quality 0/40 both states →
  **not conclusively the cause**; flag left as the recipe shipped (1).
- Hardware cross-check: Qwen replicated on **dgx3** (model pulled from
  dgx5 over the fabric, image pushed node-to-node): 0/40 garble there too.
  Inconclusive (bug too intermittent to catch in controlled runs).
- Monitor running: `/tmp/qwen_monitor.py` (probe every 2 min →
  /tmp/qwen-monitor.log) as a tripwire.

Standing: if garble recurs, need **exact time + a sample** to correlate
with engine log / dmesg. dgx3 kept up as hot spare (was off before).

### Launcher bug found (not fixed)
Prebuilt external image missing on the target node → `docker run` silently
pulls from Docker Hub over the internet → the launcher's SSH command timeout
kills the pull ("context canceled" in dockerd) but the serve still logs
"✅ VLLM started" and waits 1800 s for an API that never comes. Needs a
pre-run image presence check (or node-to-node sync) + honest error.

### Config fix
`config.toml` 1node-4 node entity was `switch.dgx3` (copy-paste) → fixed to
`switch.dgx4` (the 1node-3 node keeps `switch.dgx3`).

### EXL3 2x — ELOOP root cause + fix (serving)
Root cause (after ~10 red herrings: NFS dentry races, concurrency, cgroups,
launcher-flag diffs — all disproven by flag-for-flag replicas): the DS41
runtime opens the shard files with **O_NOFOLLOW** (`ssd_vocab_rows.py:68`
`os.open(self.path, O_RDONLY|O_CLOEXEC|O_NOFOLLOW)`). With symlink-based
staging, the final path component IS a symlink → open(2) returns ELOOP
("Too many levels of symbolic links") **deterministically** — the kernel's
ELOOP for O_NOFOLLOW, not a loop at all.
Fix: no symlinks in the container. New recipe field
`extra_mounts_manifest = "/tmp/sm121-exl3-mounts.txt"`: the before-serve hook
generates a per-node `host<TAB>container` list (one **bind per real file**,
161 model + 10 draft entries, resolved to the final hub blob), the launcher
reads it over SSH after the hooks and appends it as read-only binds. The
container sees regular files only → O_NOFOLLOW is satisfied.
Result: EXL3 2x boots (shards stream from the NAS, ~15 min), API live on
dgx1:30000 as `deepseek-v41-flash-exl3`, smoke 127×89 + Paris correct.

## Qwen garble hunt — root cause (2026-09-26 late)

The intermittent Qwen garble (empty replies, degenerate text, special-token
leakage in tool-call markup) was tracked all day. The trail:

1. Caught live on the arithmetic monitor (fresh-context probes, 2-min cadence)
   twice; caught in the user's pi session (80K+ tool-heavy context) three
   times — `stopReason: stop` at 36-229 tokens (the model stopped itself
   mid-degeneration). Engine logs at those moments: no warnings, no Xid, no
   OOM.
2. A/B matrix (all red herrings): marlin atomic-add 1 vs 0 (perf-neutral,
   garble on both), MTP k=4 vs k=3 (garble on both), min config (no MTP, no
   prefix cache — temporarily clean), GDM/display stack (was up on 88.43,
   absent on 88.50 — garble on both), both boxes (garble on both).
3. Label-inversion trap: the pi provider named "dgx5" pointed at dgx3.lan
   (88.50) — the box actually serving the user's agent — while all my
   testing/A/Bs went to the other box (88.43). Provider list now has
   unambiguous `qwen-min` / `qwen-prod` entries.
4. Root cause (forum 382733 posts 195-198, 0rand + azampatti): the same
   corruption on an UNPINNED b12x image, fixed by the pinned digest. Our
   recipe ran floating `eugr/spark-vllm-b12x:latest`, which we had pulled as
   `nightly-20260925` (digest c9e22735). The validated pin
   (azampatti repo config.env) is `nightly-20260913` (8e7e0621). So every
   nightly from 09-14..09-25 is suspect; the 165/176 bench ran on the
   suspect image (the 88-scenario pool just doesn't hit the degenerate tail).

Fix: both Qwen boxes now serve the pinned digest (pulled node-side, ~25 GB
each), recipe `docker_image` pinned with a comment (no floating tags),
bench baseline params restored (MTP k=4, atomic-add 1). Launcher fix:
`get_image_digest` now accepts digest refs (`docker image ls` silently
returns nothing for `@sha256:` refs — would have made every pinned prebuilt
recipe fail). Re-bench pending (label int4-auto-1x-pinned).

Verdict: re-bench on the pinned image scored **94/100 (164/174)** — same
quality as the suspect-image baseline (93), so the fix costs nothing. The
user's live agent session on dgx5.lan has since run clean ("works like a
clock") — the long-context degeneration that had hit ~5 sessions/day is
gone. dgx3.lan turned off after the bench; fleet: dgx5 (qwen3.8, pinned) +
dgx1/2 (EXL3 2x) live.