launcher/history/night-shifts/2026-09-08.md

Night shift 2026-09-08

RoCEnante 2x A/B (r27-spark, 4nodes-switch) — DONE

Goal: operator asked to (1) make VLLM_ENABLE_ROCE_ALLREDUCE=1 the default in the r27 recipes and (2) test the 2x variant (the 4x DCP4 A/B from 09-07 showed decode +31…+90%).

Timeline (UTC; node clocks are +2 h ahead — keep that in mind when reading node-side log timestamps):

• ~09:19 production 4x aiden stopped (operator OK) for the 2x work.
• Arm A (NCCL control, glm53-flash-r27-spark as-is): 2x on dgx1+dgx2, bench c1,2 seed 42 → 158/176 (90).
• Recipe rollout: VLLM_ENABLE_ROCE_ALLREDUCE=1 added + --disable-custom-all-reduce removed in both r27 recipes; the interim glm53-flash-r27-4x-dcp4-roce clone deleted, its reports consolidated into glm53-flash-r27-4x-dcp4/reports/ (commit fb3fbf5).
• Arm B attempt 1 (10:13): picked dgx1+dgx3; the dgx3 worker container died within ~2 min, the rank-0 head hung in NCCL init (orphan). First sighting of the stop-timing race: sm121 stop returned before the previous (production 4x aiden) container had fully released the GPU on dgx3.
• Operator directive: stop must actually wait for the containers to be gone; --no-wait to opt out. Implemented: stop now runs a per-node verification poll (30 × 2 s) after docker stop and errors if labeled containers are still running (commit a33cfa8). The web dashboard stop job uses the waiting variant.
• Arm B attempt 2 (10:24, dgx2+dgx3): same dgx3 worker death.
• Arm B attempt 3 (10:54, dgx1+dgx2): both workers survived; RoCEnante activated (world=2, hcas=rocep1s0f0,roceP2p1s0f0, dispatch ['B12X_ROCENANTE','PYNCCL']). Bench ran to ~43% and the engine died: TileLang trace crash in glm5next/nvidia/ops/glm_kpool.py ('int' object has no attribute 'to' at block.to(tl.int64)) → EngineDeadError, clean shutdown. NOT RoCEnante-specific on its face (the NCCL arm completed the same scenarios fine — but see below), one-off.
• Arm B re-run (11:23, dgx1+dgx2, RoCEnante): clean full run → 162/176 (92), 100% completion, zero 500s. Crash did not reproduce.

2x A/B result (dev46, seed 42, c1,2):

cell	NCCL tg	RoCE tg	Δ tg
d0 c1	23.4	27.4	+17%
d0 c2	33.5	33.3	−0.6%
d4096 c1	25.3	24.4	−3.6%
d4096 c2	31.8	30.6	−3.8%
d8192 c1	25.2	24.7	−2.0%
d8192 c2	29.1	29.4	+1.0%

Quality: 158 → 162 (+4). Scenario flips: up TC-03/33/49/61/62/68/75 (TC-33 fail→pass); down TC-40/50 (pass→partial), TC-88 (pass→fail). Conclusion: at 2x RoCEnante is a quality win and a small d0-c1 decode win; the 4x DCP4 decode win (+31…+90%) is the headline. RoCEnante is now the default in both r27 recipes.

sm121 bench (operator request)

"Given how we run our benches, should we add a command to the launcher that does all the dirty work?" — yes. Implemented (commit 1ef3eeb): sm121 bench --recipe R --cluster C --label L [--concurrency 1,2] [--seed 42] — waits for API readiness (900 s), runs tool-eval-bench (PATH) in a scratch cwd under ~/.cache/sm121/bench/, lands <UTC-date>_<label>.{md,json,log} into recipes/<lane>/reports/, prints the final score + perf rows. -d dry-run prints the exact command. Served name/port come from the recipe (--served-model-name in args / port). The same commit moves the stop handler from main.rs to docker/stop.rs (line-budget).

r28 full-stack evaluation + version profiles (evening shift)

Operator request: a build_profile entity — a named pin set referenced from recipes instead of copy-pasting the whole [build.versions] table. The half-cut r28 recipe (vLLM advanced, b12x/LMCache left at r27) was exactly the failure mode this kills.

• Feature (15e9b62): build_profile = "<name>" → docker-images/<dir>/ profiles/<name>.env (same KEY=VAL format); merge base < profile < recipe [build.versions]; merged file written to the node before build.sh and folded into the sm121.checksum label. profiles/glm53-r28.env carries the full r28 lock (vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a — all three forks cut simultaneously 09-06, strictly ahead of r27 pins).
• TOML context trap (b5f0662): the first build_profile edit landed after the [[extra_mounts]] block → the bare key parsed into the LAST mount table and serde silently ignored it; the build ran with base pins while the recipe "had" the override. Bare keys must stay top-level (tables like [build.versions] reset the context and are safe anywhere). Verified end-to-end with a live load_recipe + build_checksum probe before re-running the build; pitfall documented in both skills.
• Build: ~6 min (buildx cache), label 14271940…; image verified in-container (vllm r28, lmcache 934c287a, b12x 1.3.0 — the r28 b12x moved to static versioning, so the scm pretend no longer applies; pin still enforced by the HEAD+tree gates).
• Memory gate flake: first serve died at worker init — Free memory 107.86 < 0.9×121.63 = 109.46 GiB (the gpu_memory_utilization admission gate, vLLM #48140): docker load of the 16 GB image + ~12 GiB of starting vLLM processes pushed the check under the line. The launcher's existing drop_caches (post-sync, pre-container, default on) ran but the margin is still razor-thin; a retry with settled cache booted fine. Follow-up considered: GMU 0.88 in the 4x recipes (gate → 107.0 GiB; KV is explicitly pinned to 16 GiB so capacity is unchanged).
• Bench (full r28, seed 42, hardmode, c1,2): 155/176 (88/100) vs r27 RoCE 160/176 and mixed-stack r28 159/176; median turn 3.3 s (r27 3.9 s — the r28 scheduler win holds). 8 hard scenarios lost points vs r27 (TC-21/ 49/51/61/67/80/82/88); TC-80+TC-88 failed only on full r28. But: r27 itself spread 155–162 across its A/B runs, and the r28 b12x M8/M1 changes are layout-only ("no model math, scale precision, or output reduction is changed"), so 155 is inside the observed run-to-run band. Re-run queued to pin the variance before judging the stack.
• Re-run: 159/176 — the 155 was variance, not a regression: 8 scenarios flipped between the two identical-stack runs (incl. TC-80/TC-88, which now PASS). Full r28 (155–159) and r27 (155–162) are statistically tied on quality; r28 wins on speed (median turn 3.0–3.3 s vs 3.9 s; TTFT ~3.6× better per the mixed run). Full r28 stack = production candidate. Also hardened startup: GMU 0.90 → 0.88 in all r27/r28 4x/2x recipes (ff1f727) so the gpu_memory_utilization admission gate (107.0 GiB, not 109.5) survives the ~12 GiB of starting vLLM processes on top of drop_caches (which already ran post-sync, pre-container, default on).

Pending: (1) b12x O_DIRECT loader test (--load-format b12x, recipe glm53-flash-r28-4x-dcp4-b12xload ready) — the r28 b12x added B12xModelLoader (O_DIRECT reads into CUDA managed memory; weight loads stop polluting the page cache — relevant to the memory gate above); (2) Discord env vars VLLM_GLM53_SPLIT_TARGET_BLOCK_SIZE=2048 / VLLM_GLM53_SPLIT_MAMBA_BLOCK_SIZE=256 — split GLM-5.3 KV into independent target-MLA (2048) + recurrent-Mamba (256) page pools, requires mamba_cache_mode='align', fine-grained prefix reuse; research done in the fork source (vllm/platforms/interface.py:811), test pending.

Ops notes / lessons

• Node clocks on the Sparks run +2 h ahead of UTC; the launcher's own INFO lines are UTC. Compare timestamps across both before drawing conclusions.
• The 12:0x stop && serve restore chain broke silently: time stop | tail in a backgrounded list swallowed the stop's output, and the serve half never ran — production sat down 07:18→12:21 without a notice. Rule: background chains that chain state-changing commands must log to a file both halves read, or be run foreground; verify the second half actually started. The 12:21 direct serve started fine (4/4, loading at 12:23).
• Per operator: the 4x node is not needed right now — the 12:21 4x aiden serve was stopped again at 12:24 and left down. Cluster is idle/clean; restore with sm121 serve --recipe glm53-flash-aiden-4x --cluster 4nodes-switch when the operator wants the agent back.