# Night shift 2026-09-08

## RoCEnante 2x A/B (r27-spark, 4nodes-switch) — DONE

Goal: operator asked to (1) make `VLLM_ENABLE_ROCE_ALLREDUCE=1` the default in
the r27 recipes and (2) test the 2x variant (the 4x DCP4 A/B from 09-07 showed
decode +31…+90%).

Timeline (UTC; node clocks are +2 h ahead — keep that in mind when reading
node-side log timestamps):

- ~09:19 production 4x aiden stopped (operator OK) for the 2x work.
- Arm A (NCCL control, `glm53-flash-r27-spark` as-is): 2x on dgx1+dgx2,
  bench c1,2 seed 42 → **158/176 (90)**.
- Recipe rollout: `VLLM_ENABLE_ROCE_ALLREDUCE=1` added +
  `--disable-custom-all-reduce` removed in both r27 recipes; the interim
  `glm53-flash-r27-4x-dcp4-roce` clone deleted, its reports consolidated
  into `glm53-flash-r27-4x-dcp4/reports/` (commit fb3fbf5).
- Arm B attempt 1 (10:13): picked dgx1+dgx3; the dgx3 worker container died
  within ~2 min, the rank-0 head hung in NCCL init (orphan). First sighting
  of the stop-timing race: `sm121 stop` returned before the previous
  (production 4x aiden) container had fully released the GPU on dgx3.
- Operator directive: `stop` must actually wait for the containers to be
  gone; `--no-wait` to opt out. Implemented: `stop` now runs a per-node
  verification poll (30 × 2 s) after `docker stop` and errors if labeled
  containers are still running (commit a33cfa8). The web dashboard stop job
  uses the waiting variant.
- Arm B attempt 2 (10:24, dgx2+dgx3): same dgx3 worker death.
- Arm B attempt 3 (10:54, dgx1+dgx2): both workers survived; RoCEnante
  activated (`world=2, hcas=rocep1s0f0,roceP2p1s0f0`, dispatch
  `['B12X_ROCENANTE','PYNCCL']`). Bench ran to ~43% and the engine died:
  TileLang trace crash in `glm5next/nvidia/ops/glm_kpool.py`
  (`'int' object has no attribute 'to'` at `block.to(tl.int64)`) →
  EngineDeadError, clean shutdown. NOT RoCEnante-specific on its face (the
  NCCL arm completed the same scenarios fine — but see below), one-off.
- Arm B re-run (11:23, dgx1+dgx2, RoCEnante): clean full run →
  **162/176 (92)**, 100% completion, zero 500s. Crash did not reproduce.

2x A/B result (dev46, seed 42, c1,2):

| cell | NCCL tg | RoCE tg | Δ tg |
|---|---|---|---|
| d0 c1 | 23.4 | 27.4 | +17% |
| d0 c2 | 33.5 | 33.3 | −0.6% |
| d4096 c1 | 25.3 | 24.4 | −3.6% |
| d4096 c2 | 31.8 | 30.6 | −3.8% |
| d8192 c1 | 25.2 | 24.7 | −2.0% |
| d8192 c2 | 29.1 | 29.4 | +1.0% |

Quality: 158 → 162 (+4). Scenario flips: up TC-03/33/49/61/62/68/75
(TC-33 fail→pass); down TC-40/50 (pass→partial), TC-88 (pass→fail).
Conclusion: at 2x RoCEnante is a quality win and a small d0-c1 decode win;
the 4x DCP4 decode win (+31…+90%) is the headline. RoCEnante is now the
default in both r27 recipes.

## `sm121 bench` (operator request)

"Given how we run our benches, should we add a command to the launcher that
does all the dirty work?" — yes. Implemented (commit 1ef3eeb):
`sm121 bench --recipe R --cluster C --label L [--concurrency 1,2] [--seed 42]`
— waits for API readiness (900 s), runs tool-eval-bench (PATH) in a scratch
cwd under `~/.cache/sm121/bench/`, lands `<UTC-date>_<label>.{md,json,log}`
into `recipes/<lane>/reports/`, prints the final score + perf rows.
`-d` dry-run prints the exact command. Served name/port come from the recipe
(`--served-model-name` in args / `port`). The same commit moves the `stop`
handler from `main.rs` to `docker/stop.rs` (line-budget).

## Ops notes / lessons

- Node clocks on the Sparks run +2 h ahead of UTC; the launcher's own INFO
  lines are UTC. Compare timestamps across both before drawing conclusions.
- The 12:0x `stop && serve` restore chain broke silently: `time stop | tail`
  in a backgrounded list swallowed the stop's output, and the serve half
  never ran — production sat down 07:18→12:21 without a notice. Rule:
  background chains that chain state-changing commands must log to a file
  both halves read, or be run foreground; verify the second half actually
  started. The 12:21 direct serve started fine (4/4, loading at 12:23).
- Per operator: the 4x node is not needed right now — the 12:21 4x aiden
  serve was stopped again at 12:24 and left down. Cluster is idle/clean;
  restore with `sm121 serve --recipe glm53-flash-aiden-4x --cluster 4nodes-switch`
  when the operator wants the agent back.