# Night shift 2026-09-07 → 09-08 — RoCEnante A/B (4x DCP4, 4nodes-switch)

Goal: close research-queue item #2 — A/B RoCEnante (b12x one-shot RoCE
all-reduce/all-gather, already in the r27 image + vLLM adapter) vs the NCCL
control, in the exact documented A/B shape (GLM-5.3-Flash TP4 + MTP, 4 Sparks,
CX-7 200 GbE RoCE fabric, both rails).

Ops: production `glm53-flash-aiden-4x` stopped at start (explicit operator OK
for all 4 nodes), restored at end (serving again by 07:2x UTC). Tool version:
tool-eval-bench 2.6.1.dev46+g9d412464a, seed 42, hardmode, perf c1,c2.

## Arms

- Arm A (NCCL control): `glm53-flash-r27-4x-dcp4` — as-is (incl.
  `--disable-custom-all-reduce`).
- Arm B (RoCEnante): `glm53-flash-r27-4x-dcp4-roce` — same recipe +
  `VLLM_ENABLE_ROCE_ALLREDUCE=1` and WITHOUT `--disable-custom-all-reduce`
  (required: in the r27 fork the RoCEnante adapter is gated on
  `use_custom_allreduce` — with the disable flag the branch is silently
  skipped, verified by absent adapter logs + missing proxy build). On a
  4x 1-GPU-node cluster vLLM's own CUDA-IPC custom AR is functionally inert
  (logs "multi-node group does not support MNNVL multicast"), so the removed
  flag only opens the b12x/RoCEnante path — not a confounder.

Activation (arm B, rank-0 logs):
- `Using RoCEnante (b12x one-shot RoCE collectives): world=4,
  hcas=rocep1s0f0,roceP2p1s0f0, all-reduce max=2097152 bytes, all-gather
  shard max=16777216 bytes`
- tp:0 dispatch `['B12X_ROCENANTE', 'PYNCCL']`; dcp:0/ep:0 stay PYNCCL
  (adapter only targets the tp group — consistent across both arms)
- `RoCEnante all-reduce is live: first routed all-reduce is 131072 bytes
  (bfloat16)` + `all-gather is live: first routed shard is (16, 38720)
  bfloat16 along dim 1` (the doc's microbench shape)
- Proxy built at first use, persisted per node:
  `~/sm121-cache/glm53-jit-121f/b12x/roce/roce_proxy-a35f54cf75d6abf2.so`
  (b12x cache mount → survives across serves; no rebuild cost on later arms)

## Results

| cell | pp c1 A→B | tg c1 A→B | tg c2 A→B | TTFT c1 A→B | total c1 A→B |
|---|---:|---:|---:|---:|---:|
| d0 | 3,319→2,248 | 29.9→**46.8** (+57%) | 46.3→**62.4** (+35%) | 1,093→988 | 4,847→3,561 (−26%) |
| d4096 | 2,062→1,883 | 26.5→**50.4** (+90%) | 32.8→**50.2** (+53%) | 3,203→3,053 | 7,476→5,433 (−27%) |
| d8192 | 2,072→1,999 | 31.0→**47.5** (+53%) | 24.8→**32.4** (+31%) | 4,880→4,684 | 8,474→7,216 (−15%) |

- **Decode: +31…+90% tg in every cell; total request time −15…−28%.**
  Bigger than the published +22–32% @ c16 — our r27 DCP4 shape gains more at
  c1/c2 (the DCP ring stays on NCCL in both arms, so the delta is purely the
  TP collectives moving to RoCEnante).
- **Prefill**: 5/6 cells within −9…+8%; the d0 c1 cell is an outlier
  (3,319→2,248, −32%) — one short 2048-token prefill measurement; candidates:
  RoCEnante intercepting sub-2MB prefill-chunk all-reduces (per-message RDMA
  overhead) or run variance. Watch if it repeats; the doc's "prefill
  unchanged" holds for the rest of the table.
- Quality: **160/176 (91) vs 155/176 (88)**. +5 = 9 borderline flips (6 up,
  3 down): TC-40/60/61/80/82 gained, TC-53/62/81 lost, TC-51 fail→partial —
  consistent with RoCEnante's different reduction arithmetic (float32
  accumulate, within 6.8e-3 rel of NCCL) nudging near-threshold scenarios in
  both directions. Two of the 09-07 anchor failures (TC-51, TC-80) cleared.
  Not a systematic quality change; net happens to be positive.
- Deployability 76 vs 72, responsiveness 40 vs 33 (direct latency effect).

## Ops notes

- First arm-A bench kill failed (process kept running); the restarted bench
  overlapped it ~15 min — arm A's quality is corroborated by that parallel
  run (157/176) and the clean 09-07 anchor (155/176); arm A's perf cells ran
  after the overlap ended (clean). Arm B ran alone.
- Bench convention reminder (now in AGENTS.md + state-doc header): keep the
  JSON report (`--json-file <abs path>`, absolute — bench runs outside the
  repo) alongside the .md; perf cells c1,c2.
- 4nodes-switch serve selection hiccup once ("only 3 nodes") — transient SSH
  drop, retry succeeded.