launcher/history/night-shifts/2026-09-07.md

Night shift 2026-09-07 → 09-08 — RoCEnante A/B (4x DCP4, 4nodes-switch)

Goal: close research-queue item #2 — A/B RoCEnante (b12x one-shot RoCE all-reduce/all-gather, already in the r27 image + vLLM adapter) vs the NCCL control, in the exact documented A/B shape (GLM-5.3-Flash TP4 + MTP, 4 Sparks, CX-7 200 GbE RoCE fabric, both rails).

Ops: production glm53-flash-aiden-4x stopped at start (explicit operator OK for all 4 nodes), restored at end (serving again by 07:2x UTC). Tool version: tool-eval-bench 2.6.1.dev46+g9d412464a, seed 42, hardmode, perf c1,c2.

Arms

• Arm A (NCCL control): glm53-flash-r27-4x-dcp4 — as-is (incl. --disable-custom-all-reduce).
• Arm B (RoCEnante): glm53-flash-r27-4x-dcp4-roce — same recipe + VLLM_ENABLE_ROCE_ALLREDUCE=1 and WITHOUT --disable-custom-all-reduce (required: in the r27 fork the RoCEnante adapter is gated on use_custom_allreduce — with the disable flag the branch is silently skipped, verified by absent adapter logs + missing proxy build). On a 4x 1-GPU-node cluster vLLM's own CUDA-IPC custom AR is functionally inert (logs "multi-node group does not support MNNVL multicast"), so the removed flag only opens the b12x/RoCEnante path — not a confounder.

Activation (arm B, rank-0 logs):

• Using RoCEnante (b12x one-shot RoCE collectives): world=4, hcas=rocep1s0f0,roceP2p1s0f0, all-reduce max=2097152 bytes, all-gather shard max=16777216 bytes
• tp:0 dispatch ['B12X_ROCENANTE', 'PYNCCL']; dcp:0/ep:0 stay PYNCCL (adapter only targets the tp group — consistent across both arms)
• RoCEnante all-reduce is live: first routed all-reduce is 131072 bytes (bfloat16) + all-gather is live: first routed shard is (16, 38720) bfloat16 along dim 1 (the doc's microbench shape)
• Proxy built at first use, persisted per node: ~/sm121-cache/glm53-jit-121f/b12x/roce/roce_proxy-a35f54cf75d6abf2.so (b12x cache mount → survives across serves; no rebuild cost on later arms)

Results

cell	pp c1 A→B	tg c1 A→B	tg c2 A→B	TTFT c1 A→B	total c1 A→B
d0	3,319→2,248	29.9→46.8 (+57%)	46.3→62.4 (+35%)	1,093→988	4,847→3,561 (−26%)
d4096	2,062→1,883	26.5→50.4 (+90%)	32.8→50.2 (+53%)	3,203→3,053	7,476→5,433 (−27%)
d8192	2,072→1,999	31.0→47.5 (+53%)	24.8→32.4 (+31%)	4,880→4,684	8,474→7,216 (−15%)

• Decode: +31…+90% tg in every cell; total request time −15…−28%. Bigger than the published +22–32% @ c16 — our r27 DCP4 shape gains more at c1/c2 (the DCP ring stays on NCCL in both arms, so the delta is purely the TP collectives moving to RoCEnante).
• Prefill: 5/6 cells within −9…+8%; the d0 c1 cell is an outlier (3,319→2,248, −32%) — one short 2048-token prefill measurement; candidates: RoCEnante intercepting sub-2MB prefill-chunk all-reduces (per-message RDMA overhead) or run variance. Watch if it repeats; the doc's "prefill unchanged" holds for the rest of the table.
• Quality: 160/176 (91) vs 155/176 (88). +5 = 9 borderline flips (6 up, 3 down): TC-40/60/61/80/82 gained, TC-53/62/81 lost, TC-51 fail→partial — consistent with RoCEnante's different reduction arithmetic (float32 accumulate, within 6.8e-3 rel of NCCL) nudging near-threshold scenarios in both directions. Two of the 09-07 anchor failures (TC-51, TC-80) cleared. Not a systematic quality change; net happens to be positive.
• Deployability 76 vs 72, responsiveness 40 vs 33 (direct latency effect).

Ops notes

• First arm-A bench kill failed (process kept running); the restarted bench overlapped it ~15 min — arm A's quality is corroborated by that parallel run (157/176) and the clean 09-07 anchor (155/176); arm A's perf cells ran after the overlap ended (clean). Arm B ran alone.
• Bench convention reminder (now in AGENTS.md + state-doc header): keep the JSON report (--json-file <abs path>, absolute — bench runs outside the repo) alongside the .md; perf cells c1,c2.
• 4nodes-switch serve selection hiccup once ("only 3 nodes") — transient SSH drop, retry succeeded.