|
@@ -957,3 +957,45 @@ frontend is upstream-experimental (not feature-complete; parser parity
|
|
|
for glm45/glm47 tool/reasoning unverified). Revisit only if upstream
|
|
for glm45/glm47 tool/reasoning unverified). Revisit only if upstream
|
|
|
matures it (or if per-request CPU overhead ever becomes a bottleneck in
|
|
matures it (or if per-request CPU overhead ever becomes a bottleneck in
|
|
|
our benches).
|
|
our benches).
|
|
|
|
|
+
|
|
|
|
|
+## RoCEnante (b12x PR #295) — queued for research (2026-09-07)
|
|
|
|
|
+
|
|
|
|
|
+`b12x.comm.roce`: one-shot all-reduce + all-gather over RoCE for multi-node
|
|
|
|
|
+Spark TP (1 GPU/node). Peers stage inputs into pinned host memory and
|
|
|
|
|
+RDMA-write them directly — no GPUDirect/dmabuf/peermem needed (unavailable
|
|
|
|
|
+on Spark); a C proxy posts the writes (ibv_reg_mr), a CuTe kernel reduces
|
|
|
|
|
+in fixed rank order → bit-identical results on all ranks; CUDA-graph
|
|
|
|
|
+replay safe via a device-resident epoch; fail-stop with self-poisoning
|
|
|
|
|
+(never a fallback). Measured on our exact production shape: GLM-5.3-Flash
|
|
|
|
|
+TP4 + MTP5, 4 Sparks. Microbench (bf16, graph replay): 3–3.8× vs NCCL up to
|
|
|
|
|
+1 MB (48 KB 23.6 vs 54 µs; 1 MB 238.9 vs 899 µs). Serving A/B: decode step
|
|
|
|
|
+latency lower in 14/15 cells (c=1 62–64 vs 65–66 ms; c=16 194–212 vs
|
|
|
|
|
+246–259 ms), throughput c=16 +22–32%, c=4 +6–22%, prefill unchanged
|
|
|
|
|
+(64 MB all-reduces stay on NCCL above the 2 MB cap), coding-peak c=1 −2%
|
|
|
|
|
+(noise). With both collectives routed, the decode profile shows zero NCCL
|
|
|
|
|
+kernels — i.e. it attacks exactly our measured decode profile
|
|
|
|
|
+(bandwidth-bound, scales to ~N=4; prefill untouched).
|
|
|
|
|
+
|
|
|
|
|
+Integration surface — verified already in our tree:
|
|
|
|
|
+- the vLLM adapter `b12x_roce_all_reduce.py` is **already present in our
|
|
|
|
|
+ r27 pin** (63a82f8d, blob a311ae52) — the voipmonitor fork mirrors
|
|
|
|
|
+ local-inference-lab/vllm (our pin resolves there), and the A/B receipt's
|
|
|
|
|
+ adapter (aef3576) predates our pin, so this is a zero-port item;
|
|
|
|
|
+- b12x env falls back to `NCCL_IB_HCA` / `NCCL_IB_GID_INDEX` — our config
|
|
|
|
|
+ already pins both CX-7 functions (`rocep1s0f0,roceP2p1s0f0`) and the
|
|
|
|
|
+ per-node GID auto-detect propagates automatically;
|
|
|
|
|
+- enable = `VLLM_ENABLE_ROCE_ALLREDUCE=1` in recipe `[env]` (caps
|
|
|
|
|
+ `VLLM_ROCE_ALLREDUCE_MAX_SIZE`=2M / `VLLM_ROCE_ALLGATHER_MAX_SIZE`=16M).
|
|
|
|
|
+
|
|
|
|
|
+Remaining work (r27 context, production untouched):
|
|
|
|
|
+1. Bump `B12X_PIN` e8ad299b → 00280f2 (resolve full SHA + tree hash,
|
|
|
|
|
+ one versions.env line) + dgx5 clone-at-pin rebuild (~1–2 h).
|
|
|
|
|
+2. A/B on the r27 2x lane (2 free nodes): decode c1/c2 vs the anchors
|
|
|
|
|
+ (24.5–27 t/s tg, 1,356 pp c1); re-verify quality — bf16 sums are within
|
|
|
|
|
+ 6.8e-3 rel of NCCL (float32 accumulate), so expect the 156 to hold.
|
|
|
|
|
+3. A 4x A/B would mean production sign-off (the toggle is an env var only).
|
|
|
|
|
+
|
|
|
|
|
+Queue: item #4, after the r27 DFlash2 fork patch + the r18 16 GiB
|
|
|
|
|
+boot-verify. Sources: local-inference-lab/b12x PR #295, docs/rocenante.md,
|
|
|
|
|
+receipts docs/evidence/rocenante/ (20260902 serving A/B + 20260903
|
|
|
|
|
+standalone, b12x 00280f2).
|
