# BUG-2026-09-19 — r38 (JJ tip + b12x master) boot deadlock: b12x preparation pool stuck at "waiting for ranks"

**Date:** 2026-09-19
**Lane:** GLM-5.3-Flash-NVFP4-Spark, 2x GB10 (2nodes-dac), TP2
**Image:** `glm53-jovian-r38:local` — vLLM `local-inference-lab/vllm` @ `8e1f1e587f8` (tree `d9f35de1`), b12x `local-inference-lab/b12x` @ `0f3a8cbfd1c` (tree `ff2218d7`)
**Severity:** high (blocks all boot on this engine pair; model loading itself is fine)
**Status:** open (worked around with r35 for now; **r39 tip verified fixed on 2x, 2026-09-22**)

## 2026-09-22 update — 4-rank reproduction + r39 fix verified

**4x reproduction (Track C, this night).** Same image on `4nodes-switch`
(dgx1–4, 200G switched fabric, TP4) deadlocks **at the very start** of the
preparation session — different stall point, same livelock:

```
b12x waiting for ranks: 0/1822 ready, 0 measured, 0 cached, 0 compilations, 3:52
b12x waiting for ranks: 0/1822 ready, 0 measured, 0 cached, 0 compilations, 4:02
... (80+ lines over 13+ min, counter never moves)
```

- Ranks 1–3: **zero log output** after the 02:43:33 startup lines; Worker_TP
  spinning at ~50% CPU (48.9 / 54.4 / 51.7 over the observation window).
- Rank 0: Worker_TP ~107% CPU emitting the 10 s poll line; system 95% idle
  otherwise.
- So the stall is **not** specific to the late `comm.roce` phase seen on 2x
  (there the pool reached 3806 measured / 32-of-33 ready before freezing on
  the final rendezvous) — on 4 ranks it dies at the first barrier (0/1822).
  Both modes are the same mutual `waiting for ranks` livelock with all
  workers busy-spinning.

**Fix verified in tip (r39).** `jovian-judgement:r39-glm53` (vLLM
`local-inference-lab/vllm` @ `9e5d1793fa3`, b12x @ `c2dc1cf022` "Configure RoCE
native traffic class for switched Spark fabrics" + ~100 commits over r38)
boots the same Spark checkpoint cleanly on 2x: the `comm.roce` phase that
froze r38 at 32/33 completes `33/33 ready` in 0:40, engine up, API serving,
bench 164/176 (parity with the r35 baseline 165/174, v2.7.0). The RoCE traffic
class commit targets exactly our switched Spark fabric — strong candidate
cause for the 2x stall; the 4x 0/1822 stall suggests the preparation-session
handshake also has a rank-count/topology edge beyond RoCE TC.

## Next steps

- Report to the lab with the 2x + 4x reproduction matrix (both r38 pins, GB10
  aarch64; signature lines above; CPU-spin evidence) and the r39 boot as
  evidence the tip clears it. Ask them to confirm whether the RoCE TC commit
  (b12x `c2dc1cf022`) is the intended fix and whether the 4x 0/1822 handshake
  edge is known.
- If r39 is adopted, the 4x r39 lane becomes the next perf target (r38 4x
  was never measurable).
- Optional: `B12X_PREPARATION_TRACE_DIR` traces from an r38 boot if the lab
  wants the exact stalled requirement (CollectiveRequirement vs
  TuningCacheRequirement).

## Symptom

Boot gets through weight loading (InstantTensor, 88 GiB/rank, clean — **no
`KeyError`, the JJ tip maps the Spark checkpoint's GDN naming fine**) and
immediately after `Using BLHNC KV cache layout` the rank-0 log freezes on:

```
b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 0:00
b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 9:31
... (indefinitely; 30+ min observed)
```

`0/1821` = `completed_requests/total_requests` of the b12x **kernel preparation
session** (1821 preparation requests; 1822 with `VLLM_ENABLE_ROCE_ALLREDUCE=1`).
The phase label maps to the `WAIT` stage in `b12x/preparation/_progress.py`.

## Observed state at steady state

- rank 0: `VLLM::Worker_TP` ~105% CPU, `VLLM::EngineCore` ~19% CPU — busy-spin
  (the waiting loop is a polling loop, not a futex sleep).
- rank 1: `VLLM::Worker_TP` ~60% CPU, main thread `R / wait_woken`, all 66
  threads otherwise `S`; **no new log output** after the same
  "Model loading took 88.07 GiB" line.
- **No compiler processes** anywhere on either node (no `c++`/`nvcc`/tvm-ffi
  subprocesses) — the pool never starts a single compilation
  (`0 compilations`, `0 cached`, `0 measured`).
- NCCL/gloo watchdog threads idle; no crash, no OOM, no error line on either rank.

Both ranks wait for each other in the vLLM↔b12x preparation protocol
(`session.py`: the job blocks on `CollectiveRequirement` /
`TuningCacheRequirement` until "participating ranks" contribute; the progress
phase flips to "waiting for ranks" as soon as any ready_* flag is set while
`done` is false). Neither side ever makes progress; the wait is mutual and
indefinite.

## What was tried

| # | Change | Result |
|---|--------|--------|
| 1 | RoCE AR off (`VLLM_ENABLE_ROCE_ALLREDUCE=0`) | Same deadlock (`0/1821` — one fewer collective request, same stall) |
| 2 | Fresh host JIT-cache dirs (`~/sm121-cache/glm53-jit-121f/*`) | Same — not a stale-lock issue |
| 3 | Watched 30+ min | No progress at all (not slow — dead) |

## Hypotheses (for the lab)

1. The preparation-session rank participation handshake deadlocks on a
   2-rank TP2 aarch64/GB10 topology. The lab's RTX 6000 Pro fleet (x86,
   cu134 wheels) presumably exercises 4/8-rank topologies; a 2-rank GB10
   group may take a path their CI never hits.
2. The vLLM↔b12x step-channel protocol changed on both tips simultaneously
   (new commits: "b12x lazy graph pools", "graph pool capture", "device timing
   without profiling capture", "ii-active-b12x-allreduce-accessor"); a pairing
   or rank-count edge in the new handshake.
3. `B12X_PREPARATION_TRACE_DIR` will produce the JSONL trace of where the
   protocol stalls — not yet collected (see next steps).

## Workaround in use

`glm53-jovian-r35:local` (r35-generation vLLM `de982a50` + its b12x) boots the
same Spark checkpoint cleanly (weights + GDN mapping + b12x preparation all
pass) with `VLLM_DISABLED_KERNELS=FlashInferCutlassMxfp8LinearKernel` set,
which routes mxfp8 linears away from the flashinfer cutlass kernel that rejects
the small GDN GEMMs (`min n = 128 / min k = 128` in `gemm_base.py` — the r35
wall). Engine up: KV 1,406,229 tokens @524K (2.68x), MTP-3 acceptance 3.13,
clean text output.

## Next steps (original 2026-09-19)

- ~~Collect `B12X_PREPARATION_TRACE_DIR` traces~~ — deprioritized: r39 tip
  boots clean, the fix is upstream; traces only if the lab asks.
- ~~Try r38 on 4x~~ — **done 2026-09-22: reproduces (0/1822 stall)**, see
  update above.
- ~~Report to the lab~~ — ready to send (see updated next steps).