BUG-2026-09-19-r38-b12x-preparation-pool-deadlock.md 6.3 KB
BUG-2026-09-19 — r38 (JJ tip + b12x master) boot deadlock: b12x preparation pool stuck at "waiting for ranks"
Date: 2026-09-19
Lane: GLM-5.3-Flash-NVFP4-Spark, 2x GB10 (2nodes-dac), TP2
Image: glm53-jovian-r38:local — vLLM local-inference-lab/vllm @ 8e1f1e587f8 (tree d9f35de1), b12x local-inference-lab/b12x @ 0f3a8cbfd1c (tree ff2218d7)
Severity: high (blocks all boot on this engine pair; model loading itself is fine)
Status: open (worked around with r35 for now; r39 tip verified fixed on 2x, 2026-09-22)
2026-09-22 update — 4-rank reproduction + r39 fix verified
4x reproduction (Track C, this night). Same image on 4nodes-switch
(dgx1–4, 200G switched fabric, TP4) deadlocks at the very start of the
preparation session — different stall point, same livelock:
b12x waiting for ranks: 0/1822 ready, 0 measured, 0 cached, 0 compilations, 3:52
b12x waiting for ranks: 0/1822 ready, 0 measured, 0 cached, 0 compilations, 4:02
... (80+ lines over 13+ min, counter never moves)
- Ranks 1–3: zero log output after the 02:43:33 startup lines; Worker_TP spinning at ~50% CPU (48.9 / 54.4 / 51.7 over the observation window).
- Rank 0: Worker_TP ~107% CPU emitting the 10 s poll line; system 95% idle otherwise.
- So the stall is not specific to the late
comm.rocephase seen on 2x (there the pool reached 3806 measured / 32-of-33 ready before freezing on the final rendezvous) — on 4 ranks it dies at the first barrier (0/1822). Both modes are the same mutualwaiting for rankslivelock with all workers busy-spinning.
Fix verified in tip (r39). jovian-judgement:r39-glm53 (vLLM
local-inference-lab/vllm @ 9e5d1793fa3, b12x @ c2dc1cf022 "Configure RoCE
native traffic class for switched Spark fabrics" + ~100 commits over r38)
boots the same Spark checkpoint cleanly on 2x: the comm.roce phase that
froze r38 at 32/33 completes 33/33 ready in 0:40, engine up, API serving,
bench 164/176 (parity with the r35 baseline 165/174, v2.7.0). The RoCE traffic
class commit targets exactly our switched Spark fabric — strong candidate
cause for the 2x stall; the 4x 0/1822 stall suggests the preparation-session
handshake also has a rank-count/topology edge beyond RoCE TC.
Next steps
- Report to the lab with the 2x + 4x reproduction matrix (both r38 pins, GB10
aarch64; signature lines above; CPU-spin evidence) and the r39 boot as
evidence the tip clears it. Ask them to confirm whether the RoCE TC commit
(b12x
c2dc1cf022) is the intended fix and whether the 4x 0/1822 handshake edge is known. - If r39 is adopted, the 4x r39 lane becomes the next perf target (r38 4x was never measurable).
- Optional:
B12X_PREPARATION_TRACE_DIRtraces from an r38 boot if the lab wants the exact stalled requirement (CollectiveRequirement vs TuningCacheRequirement).
Symptom
Boot gets through weight loading (InstantTensor, 88 GiB/rank, clean — no
KeyError, the JJ tip maps the Spark checkpoint's GDN naming fine) and
immediately after Using BLHNC KV cache layout the rank-0 log freezes on:
b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 0:00
b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 9:31
... (indefinitely; 30+ min observed)
0/1821 = completed_requests/total_requests of the b12x kernel preparation
session (1821 preparation requests; 1822 with VLLM_ENABLE_ROCE_ALLREDUCE=1).
The phase label maps to the WAIT stage in b12x/preparation/_progress.py.
Observed state at steady state
- rank 0:
VLLM::Worker_TP~105% CPU,VLLM::EngineCore~19% CPU — busy-spin (the waiting loop is a polling loop, not a futex sleep). - rank 1:
VLLM::Worker_TP~60% CPU, main threadR / wait_woken, all 66 threads otherwiseS; no new log output after the same "Model loading took 88.07 GiB" line. - No compiler processes anywhere on either node (no
c++/nvcc/tvm-ffi subprocesses) — the pool never starts a single compilation (0 compilations,0 cached,0 measured). - NCCL/gloo watchdog threads idle; no crash, no OOM, no error line on either rank.
Both ranks wait for each other in the vLLM↔b12x preparation protocol
(session.py: the job blocks on CollectiveRequirement /
TuningCacheRequirement until "participating ranks" contribute; the progress
phase flips to "waiting for ranks" as soon as any ready_* flag is set while
done is false). Neither side ever makes progress; the wait is mutual and
indefinite.
What was tried
| # | Change | Result |
|---|---|---|
| 1 | RoCE AR off (VLLM_ENABLE_ROCE_ALLREDUCE=0) |
Same deadlock (0/1821 — one fewer collective request, same stall) |
| 2 | Fresh host JIT-cache dirs (~/sm121-cache/glm53-jit-121f/*) |
Same — not a stale-lock issue |
| 3 | Watched 30+ min | No progress at all (not slow — dead) |
Hypotheses (for the lab)
- The preparation-session rank participation handshake deadlocks on a 2-rank TP2 aarch64/GB10 topology. The lab's RTX 6000 Pro fleet (x86, cu134 wheels) presumably exercises 4/8-rank topologies; a 2-rank GB10 group may take a path their CI never hits.
- The vLLM↔b12x step-channel protocol changed on both tips simultaneously (new commits: "b12x lazy graph pools", "graph pool capture", "device timing without profiling capture", "ii-active-b12x-allreduce-accessor"); a pairing or rank-count edge in the new handshake.
B12X_PREPARATION_TRACE_DIRwill produce the JSONL trace of where the protocol stalls — not yet collected (see next steps).
Workaround in use
glm53-jovian-r35:local (r35-generation vLLM de982a50 + its b12x) boots the
same Spark checkpoint cleanly (weights + GDN mapping + b12x preparation all
pass) with VLLM_DISABLED_KERNELS=FlashInferCutlassMxfp8LinearKernel set,
which routes mxfp8 linears away from the flashinfer cutlass kernel that rejects
the small GDN GEMMs (min n = 128 / min k = 128 in gemm_base.py — the r35
wall). Engine up: KV 1,406,229 tokens @524K (2.68x), MTP-3 acceptance 3.13,
clean text output.
Next steps (original 2026-09-19)
Collect— deprioritized: r39 tip boots clean, the fix is upstream; traces only if the lab asks.B12X_PREPARATION_TRACE_DIRtracesTry r38 on 4x— done 2026-09-22: reproduces (0/1822 stall), see update above.Report to the lab— ready to send (see updated next steps).