BUG-2026-09-19 — r38 (JJ tip + b12x master) boot deadlock: b12x preparation pool stuck at "waiting for ranks"

Date: 2026-09-19 Lane: GLM-5.3-Flash-NVFP4-Spark, 2x GB10 (2nodes-dac), TP2 Image: glm53-jovian-r38:local — vLLM local-inference-lab/vllm @ 8e1f1e587f8 (tree d9f35de1), b12x local-inference-lab/b12x @ 0f3a8cbfd1c (tree ff2218d7) Severity: high (blocks all boot on this engine pair; model loading itself is fine) Status: open (worked around with r35 for now)

Symptom

Boot gets through weight loading (InstantTensor, 88 GiB/rank, clean — no KeyError, the JJ tip maps the Spark checkpoint's GDN naming fine) and immediately after Using BLHNC KV cache layout the rank-0 log freezes on:

b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 0:00
b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 9:31
... (indefinitely; 30+ min observed)

0/1821 = completed_requests/total_requests of the b12x kernel preparation session (1821 preparation requests; 1822 with VLLM_ENABLE_ROCE_ALLREDUCE=1). The phase label maps to the WAIT stage in b12x/preparation/_progress.py.

Observed state at steady state

rank 0: VLLM::Worker_TP ~105% CPU, VLLM::EngineCore ~19% CPU — busy-spin (the waiting loop is a polling loop, not a futex sleep).
rank 1: VLLM::Worker_TP ~60% CPU, main thread R / wait_woken, all 66 threads otherwise S; no new log output after the same "Model loading took 88.07 GiB" line.
No compiler processes anywhere on either node (no c++/nvcc/tvm-ffi subprocesses) — the pool never starts a single compilation (0 compilations, 0 cached, 0 measured).
NCCL/gloo watchdog threads idle; no crash, no OOM, no error line on either rank.

Both ranks wait for each other in the vLLM↔b12x preparation protocol (session.py: the job blocks on CollectiveRequirement / TuningCacheRequirement until "participating ranks" contribute; the progress phase flips to "waiting for ranks" as soon as any ready_* flag is set while done is false). Neither side ever makes progress; the wait is mutual and indefinite.

What was tried

#	Change	Result
1	RoCE AR off (`VLLM_ENABLE_ROCE_ALLREDUCE=0`)	Same deadlock (`0/1821` — one fewer collective request, same stall)
2	Fresh host JIT-cache dirs (`~/sm121-cache/glm53-jit-121f/*`)	Same — not a stale-lock issue
3	Watched 30+ min	No progress at all (not slow — dead)

Hypotheses (for the lab)

The preparation-session rank participation handshake deadlocks on a 2-rank TP2 aarch64/GB10 topology. The lab's RTX 6000 Pro fleet (x86, cu134 wheels) presumably exercises 4/8-rank topologies; a 2-rank GB10 group may take a path their CI never hits.
The vLLM↔b12x step-channel protocol changed on both tips simultaneously (new commits: "b12x lazy graph pools", "graph pool capture", "device timing without profiling capture", "ii-active-b12x-allreduce-accessor"); a pairing or rank-count edge in the new handshake.
B12X_PREPARATION_TRACE_DIR will produce the JSONL trace of where the protocol stalls — not yet collected (see next steps).

Workaround in use

glm53-jovian-r35:local (r35-generation vLLM de982a50 + its b12x) boots the same Spark checkpoint cleanly (weights + GDN mapping + b12x preparation all pass) with VLLM_DISABLED_KERNELS=FlashInferCutlassMxfp8LinearKernel set, which routes mxfp8 linears away from the flashinfer cutlass kernel that rejects the small GDN GEMMs (min n = 128 / min k = 128 in gemm_base.py — the r35 wall). Engine up: KV 1,406,229 tokens @524K (2.68x), MTP-3 acceptance 3.13, clean text output.

Next steps

Collect B12X_PREPARATION_TRACE_DIR traces from both ranks of an r38 boot to pinpoint the stalled requirement (CollectiveRequirement vs TuningCacheRequirement vs _TuningBatch).
Try r38 on 4x (4-rank group) to test the rank-count hypothesis.
Report to the lab (b12x preparation protocol; their fleet may not cover 2-rank aarch64).

BUG-2026-09-19-r38-b12x-preparation-pool-deadlock.md 4.0 KB Histórico Raw