# BUG-2026-09-19 — r38 (JJ tip + b12x master) boot deadlock: b12x preparation pool stuck at "waiting for ranks"

**Date:** 2026-09-19
**Lane:** GLM-5.3-Flash-NVFP4-Spark, 2x GB10 (2nodes-dac), TP2
**Image:** `glm53-jovian-r38:local` — vLLM `local-inference-lab/vllm` @ `8e1f1e587f8` (tree `d9f35de1`), b12x `local-inference-lab/b12x` @ `0f3a8cbfd1c` (tree `ff2218d7`)
**Severity:** high (blocks all boot on this engine pair; model loading itself is fine)
**Status:** open (worked around with r35 for now)

## Symptom

Boot gets through weight loading (InstantTensor, 88 GiB/rank, clean — **no
`KeyError`, the JJ tip maps the Spark checkpoint's GDN naming fine**) and
immediately after `Using BLHNC KV cache layout` the rank-0 log freezes on:

```
b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 0:00
b12x waiting for ranks: 0/1821 ready, 0 measured, 0 cached, 0 compilations, 9:31
... (indefinitely; 30+ min observed)
```

`0/1821` = `completed_requests/total_requests` of the b12x **kernel preparation
session** (1821 preparation requests; 1822 with `VLLM_ENABLE_ROCE_ALLREDUCE=1`).
The phase label maps to the `WAIT` stage in `b12x/preparation/_progress.py`.

## Observed state at steady state

- rank 0: `VLLM::Worker_TP` ~105% CPU, `VLLM::EngineCore` ~19% CPU — busy-spin
  (the waiting loop is a polling loop, not a futex sleep).
- rank 1: `VLLM::Worker_TP` ~60% CPU, main thread `R / wait_woken`, all 66
  threads otherwise `S`; **no new log output** after the same
  "Model loading took 88.07 GiB" line.
- **No compiler processes** anywhere on either node (no `c++`/`nvcc`/tvm-ffi
  subprocesses) — the pool never starts a single compilation
  (`0 compilations`, `0 cached`, `0 measured`).
- NCCL/gloo watchdog threads idle; no crash, no OOM, no error line on either rank.

Both ranks wait for each other in the vLLM↔b12x preparation protocol
(`session.py`: the job blocks on `CollectiveRequirement` /
`TuningCacheRequirement` until "participating ranks" contribute; the progress
phase flips to "waiting for ranks" as soon as any ready_* flag is set while
`done` is false). Neither side ever makes progress; the wait is mutual and
indefinite.

## What was tried

| # | Change | Result |
|---|--------|--------|
| 1 | RoCE AR off (`VLLM_ENABLE_ROCE_ALLREDUCE=0`) | Same deadlock (`0/1821` — one fewer collective request, same stall) |
| 2 | Fresh host JIT-cache dirs (`~/sm121-cache/glm53-jit-121f/*`) | Same — not a stale-lock issue |
| 3 | Watched 30+ min | No progress at all (not slow — dead) |

## Hypotheses (for the lab)

1. The preparation-session rank participation handshake deadlocks on a
   2-rank TP2 aarch64/GB10 topology. The lab's RTX 6000 Pro fleet (x86,
   cu134 wheels) presumably exercises 4/8-rank topologies; a 2-rank GB10
   group may take a path their CI never hits.
2. The vLLM↔b12x step-channel protocol changed on both tips simultaneously
   (new commits: "b12x lazy graph pools", "graph pool capture", "device timing
   without profiling capture", "ii-active-b12x-allreduce-accessor"); a pairing
   or rank-count edge in the new handshake.
3. `B12X_PREPARATION_TRACE_DIR` will produce the JSONL trace of where the
   protocol stalls — not yet collected (see next steps).

## Workaround in use

`glm53-jovian-r35:local` (r35-generation vLLM `de982a50` + its b12x) boots the
same Spark checkpoint cleanly (weights + GDN mapping + b12x preparation all
pass) with `VLLM_DISABLED_KERNELS=FlashInferCutlassMxfp8LinearKernel` set,
which routes mxfp8 linears away from the flashinfer cutlass kernel that rejects
the small GDN GEMMs (`min n = 128 / min k = 128` in `gemm_base.py` — the r35
wall). Engine up: KV 1,406,229 tokens @524K (2.68x), MTP-3 acceptance 3.13,
clean text output.

## Next steps

- Collect `B12X_PREPARATION_TRACE_DIR` traces from both ranks of an r38 boot
  to pinpoint the stalled requirement (CollectiveRequirement vs
  TuningCacheRequirement vs _TuningBatch).
- Try r38 on 4x (4-rank group) to test the rank-count hypothesis.
- Report to the lab (b12x preparation protocol; their fleet may not cover
  2-rank aarch64).