|
|
@@ -148,21 +148,26 @@ mismatches; per-rank row tiling recomputed from the 1017 snapshot headers by
|
|
|
|
|
|
## Open actions
|
|
|
|
|
|
-0. **RESOLVED (2026-10-17 evening): the lane is servable — via the card's
|
|
|
- manual flow.** Replicating the card byte-for-byte (docker run --privileged
|
|
|
- + --shm-size=16g + the card's env incl. UCX_NET_DEVICES=ETH iface +
|
|
|
- MASTER_PORT=29501, **worker started before the head**, node-local
|
|
|
- checkpoint via the verified mirror) booted the lane on dgx3/dgx4: routing
|
|
|
- passed 42/42 (91.99 GB selected) where every launcher attempt died at
|
|
|
- 7/42, and the bench scored **92/100 (178/194, ★★★★★)** vs the kindling 2x
|
|
|
- anchor 95/100. The launcher-flow stall is therefore an integration delta,
|
|
|
- not a stack/hardware limit: candidates (in env-diff order) — UCX on ETH vs
|
|
|
- RoCE twins, simultaneous vs workers-first start, launcher-added NCCL env
|
|
|
- (NCCL_PROTO/CROSS_NIC/GID_INDEX=auto), MASTER_PORT. Bisecting that delta
|
|
|
- is the remaining launcher work item; the manual scripts live on the nodes
|
|
|
- (`/home/kilork/glm-head.sh`, `glm-worker.sh`; containers `glm-manual-*`,
|
|
|
- port 30000, served id `glm53-fp4-local` — NOT runlock-tracked,
|
|
|
- single-launcher rule caveat).
|
|
|
+0. **FULLY RESOLVED (2026-10-17 evening): the launcher now serves the lane.**
|
|
|
+ The bisect exposed the real root causes, none of which were the b12x stack:
|
|
|
+ 1. the recipe lacked `distributed_backend` → the launcher emitted NO dist
|
|
|
+ flags (--nnodes/--node-rank/--master-addr/--tensor-parallel-size) and
|
|
|
+ each node booted an INDEPENDENT single-node engine loading the FULL
|
|
|
+ 166 GB checkpoint into a 121 GiB box — the four "stalls";
|
|
|
+ 2. after adding the backend, the launcher passed the node's GLOBAL cluster
|
|
|
+ rank (dgx3=2, dgx4=3) as `--node-rank` with `--nnodes 2` — vLLM rejects
|
|
|
+ it; fixed by renumbering ranks to the serve-local index in start.rs;
|
|
|
+ 3. the cold GMU 0.72 left only 0.47 GiB KV vs 1.07 needed at 16384 ctx —
|
|
|
+ raised to 0.75 (the boot-safe floor for the correct ~87 GiB/rank
|
|
|
+ footprint).
|
|
|
+ Launcher commits: multi-node-without-backend guard `4029b28`, rank
|
|
|
+ renumber `d7a8ee2`; recipe fixes `c5a8444`/`5dc7678`. The lane now boots
|
|
|
+ through `sm121 serve --recipe glm53-flash-fp4-local-2x --cluster
|
|
|
+ 4nodes-switch --cold --mirror` in ~4 min (warm b12x cache) and is
|
|
|
+ runlock-tracked. The manual card flow (B1/B2a experiments) had confirmed
|
|
|
+ the stack itself was fine — UCX value and --shm-size ruled out as causes;
|
|
|
+ the "workers before the head" start order is NOT required (the launcher's
|
|
|
+ simultaneous start works once the flags are right).
|
|
|
|
|
|
1. Patch `REASONING_EFFORT_MAPPINGS` (50/75/100) via idempotent `[mods]` in the
|
|
|
1017 recipe (or wait for Aiden 1.7.2); document the env-100 default.
|
