|
|
@@ -127,15 +127,24 @@ mismatches; per-rank row tiling recomputed from the 1017 snapshot headers by
|
|
|
(card line: "vLLM container: eugr/spark-vllm-b12x:latest") and our arm64
|
|
|
digest matches the current `:latest`; post-build repo commits (10-15/16)
|
|
|
touched recipes/runner only, not the patchset — the image itself was NOT the
|
|
|
- Nemotron-class mistake. Third stall reproduced at a new observation point:
|
|
|
- `b12x routing checkpoint shards: 7/42 … 16.61 GB selected` → dgx3 container
|
|
|
- `Exited (137)`, dgx4 host frozen. The card's own instructions say **"cache
|
|
|
- the image and checkpoint on every node"** — our NAS-streamed load violates
|
|
|
- that. `serve --mirror` was attempted: the per-node copy reached 166G but
|
|
|
- exited 1 with **41/42 safetensors** on disk — (a) the copy must be retried
|
|
|
- to completion, and (b) the launcher's mirror-presence check
|
|
|
- (`test -f <repo>/refs/main`) can pass on a partial copy → the check should
|
|
|
- verify content (shard count or ledger) before trusting the mirror.
|
|
|
+ Nemotron-class mistake. FOUR stalls now, each narrowed the cause:
|
|
|
+ 1. card defaults (GMU 0.87, 500K ctx, NAS) — both nodes froze;
|
|
|
+ 2. reduced (GMU 0.75, 32K, NAS) — both froze;
|
|
|
+ 3. cold (GMU 0.72, 16K, NAS) — dgx3 `Exited (137)` at `b12x routing
|
|
|
+ checkpoint shards: 7/42 … 16.61 GB selected`, dgx4 host frozen;
|
|
|
+ 4. cold + `--mirror` (checkpoint on node-local NVMe, content-verified
|
|
|
+ complete) — **dgx3 froze during routing, dgx4 stayed healthy** (it never
|
|
|
+ reached its own routing pass — the head died first).
|
|
|
+ The NVMe result kills the NFS-page-cache hypothesis: the b12x shard-routing
|
|
|
+ phase itself exhausts the 121.63 GiB unified pool with this 166 GB
|
|
|
+ checkpoint on 2 nodes, regardless of storage. GMU/KV are irrelevant (the
|
|
|
+ death is pre-KV). Remaining unreplicated card-flow details: "start workers
|
|
|
+ before the head", `MASTER_PORT=29501`, `HF_TOKEN`, `--shm-size=16g`,
|
|
|
+ `UCX_NET_DEVICES=<ETH>` (ours: RoCE twins) — none obviously memory-related
|
|
|
+ except possibly the mega-AOT artifact state. Mirror copy also exposed and
|
|
|
+ fixed a launcher hole: the old `refs/main`-only presence check could trust a
|
|
|
+ partial copy; the mirror now verifies entry-count + apparent size against
|
|
|
+ the source and re-copies on mismatch (launcher `59291f3`).
|
|
|
|
|
|
## Open actions
|
|
|
|
