Home Verkennen Help
Inloggen
sm121
/
launcher
2
0
Vork 0
Bestanden Issues 0 Pull-aanvragen 0 Wiki
Aftakking: main
Aftakkingen Labels
launcher
/
history
/
night-shifts
/
2026-09-22.md

2026-09-22.md 2.5 KB
Permalink Geschiedenis Ruwe

Night shift 2026-09-21 → 22 (tracks A + C + E, approved plan)

Results (summary — detail in specs/tech-architecture/glm53-recipe-state.md)

  1. A — r35 rebuild + registry prep. Rebuilt jovian-judgement:r35-glm53 on dgx4 (first real deps.lock node-side validation → found + fixed a latent lock-header bug). Control serve on 2nodes-dac (launcher auto-synced the image dgx1→dgx2). Control bench on v2.7.0: 165/174 (new baseline; the dev65-era 160/176 is not comparable across the harness pin).
  2. F — harness re-pin. tool-eval-bench → v2.7.0 (stable of the dev65 line; tui extra gone, install is [perf]; new reasoning_truncated failure kind).
  3. E — r39 cut + boot + bench. Lab moved every external CMake pin in one cut (cutlass 4.7.1, DeepGEMM→vllm-project fork, new DeepSelect dep) → new lineage dir jovian-judgement-r39 (r35 tree and r39 tree can't share a fetch-deps). Build on dgx3: first fail = new DeepGEMM needs libdw-dev (elfutils) → apt pin added → clean build. r39 2x boots clean — the r38 b12x preparation-pool deadlock is GONE upstream (comm.roce 33/33 in 0:40 where r38 froze at 32/33). Bench: 164/176 ≈ r35 parity. Suspected fix: b12x c2dc1cf022 "Configure RoCE native traffic class for switched Spark fabrics".
  4. C — r38 on 4x (evidence for the bug report). Deadlock reproduces, different stall point: 0/1822 ready frozen at the FIRST barrier (2x died late, at comm.roce), ranks 1–3 silent, all workers livelock at 48–107% CPU. Bug spec updated with the full 2x+4x matrix.
  5. Registry go-live: one manual step left — operator docker login ghcr.io on a node + uncomment [registry] (or sm121 publish --push later).

Ops notes

  • docker save IMG -o - does not work (invalid archive); pipe without -o -.
  • Fabric IP slip cost a push: 192.168.200.14 ≠ dgx1 (dgx1 = 200.15/200.16). A "Loaded image" appeared in the source log of the mis-addressed push — unexplained, the image was NOT on dgx1; re-push to 200.15 landed clean.
  • r39 image distribution was manual cross-cluster push (dgx3→dgx1, fabric) because dgx3 ∉ 2nodes-dac (the launcher sync only spans the launch set).
  • r39 compile phase is ~30 min (new kernel surface) — the shm_broadcast "No available shared memory broadcast block" spam during that window is normal startup work, not a stall.
  • Nodes left: dgx1–4 on, no containers; r38/r39/r35 images on dgx1–4 as noted in the recipe-state spec; r38 tagged jovian-judgement:r38 on all four.