|
|
@@ -0,0 +1,43 @@
|
|
|
+# Night shift 2026-09-21 → 22 (tracks A + C + E, approved plan)
|
|
|
+
|
|
|
+## Results (summary — detail in `specs/tech-architecture/glm53-recipe-state.md`)
|
|
|
+
|
|
|
+1. **A — r35 rebuild + registry prep.** Rebuilt `jovian-judgement:r35-glm53`
|
|
|
+ on dgx4 (first real `deps.lock` node-side validation → found + fixed a
|
|
|
+ latent lock-header bug). Control serve on 2nodes-dac (launcher auto-synced
|
|
|
+ the image dgx1→dgx2). **Control bench on v2.7.0: 165/174** (new baseline;
|
|
|
+ the dev65-era 160/176 is not comparable across the harness pin).
|
|
|
+2. **F — harness re-pin.** tool-eval-bench → **v2.7.0** (stable of the dev65
|
|
|
+ line; `tui` extra gone, install is `[perf]`; new `reasoning_truncated`
|
|
|
+ failure kind).
|
|
|
+3. **E — r39 cut + boot + bench.** Lab moved every external CMake pin in one
|
|
|
+ cut (cutlass 4.7.1, DeepGEMM→vllm-project fork, new DeepSelect dep) → new
|
|
|
+ lineage dir `jovian-judgement-r39` (r35 tree and r39 tree can't share a
|
|
|
+ fetch-deps). Build on dgx3: first fail = new DeepGEMM needs `libdw-dev`
|
|
|
+ (elfutils) → apt pin added → clean build. **r39 2x boots clean — the r38
|
|
|
+ b12x preparation-pool deadlock is GONE upstream** (comm.roce 33/33 in 0:40
|
|
|
+ where r38 froze at 32/33). **Bench: 164/176** ≈ r35 parity. Suspected fix:
|
|
|
+ b12x `c2dc1cf022` "Configure RoCE native traffic class for switched Spark
|
|
|
+ fabrics".
|
|
|
+4. **C — r38 on 4x (evidence for the bug report).** Deadlock **reproduces**,
|
|
|
+ different stall point: `0/1822 ready` frozen at the FIRST barrier (2x died
|
|
|
+ late, at comm.roce), ranks 1–3 silent, all workers livelock at 48–107%
|
|
|
+ CPU. Bug spec updated with the full 2x+4x matrix.
|
|
|
+5. Registry go-live: one manual step left — operator `docker login ghcr.io`
|
|
|
+ on a node + uncomment `[registry]` (or `sm121 publish --push` later).
|
|
|
+
|
|
|
+## Ops notes
|
|
|
+
|
|
|
+- **`docker save IMG -o -` does not work** (invalid archive); pipe without
|
|
|
+ `-o -`.
|
|
|
+- Fabric IP slip cost a push: 192.168.200.14 ≠ dgx1 (dgx1 = 200.15/200.16).
|
|
|
+ A "Loaded image" appeared in the source log of the mis-addressed push —
|
|
|
+ unexplained, the image was NOT on dgx1; re-push to 200.15 landed clean.
|
|
|
+- r39 image distribution was manual cross-cluster push (dgx3→dgx1, fabric)
|
|
|
+ because dgx3 ∉ 2nodes-dac (the launcher sync only spans the launch set).
|
|
|
+- r39 compile phase is ~30 min (new kernel surface) — the
|
|
|
+ `shm_broadcast` "No available shared memory broadcast block" spam during
|
|
|
+ that window is normal startup work, not a stall.
|
|
|
+- Nodes left: dgx1–4 on, no containers; r38/r39/r35 images on dgx1–4 as
|
|
|
+ noted in the recipe-state spec; r38 tagged `jovian-judgement:r38` on all
|
|
|
+ four.
|
