# Night shift 2026-09-21 → 22 (tracks A + C + E, approved plan)

## Results (summary — detail in `specs/tech-architecture/glm53-recipe-state.md`)

1. **A — r35 rebuild + registry prep.** Rebuilt `jovian-judgement:r35-glm53`
   on dgx4 (first real `deps.lock` node-side validation → found + fixed a
   latent lock-header bug). Control serve on 2nodes-dac (launcher auto-synced
   the image dgx1→dgx2). **Control bench on v2.7.0: 165/174** (new baseline;
   the dev65-era 160/176 is not comparable across the harness pin).
2. **F — harness re-pin.** tool-eval-bench → **v2.7.0** (stable of the dev65
   line; `tui` extra gone, install is `[perf]`; new `reasoning_truncated`
   failure kind).
3. **E — r39 cut + boot + bench.** Lab moved every external CMake pin in one
   cut (cutlass 4.7.1, DeepGEMM→vllm-project fork, new DeepSelect dep) → new
   lineage dir `jovian-judgement-r39` (r35 tree and r39 tree can't share a
   fetch-deps). Build on dgx3: first fail = new DeepGEMM needs `libdw-dev`
   (elfutils) → apt pin added → clean build. **r39 2x boots clean — the r38
   b12x preparation-pool deadlock is GONE upstream** (comm.roce 33/33 in 0:40
   where r38 froze at 32/33). **Bench: 164/176** ≈ r35 parity. Suspected fix:
   b12x `c2dc1cf022` "Configure RoCE native traffic class for switched Spark
   fabrics".
4. **C — r38 on 4x (evidence for the bug report).** Deadlock **reproduces**,
   different stall point: `0/1822 ready` frozen at the FIRST barrier (2x died
   late, at comm.roce), ranks 1–3 silent, all workers livelock at 48–107%
   CPU. Bug spec updated with the full 2x+4x matrix.
5. Registry go-live: one manual step left — operator `docker login ghcr.io`
   on a node + uncomment `[registry]` (or `sm121 publish --push` later).

## Ops notes

- **`docker save IMG -o -` does not work** (invalid archive); pipe without
  `-o -`.
- Fabric IP slip cost a push: 192.168.200.14 ≠ dgx1 (dgx1 = 200.15/200.16).
  A "Loaded image" appeared in the source log of the mis-addressed push —
  unexplained, the image was NOT on dgx1; re-push to 200.15 landed clean.
- r39 image distribution was manual cross-cluster push (dgx3→dgx1, fabric)
  because dgx3 ∉ 2nodes-dac (the launcher sync only spans the launch set).
- r39 compile phase is ~30 min (new kernel surface) — the
  `shm_broadcast` "No available shared memory broadcast block" spam during
  that window is normal startup work, not a stall.
- Nodes left: dgx1–4 on, no containers; r38/r39/r35 images on dgx1–4 as
  noted in the recipe-state spec; r38 tagged `jovian-judgement:r38` on all
  four.