history: 386B OOM-loop required HA power cycle (respawning workers); cluster recovered clean

history/night-shifts/2026-09-08.md

@@ -133,8 +133,10 @@ hidden 6144, 8 experts/tok, 1M context): **406 GiB of safetensors** on disk
 (84 shards), ~101.5 GiB per node on 4x. First load pass completed (294 s at
 1.5 GB/s), the engine then restarted and began a SECOND full load while the
 first copy was still resident → 2×101.5 > 121.63 GiB → all four nodes went
-OOM-thrashing (0 GiB available, sshd unreachable on 3 of 4). Kill + wait for
-the OOM killer; nodes recovering slowly (swap thrash).
+OOM-thrashing (0 GiB available, sshd unreachable on 3 of 4). Kill issued; the engine kept RESPAWNING OOM-killed workers (the container
+stayed up), so the nodes never recovered on their own — operator approved a
+power cycle via HA (switch.dgx1-4 off → on, 2026-09-09 ~05:30 UTC). All four
+back clean: 116–117 GiB available, 0 sm121 containers.
 
 **Answer to the operator: 4 nodes is not enough** — a single copy already
 leaves ~20 GiB/node, which the runtime (vLLM processes ~12 GiB + KV +

recipes/glm53-flash-r28-spark/reports/2026-09-09_053925_r28-2x-spark.log

@@ -0,0 +1,8 @@
+{"event": "backend_detected", "backend": "vllm"}
+
+╭──────────────────── ⚡ llama-benchy Throughput Benchmark ────────────────────╮
+│ glm53                                                                        │
+│ pp=[2048]  tg=[128]  depth=[0, 4096, 8192]  concurrency=[1, 2]  runs=3       │
+│ latency=generation                                                           │
+╰──────────────────────────────────────────────────────────────────────────────╯
+