mimo26-flash-mopd-2x: config C verified — 512K context live (pool 570,752)

Shift note for the Mikrotik relay fix + MOPD production cutover.
Bench B stays valid: quality is GMU-insensitive on this stack (92/100,
91/100 at 0.85/0.88); 0.90 trades ~2 GB host headroom for +89K KV tokens.

history/night-shifts/2026-09-27.md

@@ -340,3 +340,35 @@ DS4.1 lanes (TC-68 passed this time).
 - config A (GMU 0.85, pool 306,752): 170/184 = 92/100 ★★★★★ (mopd-base)
 - config B (GMU 0.88, pool 498,752 = context 327,680 + 52% headroom): 168/184 = 91/100 — noise; perf/responsiveness unchanged. 0.88 is free, keep.
 - MOPD > Flash-RL by +9 pts (161→170) at equal pool; Pro-MOPD (573.5 GB) 535/573 GB on dgx1 via hfrs — hfrs died at ~99%, the remaining ~39 GB (incl. draft shards) pending resume.
+
+## Mikrotik power-off relay fix (`5c82201`)
+
+`power off` for a `[mikrotik]` device only sent API `/system/shutdown` — the
+device stayed powered and could boot back (and `power on` became a no-op).
+`shutdown_named` now also flips the `on_entity` outlet via HA right after the
+API shutdown (warn-only). Verified: SW2-MUC (192.168.88.39) finally silent.
+The real off-switch for router SW2 is the `switch.sw6_mt` relay.
+
+## MOPD production serve + config C — 512K context verified (28 Sep)
+
+Cutover: aiden16 stopped, 4nodes-switch powered off (router incl. relay),
+2nodes-dac up for MOPD. dgx5 untouched (qwen3.8, user agents).
+- Config C (GMU 0.90, `--context-length 524288`): full pool **570,752** +
+  SWA 57,024 — the 512K target met with ~109K headroom.
+- 515,001-token completion: prefill + decode clean in 625 s (~825 t/s);
+  inputs >524,288 rejected with the clean context-length error. Host free
+  after the run 7.0/8.1 GB per node, ~8 GB GPU headroom — 0.90 is the safe
+  ceiling, no OOM pressure. Smoke OK, decode ~100 t/s c1.
+- Today's prod lane: `mimo26-flash-mopd-2x` on 2nodes-dac (dgx1+dgx2).
+
+## serve: one re-probe before "too few free nodes" (`848f902`)
+
+Operator-flagged: serve kept failing "requires 2 nodes, but cluster X has
+only 1" — twice today. Root cause is NOT stop-wait (post-stop `docker ps`
+was clean each time): the fleet probe is single-shot, and one ssh blip /
+a node still warming up after stop or power-on drops that node from the
+free set. `prepare_cluster` now sleeps `BootTiming::retry_grace_secs`
+(5 s default) and re-probes + re-selects once before bailing, and the
+error lists per-node state (free/occupied/unreachable) instead of a bare
+count. Tests: `flaky_probe_recovers_on_retry`,
+`insufficient_nodes_error_names_the_reason` (→ `orchestrate/retry_tests.rs`).

recipes/mimo26-flash-mopd-2x/recipe.toml

@@ -26,6 +26,8 @@
 # 2026-09-27); expect ~300–340K. Quality/perf of the RL lane
 # (recipes/mimo26-flash-rl-2x/reports, 161/184 on v2.7.1.dev7) is the
 # comparison baseline until this lane has its own bench.
+# Config C (2026-09-28): GMU 0.90 + --context-length 524288 → full pool
+# 570,752 tokens, the 512K target met (see the args note for the probe).
 
 engine = "sglang"
 name = "MiMo-V2.6-Flash-MOPD (SGLang DFlash, 2x Spark TP2)"
@@ -51,12 +53,13 @@ args = [
     "--enable-multimodal",
     "--moe-runner-backend", "marlin",
     "--moe-a2a-backend", "none",
-    # Memory / scheduling (128 GB unified per node). Config C (2026-09-28):
-    # 0.88 → 0.90 for the 512K target — pool at 0.88 was 481,600, short of
-    # 524,288 by ~43K tokens. Measured: pool grows ~50K tokens per 1 GB of
-    # static budget (~1.9 GB per 0.01 GMU), so 0.90 should land ~530-600K.
-    # Activation headroom after static at 0.90: ~11 GB/node — same regime as
-    # the 0.85 boots where long prefills never OOM'd.
+    # Memory / scheduling (128 GB unified per node). Config C (2026-09-28,
+    # VERIFIED): 0.88 → 0.90 for the 512K target — pool at 0.88 was 481,600,
+    # short of 524,288 by ~43K tokens. Measured at 0.90: full pool 570,752
+    # tokens (+ SWA 57,024), context_len 524288 accepted; a 515,001-token
+    # completion prefilled + decoded clean (625 s ≈ 825 t/s), >524,288 inputs
+    # rejected with the clean context-length error. Host free after the run
+    # 7.0/8.1 GB per node, ~8 GB GPU headroom left — no OOM pressure.
     "--mem-fraction-static", "0.90",
     # Model is natively 1M; the measured pool at GMU 0.85 is 340,736 tokens
     # (2x TP2: 83 GB weights/rank leaves ~25 GB/rank; 39/48 SWA layers keep it