power off: never cut a shared extra (fabric switch/fan) another cluster's live run needs

The fabric switch (switch.sw6_mt) and fan are listed in the extra_entities of
several clusters (8nodes-switch, dgx58-switch, ds4) — they are ONE physical
device shared across them. A whole-cluster 'power off' turned them off
unconditionally, so powering 8nodes-switch down killed the NCCL fabric of a
serve running on dgx58-switch over the same switch (this wedged the Ling fp4
TP4 run today).

off_extras now takes a 'held' set: extras referenced by any OTHER cluster that
has an open run-ledger entry (runlock::open_runs) are kept on. The run ledger
is the offline single-launcher source of truth; a stale open run over-reports
and keeps the device on — the safe direction (never cut a shared resource that
might still be in use). Subset (--node) still keeps ALL extras (the cluster's
own remaining nodes need the fabric). Logs 'keeping <entity> on — still in use
by an active run on cluster(s): ...' per held entity. i18n power-off-extras-held
in all 6 locales; new test off_extras_whole_cluster_keeps_extra_held_by_other_active_cluster.

locales/de/main.ftl

@@ -63,6 +63,7 @@ power-off-waiting = ⏳ warte { $secs}s bis Knoten sich beruhigen
 power-off-no-ha = keine [home_assistant]-Konfig; OS-Herunterfahren oben ausgeführt
 power-off-ssh-failures = einige Knoten meldeten SSH-Fehler beim Herunterfahren
 power-off-extras-skipped = teilweises Ausschalten ({ $nodes }): Cluster-Entitäten bleiben an ({ $extras }) — sie werden von den aktiven Knoten mitgenutzt
+power-off-extras-held = { $entity } bleibt an — noch in Nutzung durch einen aktiven Lauf auf Cluster(n): { $clusters }
 power-on-dryrun = dry-run: tatsächlicher HA-Aufruf übersprungen
 power-on-would-on = würde: HA turn_on { $entity }
 power-on-would-skip = würde überspringen: { $name } (keine Entity)

locales/en/main.ftl

@@ -56,6 +56,7 @@ power-off-waiting = ⏳ waiting { $secs }s for nodes to settle
 power-off-no-ha = no [home_assistant] config; OS shutdown was performed above
 power-off-ssh-failures = some nodes reported SSH failures during shutdown
 power-off-extras-skipped = subset power off ({ $nodes }): keeping cluster-level extras on ({ $extras }) — they are shared by the nodes that stay up
+power-off-extras-held = keeping { $entity } on — still in use by an active run on cluster(s): { $clusters }
 power-on-dryrun = dry-run: skipping actual HA call
 power-on-would-on = would: HA turn_on { $entity }
 power-on-would-skip = would skip: { $name } (no entity)

locales/es/main.ftl

@@ -63,6 +63,7 @@ power-off-waiting = ⏳ esperando { $secs}s para que los nodos se asienten
 power-off-no-ha = sin config [home_assistant]; el apagado de SO se realizó arriba
 power-off-ssh-failures = algunos nodos reportaron fallos SSH durante el apagado
 power-off-extras-skipped = apagado parcial ({ $nodes }): se mantienen las entidades del clúster ({ $extras }) — son compartidas por los nodos activos
+power-off-extras-held = se mantiene { $entity } encendido — aún en uso por una ejecución activa en el/los clúster(es): { $clusters }
 power-on-dryrun = dry-run: omitiendo llamada HA real
 power-on-would-on = sería: HA turn_on { $entity }
 power-on-would-skip = se omitiría: { $name } (sin entity)

locales/fr/main.ftl

@@ -63,6 +63,7 @@ power-off-waiting = ⏳ attente { $secs}s pour stabilisation des nœuds
 power-off-no-ha = pas de config [home_assistant] ; l'extinction OS a été effectuée ci-dessus
 power-off-ssh-failures = certains nœuds ont signalé des erreurs SSH pendant l'extinction
 power-off-extras-skipped = extinction partielle ({ $nodes }) : entités du cluster laissées activées ({ $extras }) — partagées par les nœuds restants
+power-off-extras-held = { $entity } laissée activée — encore utilisée par un service actif sur le(s) cluster(s) : { $clusters }
 power-on-dryrun = dry-run : appel HA réel ignoré
 power-on-would-on = serait : HA turn_on { $entity }
 power-on-would-skip = serait ignoré : { $name } (pas d'entity)

locales/it/main.ftl

@@ -63,6 +63,7 @@ power-off-waiting = ⏳ attesa { $secs}s per assestamento dei nodi
 power-off-no-ha = nessuna config [home_assistant]; spegnimento OS eseguito sopra
 power-off-ssh-failures = alcuni nodi hanno riportato errori SSH durante lo spegnimento
 power-off-extras-skipped = spegnimento parziale ({ $nodes }): entità del cluster lasciate accese ({ $extras }) — condivise dai nodi attivi
+power-off-extras-held = { $entity } lasciata accesa — ancora in uso da un'esecuzione attiva sul/i cluster: { $clusters }
 power-on-dryrun = dry-run: saltando chiamata HA reale
 power-on-would-on = sarebbe: HA turn_on { $entity }
 power-on-would-skip = saltato: { $name } (nessuna entity)

locales/ru/main.ftl

@@ -64,6 +64,7 @@ power-off-waiting = ⏳ ждем { $secs }с для завершения ост
 power-off-no-ha = нет [home_assistant] конфига; OS shutdown выполнен
 power-off-ssh-failures = некоторые узлы сообщили об ошибках SSH при остановке
 power-off-extras-skipped = точечное выключение ({ $nodes }): cluster-сущности оставлены включёнными ({ $extras }) — они общие для оставшихся узлов
+power-off-extras-held = { $entity } оставлено включённым — используется активным запуском на кластере(ах): { $clusters }
 power-on-dryrun = dry-run: пропускаем реальный HA вызов
 power-on-would-on = будет: HA turn_on { $entity }
 power-on-would-skip = будет пропущено: { $name } (нет entity)

src/power.rs

@@ -34,9 +34,13 @@ pub async fn power_off(
     // Cluster-level extras (the fabric switch, a cooling fan) are SHARED by
     // every node in the cluster.  Powering off a SUBSET (`--node`) must not
     // take the fabric down for the nodes that stay up — that wedges live
-    // multi-node serves (NCCL over the switch dies).  So extras are toggled
-    // off only for a whole-cluster power off (no `--node`).
-    let extras = off_extras(cluster, node_names);
+    // multi-node serves (NCCL over the switch dies).  And even a whole-cluster
+    // off must keep any extra that ANOTHER cluster with an open run references
+    // (the fabric switch is shared across clusters — powering `8nodes-switch`
+    // down must not kill a serve on `dgx58-switch` over the same switch).
+    let held = toggle::extras_held_by_other_clusters(config, cluster);
+    let held_ids: std::collections::HashSet<String> = held.keys().cloned().collect();
+    let extras = off_extras(cluster, node_names, &held_ids);
     if !node_names.is_empty() && !cluster.extra_entities.is_empty() {
         info!(
             "{}",
@@ -46,6 +50,19 @@ pub async fn power_off(
             })
         );
     }
+    if node_names.is_empty() {
+        for e in extra_ids(cluster) {
+            if let Some(clusters) = held.get(&e) {
+                info!(
+                    "{}",
+                    fluent_i18n::t!("power-off-extras-held", {
+                        "entity" => e.clone(),
+                        "clusters" => clusters.join(", ")
+                    })
+                );
+            }
+        }
+    }
 
     if dry_run {
         warn!("{}", fluent_i18n::t!("power-off-dryrun"));

src/power/tests.rs

@@ -113,19 +113,37 @@ fn cluster_extra_entities_parsed() {
 
 #[test]
 fn off_extras_whole_cluster_keeps_extras_subset_skips_them() {
+    use std::collections::HashSet;
+    let empty: HashSet<String> = HashSet::new();
     // A whole-cluster power off (no --node) turns the shared fabric off too.
     let cfg = cfg_with_ha();
     let c = crate::config::select_cluster_by_name(&cfg, "x").unwrap();
-    assert_eq!(off_extras(&c, &[]), vec!["switch.network_x".to_string()]);
+    assert_eq!(off_extras(&c, &[], &empty), vec!["switch.network_x".to_string()]);
     // A subset power off (--node) must NOT take the shared switch down —
     // the nodes that stay up depend on it (NCCL over the fabric).
-    assert_eq!(off_extras(&c, &["n1".to_string()]), Vec::<String>::new());
+    assert_eq!(off_extras(&c, &["n1".to_string()], &empty), Vec::<String>::new());
     assert_eq!(
-        off_extras(&c, &["n1".to_string(), "n2".to_string()]),
+        off_extras(&c, &["n1".to_string(), "n2".to_string()], &empty),
         Vec::<String>::new()
     );
 }
 
+#[test]
+fn off_extras_whole_cluster_keeps_extra_held_by_other_active_cluster() {
+    // The fabric switch is shared across clusters. A whole-cluster off of `x`
+    // must NOT turn off `switch.network_x` while another cluster that also
+    // lists it has an open run — that serve depends on the same switch.
+    use std::collections::HashSet;
+    let cfg = cfg_with_ha();
+    let c = crate::config::select_cluster_by_name(&cfg, "x").unwrap();
+    let held: HashSet<String> = ["switch.network_x".to_string()].into_iter().collect();
+    // held → the shared switch is kept on, nothing is turned off
+    assert_eq!(off_extras(&c, &[], &held), Vec::<String>::new());
+    // an unrelated held entity does not protect this cluster's switch
+    let other: HashSet<String> = ["switch.somewhere_else".to_string()].into_iter().collect();
+    assert_eq!(off_extras(&c, &[], &other), vec!["switch.network_x".to_string()]);
+}
+
 #[test]
 fn extra_entity_forms_blocking_and_soft() {
     use crate::config::ExtraEntity;

src/power/toggle.rs

@@ -3,9 +3,10 @@
 
 use super::HaClient;
 use super::mikro;
-use crate::config::{Cluster, MikrotikConfig, Node};
+use crate::config::{Cluster, Config, MikrotikConfig, Node};
 use crate::home_assistant::split_entity;
 use anyhow::Result;
+use std::collections::{HashMap, HashSet};
 use tracing::{info, warn};
 
 /// Entity ids of a cluster's `extra_entities` (both forms).
@@ -17,17 +18,57 @@ pub(crate) fn extra_ids(cluster: &Cluster) -> Vec<String> {
         .collect()
 }
 
-/// Cluster-level extras to toggle OFF for a power off: the full list only for
-/// a whole-cluster off (no `--node`); empty for a subset.  The fabric switch
-/// and fan are shared by every node, so powering off one node must not take
-/// the cluster's fabric down for the nodes that stay up (it wedges live
-/// multi-node serves — NCCL over the switch dies).
-pub(crate) fn off_extras(cluster: &Cluster, node_names: &[String]) -> Vec<String> {
-    if node_names.is_empty() {
-        extra_ids(cluster)
-    } else {
-        Vec::new()
+/// Extra-entity ids that must stay ON because **another** cluster (≠
+/// `target`) has an open run-ledger entry that references them. The fabric
+/// switch and cooling fan are shared across clusters (e.g. `8nodes-switch`
+/// and `dgx58-switch` both list `switch.sw6_mt`), so powering one cluster
+/// down must not cut the fabric a live serve on a *different* cluster depends
+/// on — that wedges its NCCL and kills whatever it is serving. Maps each
+/// entity to the active other-cluster names that hold it. The run ledger is
+/// the offline source of truth (the single-launcher serialization point); a
+/// stale open run over-reports and keeps the entity on — the safe direction
+/// (never cut a shared resource that might still be in use).
+pub(crate) fn extras_held_by_other_clusters(config: &Config, target: &Cluster) -> HashMap<String, Vec<String>> {
+    let active: HashSet<String> = crate::runlock::open_runs(config)
+        .into_iter()
+        .filter_map(|(_, _, cluster)| cluster)
+        .filter(|c| c != &target.name)
+        .collect();
+    let mut held: HashMap<String, Vec<String>> = HashMap::new();
+    for c in &config.clusters {
+        if !active.contains(&c.name) {
+            continue;
+        }
+        for e in &c.extra_entities {
+            held.entry(e.entity().to_string()).or_default().push(c.name.clone());
+        }
     }
+    held
+}
+
+/// Cluster-level extras to toggle OFF for a power off.  Two guards keep a
+/// shared resource (fabric switch, fan) from being cut out from under a live
+/// serve:
+/// 1. A SUBSET power off (`--node`) never touches extras — the nodes of this
+///    cluster that stay up still need the fabric (NCCL over the switch dies
+///    otherwise).
+/// 2. A WHOLE-cluster power off still keeps any extra that another cluster
+///    with an open run references (`held`, from
+///    [`extras_held_by_other_clusters`]) — the fabric is shared across
+///    clusters, so powering `8nodes-switch` down must not kill a serve
+///    running on `dgx58-switch` over the same switch.
+pub(crate) fn off_extras(
+    cluster: &Cluster,
+    node_names: &[String],
+    held: &HashSet<String>,
+) -> Vec<String> {
+    if !node_names.is_empty() {
+        return Vec::new();
+    }
+    extra_ids(cluster)
+        .into_iter()
+        .filter(|e| !held.contains(e))
+        .collect()
 }
 
 /// Handle one entity id for a power toggle: dispatch `[mikrotik]`