Prechádzať zdrojové kódy

fix: resolve review findings across modules

Address the non-SSH findings from CODE_REVIEW.md.

Bugs:
- `before_serve::truncate` sliced by byte (`&s[..max]`) and panicked on
  multibyte UTF-8 — any non-ASCII pre-serve command (Cyrillic/CJK paths,
  en-dashes) over 120 bytes crashed the launcher. Slice by char instead.
- `build_image_on_node` polled `tmux has-session` with no upper bound, so
  a hung build looped the 30s poll forever and hung the launcher. Add a
  configurable `max_build_secs` timeout (default 90 min) with a clean
  bail pointing at the tmux session and log.
- `stop --node a --node b --off` only powered off the first node because
  `power_off` took a single `Option<&str>`. Accept `&[String]` and pass
  every stopped node through; `power_on`/`select_targets` likewise.
- `doctor` wrote a Dockerfile to the literal path `recipes/temp/...`
  (hardcoded `"temp"` instead of the recipe name) then removed it —
  pointless, since the content is sent to the node via a quoted heredoc.
  Drop the local write.

Shell-quoting (call sites of the `shq` relocated in the previous commit):
- Thread `shq` through every docker/tmux command builder so
  `recipe_name`/`cluster.name`/image tags/paths can't break or inject
  through the remote `sh -c`: stop/label-lookup/cleanup/path-ref in
  docker, logs/deploy/build/sync, `upload_dir`, and tmux session/pane
  targeting. Switch tmux `grep -c` to `grep -Fc` so a `.` in a recipe
  name can't match the wrong session. Add regression tests for quotes
  and spaces in recipe names.

Standards & refactor (no user-facing behavior change):
- Remove all three `#[allow(...)]` attributes, which AGENTS.md forbids:
  bundle `prepare_cluster`'s end-effectors into a `PrepareRequest`
  struct, drop the dead `session_prefix` field (`TmuxManager` is now a
  unit struct), and delete the stale `is_degraded` allow (it is used).
- Make boot timing injectable (`BootTiming`) so the power-on test
  doesn't sleep a real 60s — the suite drops from 60s to 0.01s.
- Replace `.expect()` panics with `Result` (docker/build, host_keys).
- Log swallowed `JoinError`s in `cluster_state::probe` and `health.rs`
  instead of dropping them silently.
- Collapse `power_switch` dead `if dry_run` branches and route the
  disable phase through `toggle_entities` for consistent error handling.
- Replace the three `eprintln!` calls: the task-join diagnostic becomes
  `warn!`; the node-offline and host-key-error lines become `println!`
  (they are result data that piping previously lost).
- Route ~25 hardcoded English/Cyrillic strings through `fluent_i18n::t!`
  (worst: a raw Russian `=== Кластер ===` line in an i18n-aware binary);
  add the missing FTL keys to `locales/en`.
- Align AGENTS.md line-limit docs to the actual <=500 gate; add the
  CODE_REVIEW.md report.
Alexander Korolev 2 mesiacov pred
rodič
commit
1ba1d99e05

+ 1 - 1
AGENTS.md

@@ -148,6 +148,6 @@ commands = ["cmd", { command = "cmd", on_failure = "warn" }]
 
 ## Constraints
 
-- Files ≤800 lines unless impractical; extract submodules when a file exceeds 700 lines (`make check-lines`)
+- Files ≤500 lines (enforced by `make check-lines`); extract submodules when a file approaches the limit
 - `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`
 - **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.

+ 281 - 0
CODE_REVIEW.md

@@ -0,0 +1,281 @@
+# Code Review — sm121-launcher
+
+**Scope:** Full codebase review (33 files, ~8.4k LOC) using the code-reviewer
+methodology — pillars: Correctness, Maintainability, Readability, Efficiency,
+Security, Edge Cases/Error Handling, Testability.
+**Baseline:** `make check` passes (fmt, clippy, 100 tests, ≤500 lines). One test
+takes 60s real-time (see finding I-7).
+
+## Summary
+
+The codebase is well-structured: clean module separation, trait-based
+abstractions for testability (`NodeProber`/`PowerController`/`NodePinger`),
+good i18n groundwork, and thoughtful domain modeling (HF cache path refs,
+mod entrypoint wrapping). Tests are thorough (100 passing) and the line-limit
+discipline is mostly respected.
+
+The main weakness is **consistency of shell escaping**: a `shq()` helper
+exists and is correctly used in `docker/tail.rs` + `docker/mods.rs`, but is
+absent everywhere else, so `recipe_name` / `cluster.name` / image tags /
+paths are interpolated raw into SSH shell strings. There are also a few real
+bugs (a UTF-8 panic, a hardcoded `"temp"` path, an infinite build loop), and
+several `#[allow(...)]` attributes that violate AGENTS.md.
+
+---
+
+## Critical (bugs / security)
+
+### C-1. Shell injection via unquoted interpolations in SSH commands
+`shq()` (single-quote escaper) exists in `docker/mod.rs` but is **not used**
+where it matters most. `recipe_name`, `cluster.name`, image tags, user/host
+and paths are interpolated raw into command strings that are exec'd through
+the remote shell (`channel.exec` → `sh -c`). A recipe/cluster name containing
+`"`, `'`, `$`, `` ` ``, or `;` breaks the command or injects shell code.
+
+Affected sites:
+- `src/docker/mod.rs` — `stop_cluster` (recipe_name + cluster.name in
+  double-quoted `--filter`), `resolve_cluster_by_container_label`,
+  `start_cluster` `cleanup` (`docker rm -f sm121_{recipe_name}_{rank}`),
+  `resolve_path_ref_on_node` (`hub_dir`/`sub` in single-quoted context).
+- `src/docker/logs.rs` — `logs_tail_cmd` / `logs_follow_cmd`.
+- `src/docker/deploy.rs` — `discover_recipe_deployments` filter.
+- `src/docker/mods.rs` — `upload_mods_to_cluster` remote path
+  (`/tmp/sm121-mods-{recipe_name}`), which then hits `upload_dir` (see C-2).
+- `src/docker/build.rs` — `build_image_on_node` (recipe_name, image_tag,
+  remote_dir, logfile all unquoted in a double-quoted tmux command).
+- `src/docker/sync.rs` — daisy-chain `docker save | ssh ... docker load`
+  (image, ssh_user, ssh_ip, key path unquoted).
+- `src/ssh.rs` — `upload_dir` (`remote_dir` unquoted in `mkdir -p {dir}`).
+- `src/tmux/session.rs` — `session_name` interpolated into `-t` args / greps.
+
+Threat model: these values come from trusted `config.toml` / recipe TOML, so
+exploitation requires a malicious recipe/config — but the `shq()` helper
+exists precisely for this, the inconsistency is the smell, and a recipe name
+with a space already silently breaks commands.
+
+### C-2. `before_serve.rs::truncate` panics on multibyte UTF-8
+```rust
+fn truncate(s: &str, max: usize) -> String {
+    if s.len() <= max { s.to_string() }
+    else { format!("{}…", &s[..max]) }   // <-- byte slice, panics off char boundary
+}
+```
+`s.len()` is bytes; `&s[..max]` panics if `max` lands inside a multibyte
+char. Any non-ASCII pre-serve command (common: Cyrillic/CJK paths, `–` dashes)
+with byte length > 120 crashes the launcher. Fix with `s.chars().take(max)`.
+
+### C-3. SSH host-key verification silently skipped when `host_key` is `None`
+`ssh.rs::VerifyingHandler::check_server_key` logs "(insecure)" but returns
+`Ok(true)` — full TOFU-skip, a MITM vector. `init-host-keys` exists to
+populate these, so the safe default is to **refuse** (or pin on first use)
+rather than proceed. At minimum the skip should be opt-in, not the default.
+
+### C-4. `docker/build.rs::build_image_on_node` has no build timeout
+The poll loop `sleep(30); tmux has-session?` runs until the session dies. A
+hung build (stuck tmux, network stall) loops **forever**, hanging the
+launcher. Add a configurable max build duration (e.g. default 60–90 min) with
+a clean timeout error.
+
+---
+
+## Improvements (correctness / maintainability / standards)
+
+### I-1. AGENTS.md violation: `#[allow(...)]` attributes (forbidden by project rules)
+Three occurrences — the project explicitly forbids `#[allow(...)]` and says to
+refactor instead:
+- `src/orchestrate/mod.rs:22` — `#[allow(clippy::too_many_arguments)]` on
+  `prepare_cluster` (8 args). Fix: group `(auto_power_on, prober, power,
+  pinger)` into a `PrepareOpts` / `Ends` struct.
+- `src/tmux/mod.rs:14` — `#[allow(dead_code)]` on `session_prefix`. The field
+  is genuinely unused (`session_name` builds the name directly). Remove the
+  field (and the struct field) entirely.
+- `src/tmux/mod.rs:63` — `#[allow(dead_code)]` on `is_degraded`. It **is**
+  used (`health.rs::classify_status` calls `health.is_degraded()`). The
+  attribute is stale — just delete it.
+
+### I-2. `doctor.rs::run_cluster_repair` hardcodes literal `"temp"`
+```rust
+let dockerfile_path = format!("recipes/{}/Dockerfile.repaired", "temp");
+```
+Should be `recipe_name`. Worse, the local write is pointless: the content is
+already sent to the node via a quoted heredoc, and the file is removed at the
+end of the function. Drop the local write entirely (or use `std::env::temp_dir`
+if a local copy is ever needed).
+
+### I-3. `doctor.rs` / `docker/build.rs` use `.expect()` (panic) for control flow
+- `docker/build.rs:198` `.expect("build-node already validated")` → use
+  `ok_or_else(|| bail!(...))` for a clean error.
+- `host_keys.rs:80` `.expect("valid TOML config")` → return `Result` (the
+  config is already parsed so it's safe today, but a panic is the wrong tool).
+
+### I-4. 60-second test smell — boot timing not injectable
+`orchestrate/mod.rs::wait_for_all_nodes` hardcodes `boot_wait_secs=60`,
+`poll_interval_secs=10`, `max_poll_secs=120`. The test
+`cluster_off_auto_power_on_powers_and_selects` exercises the power-on path,
+so it sleeps a **real 60s** (`tokio::time::sleep`) — the whole `cargo test`
+run takes 60s because of this one test. Inject the timing (a trait method on
+`NodePinger`, or a `BootTiming` struct param) so tests can use 0s.
+
+### I-5. `main.rs::handle_stop` `--power-off` only powers off the first node
+`stop` accepts `--node a --node b` (repeatable, `Vec<String>`), but:
+```rust
+let node_str = if node.is_empty() { None } else { Some(node[0].as_str()) };
+crate::power::power_off(config, &cluster, node_str, dry_run).await?;
+```
+Only `node[0]` is passed to `power_off`. `power_off` itself takes a single
+`Option<&str>`. So `stop --node a --node b --off` stops a+b but powers off
+only a. Either pass all nodes (change `power_off` to accept `&[String]`) or
+document the single-node limitation and validate.
+
+### I-6. `power.rs::power_switch` dead `if dry_run` branches + inconsistent errors
+The function returns early on `dry_run` at the top, yet the later HA block
+re-checks `if dry_run { ... } else { let _ = ha.turn_off(...) }` — the
+`dry_run` arm is unreachable. Additionally the disable phase uses `let _ =`
+(swallows errors) while the enable phase uses `toggle_entities` (which logs
+them). Collapse the manual disable loop into `toggle_entities("turn_off", …)`
+for consistency and remove the dead branch.
+
+### I-7. Inconsistent i18n — ~20 raw strings bypass `fluent_i18n::t!`
+89 FTL keys exist, but many user-facing strings are hardcoded (defeating the
+6-language localization):
+- `main.rs:325` — hardcoded **Cyrillic** `=== Кластер: {} ===` (worst: a
+  Russian string in an i18n-aware binary).
+- `power.rs`, `docker/sync.rs`, `docker/health.rs`, `tmux/ops.rs`,
+  `tmux/session.rs`, `tmux/serve.rs`, `doctor.rs`, `orchestrate/mod.rs` —
+  raw English with emoji (`🛑`/`🔌`/`💡`/`⏳`/`🩺`/`✓`/`✗`).
+
+Either route all of these through FTL or drop the i18n system. The mixed
+state is the worst option.
+
+### I-8. `main.rs` mid-file `use std::collections::HashSet;`
+Placed after `resolve_cluster_for_op`'s doc comment instead of at module top.
+Unidiomatic; move to the top `use` block.
+
+### I-9. `config/types.rs::resolved_hf_cache_dir` is convoluted
+Asymmetric: non-root early-returns the full path; root sets `home` then falls
+through to format. Simplify to one expression:
+```rust
+let home = if self.ssh_user == "root" { "/root".to_string() }
+           else { format!("/home/{}", self.ssh_user) };
+PathBuf::from(format!("{home}/.cache/huggingface"))
+```
+
+### I-10. `tmux/session.rs::check_session` uses regex `grep -c`, not `grep -Fc`
+```rust
+format!("tmux list-sessions 2>/dev/null | grep -c '{}'", session_name)
+```
+`session_name` = `{recipe.name}-{engine}-r{rank}`; recipe names like
+`GLM-5.2` contain `.` (regex any-char) → can match the wrong session. Use
+`grep -Fc` (fixed-string) or `grep -F -c`.
+
+### I-11. `cluster_state.rs::probe` / `health.rs` silently drop `JoinError`
+```rust
+_ => {}        // cluster_state
+Err(_) => {}  // health
+```
+A panicked probe task is swallowed and the node treated as absent (→ "not
+free"). At least `warn!` the join error so a node disappearing from
+selection is diagnosable.
+
+### I-12. `fetch_host_key` lacks the connect timeout that `ssh_session` has
+`handle_health --fingerprint` calls `fetch_host_key`, which uses a bare
+`russh::client::connect` (no `tokio::time::timeout`), so it hangs on
+unreachable nodes while `ssh_session` correctly times out at 5s. Wrap it.
+
+### I-13. `docker/build.rs` hardcoded `"glm52-build-"` session prefix
+`session_name = format!("glm52-build-{}", recipe_name)` — a leftover from a
+specific model. Should be `sm121-build-{recipe_name}` (generic, namespaced).
+
+### I-14. `PowerController` trait only models power-on
+`turn_off` is done directly via `HaClient` in `power.rs`, outside the trait
+abstraction that `orchestrate` uses for power-on. Either add `turn_off` to the
+trait (and route `power_off` through it) or document why it's asymmetric.
+
+---
+
+## Nitpicks
+
+- **N-1.** Redundant `default_value = "false"` on 7 bool flags in `cli.rs` —
+  clap defaults `bool` to `false`; only `default_value_t = true` flags need a
+  default. Remove the `"false"` defaults.
+- **N-2.** `power_on_if_off(_state: &ClusterState, …)` — unused `_state` param.
+- **N-3.** AGENTS.md says files ≤800 (extract at 700) but Makefile/DEVELOPMENT
+  enforce ≤500. Align the docs: `docker/command.rs` (495) and `power.rs` (482)
+  are at the edge of the stricter limit.
+- **N-4.** `docker/mod.rs::start_cluster` spawns all nodes and `join_all`s — a
+  failure on node 0 doesn't cancel nodes 1–3 (they still launch, leaving a
+  partial cluster). Intentional (reports all failures) but worth a comment.
+- **N-5.** `docker/sync.rs` daisy-chain `ssh -o StrictHostKeyChecking=no` for
+  the inner hop bypasses host-key verification (documented trade-off). Acceptable
+  for trusted clusters; flagged for awareness.
+
+---
+
+## Conclusion
+
+**Request Changes.** The code is high quality overall, but before the next
+release I'd want at least C-2 (panic), C-4 (infinite loop), I-1 (the three
+`#[allow]` violations — a project-standard hard rule), and the C-1 escaping
+sweep addressed. C-3 (host-key skip) is a policy decision — confirm with the
+maintainer whether "skip when unconfigured" is intended or should be TOFU/refuse.
+
+---
+
+## Resolution Status
+
+All findings below have been **implemented** (gates green: fmt + clippy +
+108 tests in 0.01s + ≤500 lines). Test count went 100 → 108; the suite
+dropped from **60s to 0.01s** (the 60s sleep was made injectable).
+
+| # | Finding | Status |
+|---|---------|--------|
+| C-1 | Shell-escaping sweep | ✅ `shq` moved to `ssh.rs`, threaded through all SSH command builders (stop/resolve/cleanup/path-ref/logs/deploy/build/sync/upload_dir/tmux); +2 regression tests |
+| C-2 | `truncate` multibyte panic | ✅ char-based slicing; +4 tests |
+| C-3 | Host-key skip → refuse by default | ✅ `ssh_session_with_timeout` bails with an `init-host-keys` hint; `fetch_host_key` exempt |
+| C-4 | Build loop no timeout | ✅ `max_build_secs` param (default 90 min) → clean bail |
+| I-1 | Three `#[allow]` attributes | ✅ `PrepareRequest` struct; `TmuxManager` → unit struct; stale `is_degraded` allow removed |
+| I-2 | `doctor` hardcoded `"temp"` | ✅ pointless local write removed |
+| I-3 | `.expect()` panics | ✅ `build.rs` + `host_keys.rs` → `Result` |
+| I-4 | 60s test sleep | ✅ `BootTiming` injectable; suite 60s → 0.01s |
+| I-5 | `stop --power-off` multi-node | ✅ `power_off`/`power_on`/`select_targets` accept `&[String]`; +2 tests |
+| I-6 | `power_switch` dead branches | ✅ routed through `toggle_entities` |
+| I-7 | i18n bypass | ✅ ~25 raw strings routed through `t!` (Cyrillic removed); new FTL keys in `en` (others fall back) |
+| I-8 | Mid-file `use` | ✅ moved to module top |
+| I-9 | `resolved_hf_cache_dir` | ✅ simplified |
+| I-10 | `grep -c` regex | ✅ `grep -Fc` fixed-string |
+| I-11 | Silent `JoinError` drop | ✅ `warn!` in `cluster_state::probe` + `health.rs` |
+| I-12 | `fetch_host_key` no timeout | ✅ 5s connect timeout |
+| I-13 | `glm52-build-` prefix | ✅ → `sm121-build-` |
+| I-14 | `PowerController` asymmetry | ✅ documented on the trait |
+| N-1 | Redundant `default_value="false"` | ✅ removed (7 flags) |
+| N-2 | Unused `_state` param | ✅ removed |
+| N-3 | AGENTS.md line-limit doc | ✅ aligned to ≤500 |
+| N-4 | Partial-cluster comment | ✅ added |
+| N-5 | Daisy-chain SSH (sync) | acknowledged — documented trade-off in `sync.rs` |
+
+---
+
+## Fix Plan (prioritized)
+
+| # | Finding | Effort | Risk | Suggested approach |
+|---|---------|--------|------|-------------------|
+| 1 | **C-2** truncate panic | XS | low | `s.chars().take(max)` + test with multibyte |
+| 2 | **C-4** build timeout | S | low | add `max_build_secs` param (default 5400) → `bail!` on exceed |
+| 3 | **I-1** remove 3× `#[allow]` | S | low | `PrepareOpts` struct for `prepare_cluster`; delete `session_prefix` field + stale `is_degraded` allow |
+| 4 | **I-2** doctor `"temp"` | XS | low | use `recipe_name`; drop pointless local write |
+| 5 | **I-3** `.expect()` → `Result` | XS | low | `ok_or_else` in build.rs; return `Result` in host_keys |
+| 6 | **C-1** shell-escaping sweep | M | med | thread `shq()` through all SSH command builders; add tests with `"`/`'`/space in recipe_name |
+| 7 | **I-4** injectable boot timing | S | low | `BootTiming` struct or trait method; test uses 0s |
+| 8 | **I-5** stop `--power-off` multi-node | S | low | `power_off` accepts `&[String]`; pass all nodes |
+| 9 | **I-6** power_switch dead branch | XS | low | route disable through `toggle_entities`; drop dead `if dry_run` |
+| 10 | **I-7** i18n consistency | M | low | add FTL keys for the ~20 raw strings; route through `t!` |
+| 11 | **I-10** `grep -Fc` | XS | low | fixed-string grep |
+| 12 | **I-12** `fetch_host_key` timeout | XS | low | wrap `connect` in `tokio::time::timeout(5s)` |
+| 13 | **I-13** `glm52-` → `sm121-` | XS | low | rename session prefix |
+| 14 | **C-3** host-key policy | S | med | **confirm intent first**; default to refuse or TOFU |
+| 15 | **N-1..N-4** nitpicks | XS | low | batch cleanup |
+
+Recommended batching: do **1–5** (the trivial bug/standards fixes) in one PR,
+**6** (escaping sweep) as its own PR with tests, **7–9** in a refactor PR,
+**10** (i18n) separately (string-review heavy), and **14** only after the
+maintainer confirms the host-key policy.

+ 34 - 0
locales/en/main.ftl

@@ -91,3 +91,37 @@ mods-uploading = 📤 Uploading mods to { $node }...
 mods-uploaded = ✅ Mods uploaded to { $node }
 mods-entrypoint-required = Recipe '{ $name }' has mods but no entrypoint; entrypoint is required when mods are present
 power-off-container-stop-failed = ⚠️ Failed to stop containers on node { $name }: { $error}
+
+# --- Strings that previously bypassed the i18n system (hardcoded) ---
+boot-wait = ⏳ Waiting { $secs }s for nodes to boot...
+doctor-found-repaired = 🩺 Found repaired image: { $tag }
+tmux-waiting-head = ⏳ Waiting for head node to be ready...
+tmux-head-ready = ✅ Head node ready. Starting worker nodes...
+tmux-head-api-ready = Head node API ready (attempt { $attempt })
+tmux-session-started = ✅ Session started on { $name }
+tmux-stopped = ✅ Stopped '{ $sess }' on { $name }
+tmux-all-stopped = ✅ All tmux sessions stopped.
+tmux-all-ok = ✅ ALL OK
+tmux-issues = ⚠️  { $issues }
+tmux-cluster-healthy = ✅ Cluster is healthy!
+tmux-node-section = --- { $name } ({ $sess }) ---
+tmux-node-no-output = --- { $name } — no output ---
+tmux-session-created = ✅ tmux session created on { $name }
+tmux-no-pty = (russh does not support interactive PTY. Use the following command directly:)
+sync-digest-match = '{ $name }' OK (digest matches)
+sync-syncing = { $name } ({ $ip }): syncing...
+sync-synced = { $name } OK
+health-node-offline =   ✗ { $name } ({ $ip }) — offline
+health-node-idle =   ○ { $name } ({ $ip }){ $net }
+health-node-online =   ● { $name } ({ $ip }){ $net }
+health-container-count = , { $count } container(s)
+power-off-head = 🛑 power off: { $nodes } node(s) — sudo shutdown now
+power-switch-head = 🔌 switch: { $from } → { $to } ({ $nodes } node(s) + { $entities } entity(ies) to disable)
+power-switch-powering-on = 💡 switch: powering on { $nodes } node(s) + { $entities } entity(ies) to { $to }
+health-task-join = task join: { $error }
+health-no-recipes = No sm121 containers running.
+health-running-recipes = Running recipes:
+health-recipe-line =   { $recipe } — { $nodes } node(s): { $list }
+tmux-ray-backend = 🐍 Ray backend detected — starting head node first...
+tmux-mp-backend = 🔗 MP backend detected — starting all nodes in parallel...
+tmux-following-logs = 📜 Following logs from '{ $sess }' on { $name }...

+ 36 - 2
src/before_serve.rs

@@ -142,10 +142,14 @@ async fn run_for_node(
 }
 
 fn truncate(s: &str, max: usize) -> String {
-    if s.len() <= max {
+    // Iterate by chars (not bytes) so a multibyte boundary inside `max`
+    // doesn't panic on `&s[..max]` — pre-serve commands often contain
+    // non-ASCII (Cyrillic/CJK paths, en-dashes).
+    let head: String = s.chars().take(max).collect();
+    if s.chars().count() <= max {
         s.to_string()
     } else {
-        format!("{}…", &s[..max])
+        format!("{head}…")
     }
 }
 
@@ -198,6 +202,36 @@ mod tests {
         }
     }
 
+    #[test]
+    fn truncate_keeps_short_string_unchanged() {
+        assert_eq!(truncate("short", 120), "short");
+        assert_eq!(truncate("", 120), "");
+    }
+
+    #[test]
+    fn truncate_shortens_to_char_boundary() {
+        // 10 chars, cap at 4 → first 4 chars + ellipsis
+        let out = truncate("abcdefghij", 4);
+        assert_eq!(out, "abcd…");
+    }
+
+    #[test]
+    fn truncate_does_not_panic_on_multibyte() {
+        // Each 'é' is 2 bytes; a byte slice `&s[..3]` would land mid-char and
+        // panic. The char-based impl must not. (Regression for C-2.)
+        let s = "éééééé"; // 6 chars, 12 bytes
+        assert_eq!(truncate(s, 3), "ééé…");
+        // cap exactly at a multibyte boundary must also be safe
+        assert_eq!(truncate(s, 4), "éééé…");
+    }
+
+    #[test]
+    fn truncate_cjk_chars() {
+        // 3-byte CJK: byte-slice at any non-multiple-of-3 would panic.
+        let s = "你好世界你好";
+        assert_eq!(truncate(s, 2), "你好…");
+    }
+
     #[test]
     fn combined_orders_cluster_then_recipe() {
         let mut bs = HashMap::new();

+ 7 - 7
src/cli.rs

@@ -28,11 +28,11 @@ pub enum Commands {
         node: Option<String>,
 
         /// Keep containers after stop (default: auto-remove on stop)
-        #[arg(long, default_value = "false")]
+        #[arg(long)]
         keep: bool,
 
         /// Skip pre-serve hooks (cluster-level + recipe-level).
-        #[arg(long, default_value = "false")]
+        #[arg(long)]
         skip_before_serve: bool,
 
         /// Follow logs after serving (use --no-follow to skip).
@@ -40,7 +40,7 @@ pub enum Commands {
         follow: bool,
 
         /// Power on cluster if switch is off (requires HA config).
-        #[arg(short = 'O', long = "on", default_value = "false")]
+        #[arg(short = 'O', long = "on")]
         auto_power_on: bool,
 
         /// Sync Docker image across nodes before starting.
@@ -56,7 +56,7 @@ pub enum Commands {
         build_node: Option<String>,
 
         /// Force rebuild even if checksum matches.
-        #[arg(long, default_value = "false")]
+        #[arg(long)]
         force_build: bool,
     },
 
@@ -89,7 +89,7 @@ pub enum Commands {
         cluster: String,
 
         /// Power off cluster after stopping containers (requires HA config).
-        #[arg(long = "off", default_value = "false")]
+        #[arg(long = "off")]
         power_off: bool,
     },
 
@@ -164,7 +164,7 @@ pub enum Commands {
         cluster: String,
 
         /// Force rebuild even if checksum matches.
-        #[arg(long, default_value = "false")]
+        #[arg(long)]
         force_build: bool,
     },
 
@@ -215,7 +215,7 @@ pub enum TmuxCommands {
         #[arg(long, default_value = "auto")]
         cluster: String,
         /// Skip pre-serve hooks (cluster-level + recipe-level).
-        #[arg(long, default_value = "false")]
+        #[arg(long)]
         skip_before_serve: bool,
     },
     /// Stop tmux sessions

+ 6 - 2
src/cluster_state.rs

@@ -7,7 +7,7 @@
 
 use anyhow::{Context, Result};
 use std::collections::{HashMap, HashSet};
-use tracing::info;
+use tracing::{info, warn};
 
 use crate::config::{Cluster, Config, NcclConfig};
 use crate::ssh::run_ssh;
@@ -75,7 +75,11 @@ impl ClusterState {
                         },
                     );
                 }
-                _ => {}
+                Err(e) => {
+                    // A probe task panicked (JoinError) — log it so a node
+                    // silently dropping out of the state map is diagnosable.
+                    warn!("Cluster probe task panicked: {}", e);
+                }
             }
         }
         Self { nodes }

+ 3 - 3
src/config/types.rs

@@ -159,11 +159,11 @@ impl Config {
             return std::path::PathBuf::from(dir);
         }
         let home = if self.ssh_user == "root" {
-            "/root"
+            "/root".to_string()
         } else {
-            return std::path::PathBuf::from(format!("/home/{}/.cache/huggingface", self.ssh_user));
+            format!("/home/{}", self.ssh_user)
         };
-        std::path::PathBuf::from(format!("{}/.cache/huggingface", home))
+        std::path::PathBuf::from(format!("{home}/.cache/huggingface"))
     }
 }
 

+ 48 - 15
src/docker/build.rs

@@ -7,6 +7,13 @@ use std::io::Read;
 use std::path::Path;
 use tracing::info;
 
+use super::shq;
+
+/// Default upper bound on a single remote image build before the launcher
+/// gives up (90 min). A hung build (stuck tmux, network stall) would
+/// otherwise loop the 30s poll forever, hanging the launcher.
+pub const DEFAULT_MAX_BUILD_SECS: u64 = 90 * 60;
+
 /// Compute a SHA-256 checksum over all files in `dir` (recursive).
 /// Sorted by relative path to ensure reproducibility.
 pub fn compute_dir_checksum(dir: &Path) -> Result<String> {
@@ -50,8 +57,9 @@ pub async fn build_image_on_node(
     recipe_name: &str,
     image_tag: &str,
     checksum: &str,
+    max_build_secs: u64,
 ) -> Result<()> {
-    let session_name = format!("glm52-build-{}", recipe_name);
+    let session_name = format!("sm121-build-{}", recipe_name);
     let logfile = format!("/tmp/{}-build.log", recipe_name);
     let remote_dir = format!("/tmp/{}-build", recipe_name);
 
@@ -67,16 +75,19 @@ pub async fn build_image_on_node(
         config,
         node,
         format!(
-            "tmux kill-session -t '{}' 2>/dev/null || true",
-            session_name
+            "tmux kill-session -t {} 2>/dev/null || true",
+            shq(&session_name)
         ),
         false,
     )
     .await?;
 
     let tmux_cmd = format!(
-        "tmux new-session -d -s '{}' -c {} \"TAG={} bash build.sh 2>&1 | tee {}\"",
-        session_name, remote_dir, image_tag, logfile
+        "tmux new-session -d -s {} -c {} \"TAG={} bash build.sh 2>&1 | tee {}\"",
+        shq(&session_name),
+        shq(&remote_dir),
+        shq(image_tag),
+        shq(&logfile)
     );
     run_ssh(config, node, tmux_cmd, false).await?;
 
@@ -85,6 +96,18 @@ pub async fn build_image_on_node(
         tokio::time::sleep(std::time::Duration::from_secs(30)).await;
         let elapsed = start.elapsed();
 
+        if elapsed.as_secs() >= max_build_secs {
+            anyhow::bail!(
+                "Build on '{}' of '{}' timed out after {} — no completion marker. \
+                 Inspect tmux session '{}' / log '{}' on the node.",
+                node.name,
+                image_tag,
+                format_elapsed(elapsed),
+                session_name,
+                logfile
+            );
+        }
+
         // `tmux has-session` exits 0 and prints nothing when the session is
         // alive; the `|| echo DEAD` makes it print DEAD when the session is
         // gone (still exit 0, so run_ssh stays Ok). An empty `alive` therefore
@@ -94,7 +117,10 @@ pub async fn build_image_on_node(
         let alive = run_ssh(
             config,
             node,
-            format!("tmux has-session -t '{}' 2>&1 || echo DEAD", session_name),
+            format!(
+                "tmux has-session -t {} 2>&1 || echo DEAD",
+                shq(&session_name)
+            ),
             false,
         )
         .await?;
@@ -103,7 +129,10 @@ pub async fn build_image_on_node(
         if let Ok(logs) = run_ssh(
             config,
             node,
-            format!("tail -10 {} 2>/dev/null || echo '(no log yet)'", logfile),
+            format!(
+                "tail -10 {} 2>/dev/null || echo '(no log yet)'",
+                shq(&logfile)
+            ),
             false,
         )
         .await
@@ -121,8 +150,9 @@ pub async fn build_image_on_node(
                 config,
                 node,
                 format!(
-                    "grep -F -- '{}' {} >/dev/null 2>&1 && echo OK || echo NO",
-                    marker, logfile
+                    "grep -F -- {} {} >/dev/null 2>&1 && echo OK || echo NO",
+                    shq(&marker),
+                    shq(&logfile)
                 ),
                 false,
             )
@@ -137,7 +167,7 @@ pub async fn build_image_on_node(
                     format_elapsed(elapsed)
                 );
             } else {
-                let exit_log = run_ssh(config, node, format!("tail -20 {}", logfile), false)
+                let exit_log = run_ssh(config, node, format!("tail -20 {}", shq(&logfile)), false)
                     .await
                     .unwrap_or_default();
                 anyhow::bail!(
@@ -154,8 +184,10 @@ pub async fn build_image_on_node(
 
     // Stamp checksum label on the built image.
     let stamp_cmd = format!(
-        "echo 'FROM {}' | docker build --label sm121.checksum={} -t {} -",
-        image_tag, checksum, image_tag
+        "printf 'FROM %s\n' {} | docker build --label sm121.checksum={} -t {} -",
+        shq(image_tag),
+        checksum,
+        shq(image_tag)
     );
     let _stamp = run_ssh(config, node, stamp_cmd, false).await?;
     info!("Stamped checksum label '{}' on '{}'", checksum, image_tag);
@@ -195,7 +227,7 @@ pub async fn handle_build(
         .nodes
         .iter()
         .find(|n| n.name == build_node_name)
-        .expect("build-node already validated");
+        .ok_or_else(|| anyhow::anyhow!("build-node '{}' not in cluster", build_node_name))?;
 
     let checksum = compute_dir_checksum(local_dir.as_path())?;
 
@@ -207,7 +239,7 @@ pub async fn handle_build(
             build_node,
             format!(
                 "docker image ls --format '{{{{.ID}}}}' {} | head -1",
-                recipe.docker_image
+                shq(&recipe.docker_image)
             ),
             false,
         )
@@ -220,7 +252,7 @@ pub async fn handle_build(
                 build_node,
                 format!(
                     "docker inspect --format '{{{{index .Config.Labels \"sm121.checksum\"}}}}' {} 2>/dev/null || echo MISSING",
-                    recipe.docker_image
+                    shq(&recipe.docker_image)
                 ),
                 false,
             )
@@ -250,6 +282,7 @@ pub async fn handle_build(
             recipe_name,
             &recipe.docker_image,
             &checksum,
+            DEFAULT_MAX_BUILD_SECS,
         )
         .await?;
     }

+ 6 - 1
src/docker/deploy.rs

@@ -6,6 +6,8 @@ use std::collections::{HashMap, HashSet};
 use crate::config::Config;
 use crate::ssh::run_ssh;
 
+use super::shq;
+
 /// A running deployment variant — a set of nodes on a cluster running a recipe.
 pub struct DeploymentVariant {
     pub cluster: String,
@@ -29,7 +31,10 @@ pub async fn discover_recipe_deployments(
             let filter = if recipe_name.is_empty() {
                 "--filter label=sm121.recipe".to_string()
             } else {
-                format!("--filter label=sm121.recipe={}", recipe_name)
+                format!(
+                    "--filter {}",
+                    shq(&format!("label=sm121.recipe={}", recipe_name))
+                )
             };
             let cmd = format!(
                 "docker ps {} \

+ 25 - 11
src/docker/health.rs

@@ -6,6 +6,7 @@ use crate::home_assistant::Client as HaClient;
 use crate::ssh::run_ssh;
 use anyhow::Result;
 use std::collections::HashMap;
+use tracing::warn;
 
 /// Per-node result from the parallel SSH sweep.
 struct NodeHealth {
@@ -135,7 +136,10 @@ pub async fn health_check(config: &Config, cluster: &Cluster, dry_run: bool) ->
                     net_details: vec![],
                 });
             }
-            Err(join_err) => eprintln!("task join: {}", join_err),
+            Err(join_err) => warn!(
+                "{}",
+                fluent_i18n::t!("health-task-join", { "error" => &join_err.to_string() })
+            ),
         }
     }
 
@@ -162,14 +166,20 @@ pub async fn health_check(config: &Config, cluster: &Cluster, dry_run: bool) ->
         print!(" ({} online, {} offline)", online, offline);
     }
     if container_count > 0 {
-        println!(", {} container(s)", container_count);
+        println!(
+            "{}",
+            fluent_i18n::t!("health-container-count", { "count" => container_count })
+        );
     } else {
         println!(", no containers");
     }
 
     for nh in &node_health {
         if !nh.online {
-            eprintln!("  ✗ {} ({}) — offline", nh.name, nh.ib_ip);
+            println!(
+                "{}",
+                fluent_i18n::t!("health-node-offline", { "name" => &nh.name, "ip" => &nh.ib_ip })
+            );
             continue;
         }
         let net_status = if !nh.net_up {
@@ -178,9 +188,15 @@ pub async fn health_check(config: &Config, cluster: &Cluster, dry_run: bool) ->
             String::new()
         };
         if nh.containers.is_empty() {
-            println!("  ○ {} ({}){}", nh.name, nh.ib_ip, net_status);
+            println!(
+                "{}",
+                fluent_i18n::t!("health-node-idle", { "name" => &nh.name, "ip" => &nh.ib_ip, "net" => &net_status })
+            );
         } else {
-            println!("  ● {} ({}){}", nh.name, nh.ib_ip, net_status);
+            println!(
+                "{}",
+                fluent_i18n::t!("health-node-online", { "name" => &nh.name, "ip" => &nh.ib_ip, "net" => &net_status })
+            );
             for line in &nh.containers {
                 println!("    {line}");
             }
@@ -235,20 +251,18 @@ pub async fn health_summary(config: &Config) -> Result<()> {
     }
 
     if recipe_nodes.is_empty() {
-        println!("\nNo sm121 containers running.");
+        println!("\n{}", fluent_i18n::t!("health-no-recipes"));
         return Ok(());
     }
 
-    println!("\nRunning recipes:");
+    println!("\n{}", fluent_i18n::t!("health-running-recipes"));
     for (recipe, nodes) in &recipe_nodes {
         let mut sorted: Vec<_> = nodes.iter().collect();
         sorted.sort();
         let node_list: Vec<&str> = sorted.iter().map(|s| s.as_str()).collect();
         println!(
-            "  {} — {} node(s): {}",
-            recipe,
-            node_list.len(),
-            node_list.join(", ")
+            "{}",
+            fluent_i18n::t!("health-recipe-line", { "recipe" => recipe, "nodes" => node_list.len(), "list" => &node_list.join(", ") })
         );
     }
     Ok(())

+ 39 - 4
src/docker/logs.rs

@@ -6,17 +6,21 @@ use tracing::info;
 use crate::config::{Cluster, Config, Node};
 use crate::ssh::{run_ssh_follow, run_ssh_with_retry};
 
+use super::shq;
+
 pub fn logs_tail_cmd(recipe_name: &str, rank: u32, tail: usize) -> String {
     format!(
-        "docker ps -q --filter \"label=sm121.recipe={}\" --filter \"label=sm121.rank={}\" | xargs -r docker logs --timestamps --tail={} 2>&1",
-        recipe_name, rank, tail
+        "docker ps -q --filter {} --filter label=sm121.rank={} | xargs -r docker logs --timestamps --tail={} 2>&1",
+        shq(&format!("label=sm121.recipe={}", recipe_name)),
+        rank,
+        tail
     )
 }
 
 pub fn logs_follow_cmd(recipe_name: &str) -> String {
     format!(
-        "docker ps -q --filter \"label=sm121.recipe={}\" --filter \"label=sm121.rank=0\" | xargs -r docker logs -f --timestamps 2>&1",
-        recipe_name
+        "docker ps -q --filter {} --filter label=sm121.rank=0 | xargs -r docker logs -f --timestamps 2>&1",
+        shq(&format!("label=sm121.recipe={}", recipe_name))
     )
 }
 
@@ -85,3 +89,34 @@ pub async fn show_logs(
     }
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{logs_follow_cmd, logs_tail_cmd};
+
+    #[test]
+    fn tail_cmd_shell_quotes_recipe_name() {
+        // A recipe name with shell metacharacters (`"`, `;`, spaces) must be
+        // single-quoted so it can't break out of the docker --filter argument
+        // or inject a command separator. (Regression for C-1.)
+        let cmd = logs_tail_cmd("foo\";rm -rf /", 0, 100);
+        assert!(
+            cmd.contains("'label=sm121.recipe=foo\";rm -rf /'"),
+            "recipe name not shell-quoted: {cmd}"
+        );
+        // No bare, unquoted command separator reaches the shell.
+        assert!(!cmd.contains("--filter label=sm121.recipe=foo\";"));
+    }
+
+    #[test]
+    fn follow_cmd_shell_quotes_recipe_name_with_single_quote() {
+        // `$HOME` and spaces must stay literal (single-quoted) — not shell-
+        // expanded. The inner-single-quote *escaping* of `shq` itself is
+        // covered by `docker::tests::shq_escapes_inner_single_quotes`.
+        let cmd = logs_follow_cmd("a b$HOME");
+        assert!(
+            cmd.contains("'label=sm121.recipe=a b$HOME'"),
+            "recipe name not shell-quoted (would let $HOME expand): {cmd}"
+        );
+    }
+}

+ 35 - 27
src/docker/mod.rs

@@ -27,19 +27,11 @@ pub struct ServeOpts {
     pub build_node: Option<String>,
     pub force_build: bool,
 }
-pub(crate) fn shq(s: &str) -> String {
-    let mut out = String::with_capacity(s.len() + 2);
-    out.push('\'');
-    for c in s.chars() {
-        if c == '\'' {
-            out.push_str(r"'\''");
-        } else {
-            out.push(c);
-        }
-    }
-    out.push('\'');
-    out
-}
+/// Shell-quote a string for safe interpolation into a remote `sh -c`
+/// command (single-quote wrapping, embedded quotes escaped). Re-exported
+/// here so docker submodules can `use super::shq;`; the impl lives in
+/// [`crate::ssh`] (the lower-level shell layer).
+pub(crate) use crate::ssh::shq;
 pub async fn serve(
     config: &Config,
     recipe_name: &str,
@@ -56,7 +48,7 @@ pub async fn serve(
     );
     // Prepare cluster: probe, auto-power-on if requested, select.
     use crate::ends::{SshNodePinger, SshProber, ha_controller_from_config};
-    use crate::orchestrate::prepare_cluster;
+    use crate::orchestrate::{BootTiming, PrepareRequest, prepare_cluster};
 
     let power_ctrl = if opts.auto_power_on {
         ha_controller_from_config(config)
@@ -73,10 +65,13 @@ pub async fn serve(
             &recipe,
             cluster_name,
             node_name,
-            opts.auto_power_on,
-            &SshProber,
-            power_ctrl.as_ref(),
-            &SshNodePinger,
+            &PrepareRequest {
+                prober: &SshProber,
+                power: power_ctrl.as_ref(),
+                pinger: &SshNodePinger,
+                auto_power_on: opts.auto_power_on,
+                boot_timing: BootTiming::default(),
+            },
         )
         .await?
     };
@@ -202,8 +197,9 @@ pub async fn stop_cluster(
             config,
             node,
             format!(
-                "docker ps -q --filter \"label=sm121.recipe={}\" --filter \"label=sm121.cluster={}\" | xargs -r docker stop",
-                recipe_name, cluster.name
+                "docker ps -q --filter {} --filter {} | xargs -r docker stop",
+                shq(&format!("label=sm121.recipe={}", recipe_name)),
+                shq(&format!("label=sm121.cluster={}", cluster.name))
             ),
             dry_run,
         )
@@ -219,9 +215,9 @@ pub async fn resolve_cluster_by_container_label<'c>(
     // Scan all nodes for containers matching this recipe.
     for node in config.clusters.iter().flat_map(|c| &c.nodes) {
         let cmd = format!(
-            "docker ps --filter \"label=sm121.recipe={}\" \
+            "docker ps --filter {} \
              --format '{{{{.Label \"sm121.cluster\"}}}}' | head -1",
-            recipe_name
+            shq(&format!("label=sm121.recipe={}", recipe_name))
         );
         if let Ok(output) = run_ssh(config, node, cmd, false).await {
             let cluster_label = output.trim();
@@ -291,8 +287,8 @@ async fn start_cluster(
         let engine = recipe.engine.clone();
         let is_head = node.rank == 0;
         let cleanup = format!(
-            "docker rm -f sm121_{}_{} 2>/dev/null; true",
-            recipe_name, node.rank
+            "docker rm -f {} 2>/dev/null; true",
+            shq(&format!("sm121_{}_{}", recipe_name, node.rank))
         );
         let dm_ctx = command::DockerCommandCtx {
             cluster: cluster.clone(),
@@ -345,6 +341,11 @@ async fn start_cluster(
             (node.name, node.rank, engine, result)
         }));
     }
+    // NOTE: all node launches are awaited (join_all) before reporting — a
+    // failure on one node does not cancel the others, so a partial cluster
+    // may be left running. Intentional: reporting every node's result is
+    // more useful than aborting mid-launch and leaving the operator to
+    // guess which nodes actually started.
     for result in futures::future::join_all(handles).await {
         match result {
             Ok((name, rank, engine, Ok(_))) => info!(
@@ -387,11 +388,18 @@ async fn resolve_path_ref_on_node(
     let model_dir = crate::config::model_cache_dir_name(&spec.repo);
     let sub = spec.subfolder.as_deref().unwrap_or("");
     let hub_dir = format!("{}/hub/{}", hf_cache.display(), model_dir);
+    let refs_path = format!("{}/refs/main", hub_dir);
     // `cat refs/main` prints the current commit hash; trim trailing newline.
+    // All literal path components are shell-quoted (a repo/subfolder with a
+    // quote/space would otherwise break the command); `$ref` is our own
+    // captured var, so it expands safely inside the printf double-quotes.
     let cmd = format!(
-        "ref=\"$(cat '{}/refs/main' 2>/dev/null)\" && test -n \"$ref\" && \
-         dir=\"{}/snapshots/$ref/{}\" && test -d \"$dir\" && printf '%s' \"$dir\"",
-        hub_dir, hub_dir, sub
+        "ref=\"$(cat {refs_q} 2>/dev/null)\" && test -n \"$ref\" && \
+         dir=\"$(printf '%s/snapshots/%s/%s' {hub_q} \"$ref\" {sub_q})\" && \
+         test -d \"$dir\" && printf '%s' \"$dir\"",
+        refs_q = shq(&refs_path),
+        hub_q = shq(&hub_dir),
+        sub_q = shq(sub),
     );
     let resolved_path = run_ssh(config, node, cmd, false).await?;
     if resolved_path.trim().is_empty() {

+ 24 - 10
src/docker/sync.rs

@@ -3,9 +3,11 @@ use crate::ssh::{run_ssh, run_ssh_long_running};
 use anyhow::Result;
 use tracing::info;
 
+use super::shq;
+
 /// Get the Docker image digest (short ID) from a node.
 async fn get_image_digest(config: &Config, node: &Node, image: &str) -> Result<Option<String>> {
-    let cmd = format!("docker image ls --format '{{{{.ID}}}}' {image}");
+    let cmd = format!("docker image ls --format '{{{{.ID}}}}' {}", shq(image));
     let out = run_ssh(config, node, cmd, false).await?;
     let digest = out.trim().to_string();
     if digest.is_empty() {
@@ -17,7 +19,10 @@ async fn get_image_digest(config: &Config, node: &Node, image: &str) -> Result<O
 
 /// Get the Docker image creation timestamp.
 async fn get_image_created(config: &Config, node: &Node, image: &str) -> Option<String> {
-    let cmd = format!("docker image inspect --format '{{{{.Created}}}}' {image} 2>/dev/null");
+    let cmd = format!(
+        "docker image inspect --format '{{{{.Created}}}}' {} 2>/dev/null",
+        shq(image)
+    );
     run_ssh(config, node, cmd, false)
         .await
         .ok()
@@ -92,16 +97,22 @@ pub async fn sync_image_to_nodes(
             continue;
         }
         if target_digest.as_deref() == Some(source_digest) {
-            info!("  '{}' OK (digest matches)", target.name);
+            info!(
+                "{}",
+                fluent_i18n::t!("sync-digest-match", { "name" => &target.name })
+            );
             continue;
         }
-        info!("  {} ({}): syncing...", target.name, target.ssh_ip);
+        info!(
+            "{}",
+            fluent_i18n::t!("sync-syncing", { "name" => &target.name, "ip" => &target.ssh_ip })
+        );
         let pipe_cmd = format!(
-            "docker save '{}' | ssh -o StrictHostKeyChecking=no -i {} {}@{} docker load",
-            image,
-            config.ssh_key_path.display(),
-            config.ssh_user,
-            target.ssh_ip
+            "docker save {} | ssh -o StrictHostKeyChecking=no -i {} {}@{} docker load",
+            shq(image),
+            shq(&config.ssh_key_path.display().to_string()),
+            shq(&config.ssh_user),
+            shq(&target.ssh_ip)
         );
         run_ssh_long_running(config, src_node, pipe_cmd, false).await?;
         let verify = get_image_digest(config, target, image).await?;
@@ -113,7 +124,10 @@ pub async fn sync_image_to_nodes(
                 verify
             );
         }
-        info!("  {} OK", target.name);
+        info!(
+            "{}",
+            fluent_i18n::t!("sync-synced", { "name" => &target.name })
+        );
     }
     Ok(())
 }

+ 3 - 3
src/docker/tests.rs

@@ -401,8 +401,8 @@ fn custom_init_port() {
 fn logs_tail_cmd_format() {
     let cmd = super::logs::logs_tail_cmd("test-sleep", 0, 100);
     assert!(cmd.contains("docker ps -q"));
-    assert!(cmd.contains("--filter \"label=sm121.recipe=test-sleep\""));
-    assert!(cmd.contains("--filter \"label=sm121.rank=0\""));
+    assert!(cmd.contains("--filter 'label=sm121.recipe=test-sleep'"));
+    assert!(cmd.contains("--filter label=sm121.rank=0"));
     assert!(cmd.contains("xargs -r docker logs"));
     assert!(cmd.contains("--timestamps"));
     assert!(cmd.contains("--tail=100"));
@@ -424,7 +424,7 @@ fn logs_follow_cmd_format() {
     assert!(cmd.contains("xargs -r docker logs -f"));
     assert!(cmd.contains("--timestamps"));
     // Follow targets the head node only (rank=0).
-    assert!(cmd.contains("--filter \"label=sm121.rank=0\""));
+    assert!(cmd.contains("--filter label=sm121.rank=0"));
     // Follow should not include --tail (we want a live stream from now).
     assert!(!cmd.contains("--tail"));
     // Follow must redirect stderr so docker's own errors are captured

+ 7 - 7
src/doctor.rs

@@ -249,11 +249,9 @@ async fn run_cluster_repair(
     let dockerfile = generate_repair_dockerfile(&repaired_tag, recipe)
         .context("Failed to generate repair Dockerfile")?;
 
-    let dockerfile_path = format!("recipes/{}/Dockerfile.repaired", "temp");
-    tokio::fs::write(&dockerfile_path, &dockerfile)
-        .await
-        .context("Failed to write Dockerfile")?;
-
+    // The Dockerfile is sent to the node via a quoted heredoc below; no local
+    // file write is needed (an earlier version wrote `recipes/temp/…` and
+    // removed it — pointless, and it hardcoded "temp" instead of the recipe).
     let create_dockerfile = format!(
         "mkdir -p /tmp/sm121_docker && cat > /tmp/sm121_docker/Dockerfile << 'DOCKERFILE_END'\n{}DOCKERFILE_END",
         dockerfile
@@ -279,7 +277,6 @@ async fn run_cluster_repair(
         );
     }
 
-    tokio::fs::remove_file(&dockerfile_path).await.ok();
     Ok(repaired_tag)
 }
 
@@ -336,7 +333,10 @@ pub async fn try_repaired_image(
 
     match run_ssh(config, head_node, check_cmd, false).await {
         Ok(output) if !output.trim().is_empty() => {
-            info!("🩺 Found repaired image: {}", repaired_tag);
+            info!(
+                "{}",
+                fluent_i18n::t!("doctor-found-repaired", { "tag" => &repaired_tag })
+            );
             Ok(repaired_tag)
         }
         _ => Ok(recipe.docker_image.clone()),

+ 8 - 0
src/ends.rs

@@ -15,6 +15,14 @@ pub trait NodeProber {
 }
 
 /// Power on/off smart plugs via Home Assistant.
+///
+/// Intentionally models only the **power-on** path (`entity_is_on` /
+/// `turn_on`): orchestrate's `prepare_cluster` auto-powers-on a cluster
+/// when nodes are unreachable, and that is the only place the trait is
+/// used. `power off` is a direct user command (`power.rs`) that talks to
+/// `HaClient` directly — it does not flow through orchestrate, so it has
+/// no need of the trait abstraction. If orchestrate ever needs to power
+/// off, add `turn_off` here and route `power_off` through it.
 pub trait PowerController {
     /// Check whether a HA entity is currently in state `"on"`.
     fn entity_is_on(&self, entity: &str) -> impl std::future::Future<Output = Option<bool>> + Send;

+ 6 - 1
src/health.rs

@@ -1,4 +1,5 @@
 use anyhow::Result;
+use tracing::warn;
 
 use crate::config::{Cluster, Config, Recipe};
 use crate::tmux::{SessionHealth, TmuxManager};
@@ -94,7 +95,11 @@ pub async fn check_cluster_health(
                     status: SessionStatus::Unknown,
                 });
             }
-            Err(_) => {}
+            Err(e) => {
+                // A health-check task panicked (JoinError) — log it so a node
+                // silently reported "Unknown" is diagnosable.
+                warn!("Health-check task panicked: {}", e);
+            }
         }
     }
 

+ 14 - 12
src/host_keys.rs

@@ -34,7 +34,7 @@ pub async fn init_host_keys(
 
     let content = fs::read_to_string(config_path)
         .with_context(|| format!("Failed to read {}", config_path))?;
-    let new_content = update_host_keys_toml(&content, &updates);
+    let new_content = update_host_keys_toml(&content, &updates)?;
 
     if content == new_content {
         println!("\nNo changes needed in {}", config_path);
@@ -61,7 +61,7 @@ async fn collect_updates(
         for node in &cluster.nodes {
             match fetch_host_key(config, node).await {
                 Ok(fp) => out.push((node.name.clone(), node.ssh_ip.clone(), fp)),
-                Err(e) => eprintln!(
+                Err(e) => println!(
                     "{}",
                     fluent_i18n::t!("host-key-error", { "name" => &node.name, "ip" => &node.ssh_ip, "error" => &e.to_string() })
                 ),
@@ -74,16 +74,18 @@ async fn collect_updates(
 /// Parse `content` as TOML, walk the `[[clusters]]` tables, find each
 /// node whose `name` matches an entry in `updates`, and set the
 /// `host_key` field on it.  Preserves comments and formatting.
-fn update_host_keys_toml(content: &str, updates: &[(String, String, String)]) -> String {
+fn update_host_keys_toml(content: &str, updates: &[(String, String, String)]) -> Result<String> {
     use toml_edit::{DocumentMut, Value};
 
-    let mut doc = content.parse::<DocumentMut>().expect("valid TOML config");
+    let mut doc = content
+        .parse::<DocumentMut>()
+        .with_context(|| "config is not valid TOML")?;
     let clusters = doc
         .get_mut("clusters")
         .and_then(|v| v.as_array_of_tables_mut());
 
     let Some(clusters) = clusters else {
-        return content.to_string();
+        return Ok(content.to_string());
     };
 
     for cluster in clusters.iter_mut() {
@@ -107,7 +109,7 @@ fn update_host_keys_toml(content: &str, updates: &[(String, String, String)]) ->
         }
     }
 
-    doc.to_string()
+    Ok(doc.to_string())
 }
 
 #[cfg(test)]
@@ -128,7 +130,7 @@ nodes = [
             "10.0.0.1".to_string(),
             "SHA256:abc".to_string(),
         )];
-        let result = update_host_keys_toml(content, &updates);
+        let result = update_host_keys_toml(content, &updates).unwrap();
         assert!(result.contains(r#"host_key = "SHA256:abc""#));
         assert!(result.contains("dgx1.lan"));
     }
@@ -143,7 +145,7 @@ nodes = [
 ]
 "#;
         let updates = vec![("x".to_string(), "10.0.0.1".to_string(), "NEW".to_string())];
-        let result = update_host_keys_toml(content, &updates);
+        let result = update_host_keys_toml(content, &updates).unwrap();
         assert!(result.contains(r#"host_key = "NEW""#));
         assert!(!result.contains(r#"host_key = "OLD""#));
     }
@@ -163,7 +165,7 @@ nodes = [
             "10.0.0.1".to_string(),
             "SHA256:newA".to_string(),
         )];
-        let result = update_host_keys_toml(content, &updates);
+        let result = update_host_keys_toml(content, &updates).unwrap();
         assert!(
             result.contains(r#"host_key = "SHA256:newA""#),
             "result: {result}"
@@ -195,7 +197,7 @@ nodes = [
             ("n1".to_string(), "10.0.0.1".to_string(), "FP1".to_string()),
             ("n2".to_string(), "10.0.0.2".to_string(), "FP2".to_string()),
         ];
-        let result = update_host_keys_toml(content, &updates);
+        let result = update_host_keys_toml(content, &updates).unwrap();
         assert!(
             result.contains(r#"host_key = "FP1""#),
             "result has FP1: {result}"
@@ -209,7 +211,7 @@ nodes = [
     #[test]
     fn preserves_trailing_newline() {
         let content = "[[clusters]]\nname = \"c\"\nnodes = []\n";
-        let result = update_host_keys_toml(content, &[]);
+        let result = update_host_keys_toml(content, &[]).unwrap();
         assert_eq!(result, content);
     }
 
@@ -219,7 +221,7 @@ nodes = [
 name = "c"
 nodes = [{ name = "n1", rank = 0 }]
 "#;
-        let result = update_host_keys_toml(content, &[]);
+        let result = update_host_keys_toml(content, &[]).unwrap();
         assert_eq!(result, content);
     }
 }

+ 18 - 15
src/main.rs

@@ -27,6 +27,7 @@ use crate::config::{
     Cluster, Config, default_config_dir, load_recipe, resolve_cluster, select_cluster_by_name,
 };
 use crate::ssh::fetch_host_key;
+use std::collections::HashSet;
 
 /// Resolve the path to the config file.
 ///
@@ -300,12 +301,7 @@ async fn handle_stop(
     }
     if power_off {
         let cluster = resolve_cluster_for_op(config, recipe_name, cluster).await?;
-        let node_str = if node.is_empty() {
-            None
-        } else {
-            Some(node[0].as_str())
-        };
-        crate::power::power_off(config, &cluster, node_str, dry_run).await?;
+        crate::power::power_off(config, &cluster, node, dry_run).await?;
     }
     Ok(())
 }
@@ -322,13 +318,20 @@ async fn handle_health(
             None => config.clusters.clone(),
         };
         for cluster in &clusters_to_check {
-            println!("\n=== Кластер: {} ===", cluster.name);
+            println!(
+                "\n{}",
+                fluent_i18n::t!("cluster-section", { "name" => &cluster.name })
+            );
             for node in &cluster.nodes {
                 match fetch_host_key(config, node).await {
-                    Ok(fp) => {
-                        println!("{} ({}) -> host_key = \"{}\"", node.name, node.ssh_ip, fp)
-                    }
-                    Err(e) => println!("{} ({}) -> error: {}", node.name, node.ssh_ip, e),
+                    Ok(fp) => println!(
+                        "{}",
+                        fluent_i18n::t!("host-key-line", { "name" => &node.name, "ip" => &node.ssh_ip, "key" => &fp })
+                    ),
+                    Err(e) => println!(
+                        "{}",
+                        fluent_i18n::t!("host-key-error", { "name" => &node.name, "ip" => &node.ssh_ip, "error" => &e.to_string() })
+                    ),
                 }
             }
         }
@@ -371,11 +374,13 @@ async fn handle_power(config: &Config, action: PowerAction, dry_run: bool) -> Re
     match action {
         PowerAction::Off { cluster, node } => {
             let cluster = select_cluster_by_name(config, &cluster)?;
-            power::power_off(config, &cluster, node.as_deref(), dry_run).await?;
+            let nodes = node.into_iter().collect::<Vec<_>>();
+            power::power_off(config, &cluster, &nodes, dry_run).await?;
         }
         PowerAction::On { cluster, node } => {
             let cluster = select_cluster_by_name(config, &cluster)?;
-            power::power_on(config, &cluster, node.as_deref(), dry_run).await?;
+            let nodes = node.into_iter().collect::<Vec<_>>();
+            power::power_on(config, &cluster, &nodes, dry_run).await?;
         }
         PowerAction::Switch { from, to } => {
             let from = select_cluster_by_name(config, &from)?;
@@ -389,8 +394,6 @@ async fn handle_power(config: &Config, action: PowerAction, dry_run: bool) -> Re
 /// Resolve a cluster for container operations (stop/logs).
 /// When `auto`, tries live container label lookup first, then falls back
 /// to config-based auto-selection (for dry-run / offline scenarios).
-use std::collections::HashSet;
-
 async fn resolve_cluster_for_op(
     config: &Config,
     recipe_name: &str,

+ 54 - 24
src/orchestrate/mod.rs

@@ -19,37 +19,66 @@ use crate::ends::{NodePinger, NodeProber, PowerController, cluster_entities};
 ///
 /// Returns a `Cluster` whose `nodes` field contains only free (reachable
 /// and not occupied) nodes, plus the matching NCCL config.
-#[allow(clippy::too_many_arguments)]
+///
+/// Timing for [`wait_for_all_nodes`] after a power-on: how long to wait
+/// for the machines to boot before the first poll, the poll interval, and
+/// the max total poll window. Injectable so tests don't sleep real seconds
+/// (a previous hardcoded 60s boot wait made the whole suite take 60s).
+#[derive(Clone, Debug)]
+pub struct BootTiming {
+    pub boot_wait_secs: u64,
+    pub poll_interval_secs: u64,
+    pub max_poll_secs: u64,
+}
+
+impl Default for BootTiming {
+    fn default() -> Self {
+        Self {
+            boot_wait_secs: 60,
+            poll_interval_secs: 10,
+            max_poll_secs: 120,
+        }
+    }
+}
+
+/// Bundles the injectable end-effectors + power-on flag so
+/// [`prepare_cluster`] stays under clippy's `too_many_arguments` threshold
+/// without an `#[allow]`.
+pub struct PrepareRequest<'a, P: NodeProber, C: PowerController, N: NodePinger> {
+    pub prober: &'a P,
+    pub power: Option<&'a C>,
+    pub pinger: &'a N,
+    pub auto_power_on: bool,
+    pub boot_timing: BootTiming,
+}
+
 pub async fn prepare_cluster<P, C, N>(
     config: &Config,
     recipe: &Recipe,
     cluster_name: &str,
     node_name: Option<&str>,
-    auto_power_on: bool,
-    prober: &P,
-    power: Option<&C>,
-    pinger: &N,
+    req: &PrepareRequest<'_, P, C, N>,
 ) -> Result<(Cluster, NcclConfig)>
 where
     P: NodeProber,
     C: PowerController,
     N: NodePinger,
 {
-    let state = prober.probe(config).await;
+    let state = req.prober.probe(config).await;
 
     let resolved = if let Some(nn) = node_name {
         resolve_solo_node(config, nn)?
     } else {
         // Auto-power-on: if any nodes are unreachable, try powering on.
-        let state = if auto_power_on && state_has_unreachable(&state, config) {
+        let state = if req.auto_power_on && state_has_unreachable(&state, config) {
             let (c, _nccl) = resolve_cluster_simple(config, recipe, cluster_name)?;
-            if let Some(pwr) = power {
-                power_on_if_off(&state, &c, pwr).await?;
+            if let Some(pwr) = req.power {
+                power_on_if_off(&c, pwr).await?;
             }
             // Wait for nodes to become reachable.
-            wait_for_all_nodes(config, &c.nodes, pinger).await?;
+            wait_for_all_nodes(config, &c.nodes, req.pinger, &req.boot_timing).await?;
             // Re-probe after power-on.
-            prober.probe(config).await
+            req.prober.probe(config).await
         } else {
             state
         };
@@ -177,11 +206,7 @@ fn state_has_unreachable(state: &ClusterState, config: &Config) -> bool {
 }
 
 /// Power on HA entities that are not already on.
-async fn power_on_if_off(
-    _state: &ClusterState,
-    cluster: &Cluster,
-    power: &impl PowerController,
-) -> Result<bool> {
+async fn power_on_if_off(cluster: &Cluster, power: &impl PowerController) -> Result<bool> {
     let entities = cluster_entities(cluster);
     if entities.is_empty() {
         return Ok(false);
@@ -215,17 +240,20 @@ async fn wait_for_all_nodes(
     config: &Config,
     nodes: &[Node],
     pinger: &impl NodePinger,
+    timing: &BootTiming,
 ) -> Result<()> {
     info!("{}", fluent_i18n::t!("cluster-waiting-for-nodes"));
 
-    let boot_wait_secs = 60;
-    info!("⏳ Waiting {}s for nodes to boot...", boot_wait_secs);
-    tokio::time::sleep(Duration::from_secs(boot_wait_secs)).await;
+    if timing.boot_wait_secs > 0 {
+        info!(
+            "{}",
+            fluent_i18n::t!("boot-wait", { "secs" => timing.boot_wait_secs })
+        );
+        tokio::time::sleep(Duration::from_secs(timing.boot_wait_secs)).await;
+    }
 
-    let max_poll_secs = 120;
-    let poll_interval_secs = 10;
     let mut waited = 0;
-    while waited < max_poll_secs {
+    while waited < timing.max_poll_secs {
         let mut all_online = true;
         for node in nodes {
             if !pinger.is_reachable(config, node).await {
@@ -237,8 +265,10 @@ async fn wait_for_all_nodes(
             info!("{}", fluent_i18n::t!("cluster-nodes-available"));
             return Ok(());
         }
-        tokio::time::sleep(Duration::from_secs(poll_interval_secs)).await;
-        waited += poll_interval_secs;
+        tokio::time::sleep(Duration::from_secs(timing.poll_interval_secs)).await;
+        // Advance by at least 1s so a zero poll-interval (tests) can't spin
+        // forever when the pinger reports unreachable.
+        waited += timing.poll_interval_secs.max(1);
     }
     anyhow::bail!("{}", fluent_i18n::t!("cluster-nodes-not-available-timeout"));
 }

+ 44 - 16
src/orchestrate/tests.rs

@@ -179,10 +179,17 @@ async fn cluster_off_auto_power_on_powers_and_selects() {
         &recipe,
         "auto",
         None,
-        true,
-        &prober,
-        Some(&power),
-        &pinger,
+        &PrepareRequest {
+            prober: &prober,
+            power: Some(&power),
+            pinger: &pinger,
+            auto_power_on: true,
+            boot_timing: BootTiming {
+                boot_wait_secs: 0,
+                poll_interval_secs: 0,
+                max_poll_secs: 10,
+            },
+        },
     )
     .await
     .unwrap();
@@ -228,10 +235,17 @@ async fn cluster_off_no_auto_power_fails() {
         &recipe,
         "auto",
         None,
-        false,
-        &prober,
-        Some(&power),
-        &pinger,
+        &PrepareRequest {
+            prober: &prober,
+            power: Some(&power),
+            pinger: &pinger,
+            auto_power_on: false,
+            boot_timing: BootTiming {
+                boot_wait_secs: 0,
+                poll_interval_secs: 0,
+                max_poll_secs: 10,
+            },
+        },
     )
     .await;
 
@@ -276,10 +290,17 @@ async fn cluster_already_on_selects_directly() {
         &recipe,
         "auto",
         None,
-        true,
-        &prober,
-        Some(&power),
-        &pinger,
+        &PrepareRequest {
+            prober: &prober,
+            power: Some(&power),
+            pinger: &pinger,
+            auto_power_on: true,
+            boot_timing: BootTiming {
+                boot_wait_secs: 0,
+                poll_interval_secs: 0,
+                max_poll_secs: 10,
+            },
+        },
     )
     .await
     .unwrap();
@@ -323,10 +344,17 @@ async fn occupied_nodes_excluded() {
         &recipe,
         "auto",
         None,
-        false,
-        &prober,
-        Some(&power),
-        &pinger,
+        &PrepareRequest {
+            prober: &prober,
+            power: Some(&power),
+            pinger: &pinger,
+            auto_power_on: false,
+            boot_timing: BootTiming {
+                boot_wait_secs: 0,
+                poll_interval_secs: 0,
+                max_poll_secs: 10,
+            },
+        },
     )
     .await
     .unwrap();

+ 80 - 62
src/power.rs

@@ -22,10 +22,10 @@ use crate::home_assistant::Client as HaClient;
 pub async fn power_off(
     config: &Config,
     cluster: &Cluster,
-    node_name: Option<&str>,
+    node_names: &[String],
     dry_run: bool,
 ) -> Result<()> {
-    let targets = select_targets(cluster, node_name, "power off")?;
+    let targets = select_targets(cluster, node_names, "power off")?;
 
     if dry_run {
         warn!("{}", fluent_i18n::t!("power-off-dryrun"));
@@ -57,8 +57,8 @@ pub async fn power_off(
     // by design: already-offline nodes get their HA entity toggled off,
     // which is idempotent.
     info!(
-        "🛑 power off: {} node(s) — sudo shutdown now",
-        targets.len()
+        "{}",
+        fluent_i18n::t!("power-off-head", { "nodes" => targets.len() })
     );
     let mut handles = Vec::with_capacity(targets.len());
     for node in targets.iter().cloned() {
@@ -122,10 +122,10 @@ pub async fn power_off(
 pub async fn power_on(
     config: &Config,
     cluster: &Cluster,
-    node_name: Option<&str>,
+    node_names: &[String],
     dry_run: bool,
 ) -> Result<()> {
-    let targets = select_targets(cluster, node_name, "power on")?;
+    let targets = select_targets(cluster, node_names, "power on")?;
 
     let Some(ha_cfg) = &config.home_assistant else {
         anyhow::bail!(
@@ -246,40 +246,50 @@ pub async fn power_switch(
         .collect();
     let to_extras: std::collections::HashSet<&str> =
         to.extra_entities.iter().map(|e| e.as_str()).collect();
-    let disable_extras: Vec<&String> = from
+    let disable_extras: Vec<String> = from
         .extra_entities
         .iter()
         .filter(|e| !to_extras.contains(e.as_str()))
+        .cloned()
         .collect();
 
     info!(
-        "🔌 switch: {} → {} ({} node(s) + {} entity(ies) to disable)",
-        from.name,
-        to.name,
-        disable_nodes.len(),
-        disable_extras.len()
+        "{}",
+        fluent_i18n::t!("power-switch-head", { "from" => &from.name, "to" => &to.name, "nodes" => disable_nodes.len(), "entities" => disable_extras.len() })
     );
 
     if dry_run {
         for node in &disable_nodes {
             println!(
-                "would: ssh {}@{} sudo shutdown now",
-                config.ssh_user, node.ssh_ip
+                "{}",
+                fluent_i18n::t!("power-off-would", { "action" => format!("ssh {}@{} sudo shutdown now", config.ssh_user, node.ssh_ip), "target" => String::new() })
             );
             if let Some(entity) = &node.entity {
-                println!("would: HA turn_off {}", entity);
+                println!(
+                    "{}",
+                    fluent_i18n::t!("power-switch-would-off", { "entity" => entity })
+                );
             }
         }
         for extra in &disable_extras {
-            println!("would: HA turn_off {} (from extras)", extra);
+            println!(
+                "{}",
+                fluent_i18n::t!("power-switch-would-off-extra", { "extra" => extra })
+            );
         }
         for node in &to.nodes {
             if let Some(entity) = &node.entity {
-                println!("would: HA turn_on {}", entity);
+                println!(
+                    "{}",
+                    fluent_i18n::t!("power-switch-would-on", { "entity" => entity })
+                );
             }
         }
         for extra in &to.extra_entities {
-            println!("would: HA turn_on {} (to extras)", extra);
+            println!(
+                "{}",
+                fluent_i18n::t!("power-switch-would-on-extra", { "extra" => extra })
+            );
         }
         return Ok(());
     }
@@ -325,35 +335,15 @@ pub async fn power_switch(
         let ha = HaClient::new(ha_cfg)?;
         crate::home_assistant::warn_if_token_looks_short(&ha_cfg.token);
 
-        for node in &disable_nodes {
-            if let Some(entity) = &node.entity {
-                if let Err(e) = split_entity(entity) {
-                    warn!(
-                        "{}",
-                        fluent_i18n::t!("ha-entity-error", { "name" => &node.name, "entity" => entity, "error" => e.to_string() })
-                    );
-                    continue;
-                }
-                if dry_run {
-                    println!("would: HA turn_off {}", entity);
-                } else {
-                    let _ = ha.turn_off(entity).await;
-                }
-            }
-        }
-        for extra in &disable_extras {
-            if dry_run {
-                println!("would: HA turn_off {} (from extras)", extra);
-            } else {
-                let _ = ha.turn_off(extra).await;
-            }
-        }
+        // Disable phase: per-node entities + extras unique to `from`.
+        // Routed through `toggle_entities` (which validates + logs failures)
+        // instead of ad-hoc loops with `let _ =` (which swallowed errors)
+        // and dead `if dry_run` branches (dry-run returned at the top).
+        toggle_entities(&ha, "turn_off", &disable_nodes, &disable_extras, "from").await;
 
         info!(
-            "💡 switch: powering on {} node(s) + {} entity(ies) to {}",
-            to.nodes.len(),
-            to.extra_entities.len(),
-            to.name
+            "{}",
+            fluent_i18n::t!("power-switch-powering-on", { "nodes" => to.nodes.len(), "entities" => to.extra_entities.len(), "to" => &to.name })
         );
         toggle_entities(&ha, "turn_on", &to.nodes, &to.extra_entities, "to").await;
     } else {
@@ -362,25 +352,30 @@ pub async fn power_switch(
     Ok(())
 }
 
-fn select_targets(cluster: &Cluster, node_name: Option<&str>, op: &str) -> Result<Vec<Node>> {
-    let targets: Vec<Node> = match node_name {
-        Some(name) => cluster
-            .nodes
-            .iter()
-            .filter(|n| n.name == *name)
-            .cloned()
-            .collect(),
-        None => cluster.nodes.clone(),
-    };
-    if targets.is_empty() {
-        if let Some(n) = node_name {
+fn select_targets(cluster: &Cluster, node_names: &[String], op: &str) -> Result<Vec<Node>> {
+    let targets: Vec<Node> = if node_names.is_empty() {
+        cluster.nodes.clone()
+    } else {
+        let mut found: Vec<Node> = Vec::new();
+        let mut missing: Vec<String> = Vec::new();
+        for nn in node_names {
+            if let Some(node) = cluster.nodes.iter().find(|n| &n.name == nn) {
+                found.push(node.clone());
+            } else {
+                missing.push(nn.clone());
+            }
+        }
+        if !missing.is_empty() {
             anyhow::bail!(
-                "{}: node '{}' not found in cluster '{}'",
+                "{}: node(s) {} not found in cluster '{}'",
                 op,
-                n,
+                missing.join(", "),
                 cluster.name
             );
         }
+        found
+    };
+    if targets.is_empty() {
         anyhow::bail!("{}: cluster '{}' has no nodes", op, cluster.name);
     }
     Ok(targets)
@@ -439,7 +434,7 @@ ib_disable = 0
     fn select_targets_filters_by_node() {
         let cfg = cfg_with_ha();
         let c = crate::config::select_cluster_by_name(&cfg, "x").unwrap();
-        let t = select_targets(&c, Some("n1"), "test").unwrap();
+        let t = select_targets(&c, &["n1".to_string()], "test").unwrap();
         assert_eq!(t.len(), 1);
         assert_eq!(t[0].name, "n1");
     }
@@ -448,15 +443,38 @@ ib_disable = 0
     fn select_targets_all() {
         let cfg = cfg_with_ha();
         let c = crate::config::select_cluster_by_name(&cfg, "x").unwrap();
-        let t = select_targets(&c, None, "test").unwrap();
+        let t = select_targets(&c, &[], "test").unwrap();
         assert_eq!(t.len(), 2);
     }
 
+    #[test]
+    fn select_targets_multiple_nodes() {
+        // `stop --node a --node b --off` must power off *both* nodes, not just
+        // the first. (Regression for I-5.)
+        let cfg = cfg_with_ha();
+        let c = crate::config::select_cluster_by_name(&cfg, "x").unwrap();
+        let t = select_targets(&c, &["n1".to_string(), "n2".to_string()], "test").unwrap();
+        assert_eq!(t.len(), 2);
+        let names: Vec<_> = t.iter().map(|n| n.name.as_str()).collect();
+        assert!(names.contains(&"n1"));
+        assert!(names.contains(&"n2"));
+    }
+
+    #[test]
+    fn select_targets_partial_missing_errors() {
+        let cfg = cfg_with_ha();
+        let c = crate::config::select_cluster_by_name(&cfg, "x").unwrap();
+        // One valid + one missing → error listing the missing one.
+        let r = select_targets(&c, &["n1".to_string(), "nope".to_string()], "test");
+        assert!(r.is_err());
+        assert!(r.unwrap_err().to_string().contains("nope"));
+    }
+
     #[test]
     fn select_targets_unknown_node_errors() {
         let cfg = cfg_with_ha();
         let c = crate::config::select_cluster_by_name(&cfg, "x").unwrap();
-        let r = select_targets(&c, Some("nope"), "test");
+        let r = select_targets(&c, &["nope".to_string()], "test");
         assert!(r.is_err());
     }
 

+ 1 - 5
src/tmux/mod.rs

@@ -10,10 +10,7 @@ use crate::config::{Config, NcclConfig, Node, Recipe};
 
 /// Manages tmux sessions for model inference across a DGX Spark cluster.
 #[derive(Clone)]
-pub struct TmuxManager {
-    #[allow(dead_code)]
-    pub session_prefix: String,
-}
+pub struct TmuxManager;
 
 impl TmuxManager {
     /// Generates a namespaced session name for a recipe and node rank.
@@ -60,7 +57,6 @@ impl SessionHealth {
         self.session_exists && self.process_alive && self.gpu_ok && self.api_ok
     }
 
-    #[allow(dead_code)]
     pub fn is_degraded(&self) -> bool {
         self.session_exists && !self.process_alive
     }

+ 27 - 20
src/tmux/ops.rs

@@ -26,9 +26,7 @@ pub async fn tmux_stop(
     let (cluster, _) =
         crate::config::select_cluster(config, &choice, recipe.required_nodes, &HashSet::new())?;
 
-    let manager = TmuxManager {
-        session_prefix: recipe_name.to_string(),
-    };
+    let manager = TmuxManager;
 
     info!(
         "🛑 Stopping tmux sessions for '{}' on cluster '{}'...",
@@ -39,7 +37,10 @@ pub async fn tmux_stop(
     for node in &cluster.nodes {
         let sess = manager.session_name(&recipe, node.rank);
         match manager.graceful_stop(config, node, &sess, dry_run).await {
-            Ok(()) => info!("  ✅ Stopped '{}' on {}", sess, node.name),
+            Ok(()) => info!(
+                "{}",
+                fluent_i18n::t!("tmux-stopped", { "sess" => &sess, "name" => &node.name })
+            ),
             Err(e) => {
                 warn!("  ✗ Failed to stop '{}' on {}: {}", sess, node.name, e);
                 failures.push((node.name.clone(), e));
@@ -54,7 +55,7 @@ pub async fn tmux_stop(
         anyhow::bail!("Failed to stop sessions on {} node(s)", failures.len());
     }
 
-    info!("✅ All tmux sessions stopped.");
+    info!("{}", fluent_i18n::t!("tmux-all-stopped"));
     Ok(())
 }
 
@@ -78,9 +79,7 @@ pub async fn tmux_status(
     let (cluster, _) =
         crate::config::select_cluster(config, &choice, recipe.required_nodes, &HashSet::new())?;
 
-    let manager = TmuxManager {
-        session_prefix: recipe_name.to_string(),
-    };
+    let manager = TmuxManager;
 
     println!(
         "\n📊 TMUX Status: {} on cluster {}",
@@ -105,7 +104,7 @@ pub async fn tmux_status(
         );
 
         if node_health.status == health::SessionStatus::Running {
-            println!("✅ ALL OK");
+            println!("{}", fluent_i18n::t!("tmux-all-ok"));
         } else {
             let mut issues = Vec::new();
             if !node_health.session_exists {
@@ -117,7 +116,10 @@ pub async fn tmux_status(
             if !node_health.port_accessible && node_health.rank == 0 {
                 issues.push("API down");
             }
-            println!("⚠️  {}", issues.join(", "));
+            println!(
+                "{}",
+                fluent_i18n::t!("tmux-issues", { "issues" => &issues.join(", ") })
+            );
         }
 
         if node_health.session_exists {
@@ -139,7 +141,7 @@ pub async fn tmux_status(
 
     println!("{}", "─".repeat(60));
     match report.overall {
-        health::HealthStatus::Healthy => info!("✅ Cluster is healthy!"),
+        health::HealthStatus::Healthy => info!("{}", fluent_i18n::t!("tmux-cluster-healthy")),
         health::HealthStatus::Degraded => warn!("⚠️ Cluster is degraded"),
         health::HealthStatus::Unhealthy => warn!("⚠️ Cluster has issues — check details above."),
     }
@@ -168,15 +170,16 @@ pub async fn tmux_logs(
     let (cluster, _) =
         crate::config::select_cluster(config, &choice, recipe.required_nodes, &HashSet::new())?;
 
-    let manager = TmuxManager {
-        session_prefix: recipe_name.to_string(),
-    };
+    let manager = TmuxManager;
 
     if follow {
         let head_node = cluster.nodes.first().context("No nodes in cluster")?;
         let sess = manager.session_name(&recipe, 0);
 
-        info!("📜 Following logs from '{}' on {}...", sess, head_node.name);
+        info!(
+            "{}",
+            fluent_i18n::t!("tmux-following-logs", { "sess" => &sess, "name" => &head_node.name })
+        );
         let output = manager
             .stream_pane(config, head_node, &sess, 0, true, dry_run)
             .await?;
@@ -211,10 +214,16 @@ pub async fn tmux_logs(
             if let Ok((name, sess, Ok(output))) = result {
                 let trimmed = output.trim();
                 if !trimmed.is_empty() {
-                    println!("\n--- {} ({}) ---", name, sess);
+                    println!(
+                        "\n{}",
+                        fluent_i18n::t!("tmux-node-section", { "name" => &name, "sess" => &sess })
+                    );
                     println!("{}", trimmed);
                 } else {
-                    println!("\n--- {} — no output ---", name);
+                    println!(
+                        "\n{}",
+                        fluent_i18n::t!("tmux-node-no-output", { "name" => &name })
+                    );
                 }
             }
         }
@@ -249,9 +258,7 @@ pub async fn tmux_attach(
         .find(|n| n.rank == 0)
         .context("No head node (rank 0) in cluster")?;
 
-    let manager = TmuxManager {
-        session_prefix: recipe_name.to_string(),
-    };
+    let manager = TmuxManager;
     let sess = manager.session_name(&recipe, 0);
 
     info!(

+ 13 - 9
src/tmux/serve.rs

@@ -54,14 +54,12 @@ pub async fn tmux_serve(
 
     crate::before_serve::run(config, &cluster, &recipe, skip_before_serve, dry_run).await?;
 
-    let manager = TmuxManager {
-        session_prefix: recipe_name.to_string(),
-    };
+    let manager = TmuxManager;
 
     let is_ray = recipe.distributed_backend == Some(crate::config::DistributedBackend::Ray);
 
     if is_ray {
-        info!("🐍 Ray backend detected — starting head node first...");
+        info!("{}", fluent_i18n::t!("tmux-ray-backend"));
 
         let head_node = cluster
             .nodes
@@ -75,11 +73,11 @@ pub async fn tmux_serve(
             .await?;
 
         if !dry_run {
-            info!("⏳ Waiting for head node to be ready...");
+            info!("{}", fluent_i18n::t!("tmux-waiting-head"));
             let sess = manager.session_name(&recipe, 0);
             let head_ready = wait_for_head_ready(config, head_node, &sess, &recipe).await?;
             if head_ready {
-                info!("✅ Head node ready. Starting worker nodes...");
+                info!("{}", fluent_i18n::t!("tmux-head-ready"));
             } else {
                 warn!("⚠️ Head node may not be fully ready — starting workers anyway.");
             }
@@ -95,7 +93,7 @@ pub async fn tmux_serve(
                 .await?;
         }
     } else {
-        info!("🔗 MP backend detected — starting all nodes in parallel...");
+        info!("{}", fluent_i18n::t!("tmux-mp-backend"));
 
         let mut handles = Vec::new();
         for node in &cluster.nodes {
@@ -120,7 +118,10 @@ pub async fn tmux_serve(
         let mut failures = Vec::new();
         for result in results {
             match result {
-                Ok(Ok(name)) => info!("  ✅ Session started on {}", name),
+                Ok(Ok(name)) => info!(
+                    "{}",
+                    fluent_i18n::t!("tmux-session-started", { "name" => &name })
+                ),
                 Ok(Err(e)) => failures.push(e),
                 Err(e) => failures.push(anyhow::anyhow!("Task join: {}", e)),
             }
@@ -187,7 +188,10 @@ async fn wait_for_head_ready(
         if let Ok(out) = crate::ssh::run_ssh(config, head_node, api_check, false).await {
             let code = out.trim();
             if code == "200" {
-                info!("  Head node API ready (attempt {})", i + 1);
+                info!(
+                    "{}",
+                    fluent_i18n::t!("tmux-head-api-ready", { "attempt" => i + 1 })
+                );
                 return Ok(true);
             }
         }

+ 25 - 19
src/tmux/session.rs

@@ -4,6 +4,7 @@ use tokio::time::sleep;
 use tracing::{info, warn};
 
 use crate::config::{Cluster, Config, Node, Recipe};
+use crate::ssh::shq;
 use crate::ssh::{run_ssh, run_ssh_with_retry};
 use crate::tmux::{SessionHealth, TmuxManager};
 
@@ -28,7 +29,10 @@ impl TmuxManager {
         let output = run_ssh(config, node, script, dry_run).await?;
 
         if output.trim().is_empty() {
-            info!("  ✅ tmux session created on {}", node.name);
+            info!(
+                "{}",
+                fluent_i18n::t!("tmux-session-created", { "name" => &node.name })
+            );
         } else {
             info!("  tmux session output on {}: {}", node.name, output.trim());
         }
@@ -46,15 +50,15 @@ impl TmuxManager {
     ) -> Result<bool> {
         if dry_run {
             info!(
-                "  ssh {}@{}: tmux list-sessions 2>/dev/null | grep '{}'",
+                "  ssh {}@{}: tmux list-sessions 2>/dev/null | grep -F '{}'",
                 config.ssh_user, node.ssh_ip, session_name
             );
             return Ok(true);
         }
 
         let cmd = format!(
-            "tmux list-sessions 2>/dev/null | grep -c '{}'",
-            session_name
+            "tmux list-sessions 2>/dev/null | grep -Fc {}",
+            shq(session_name)
         );
         let output = run_ssh(config, node, cmd, dry_run).await?;
 
@@ -96,13 +100,13 @@ impl TmuxManager {
         let process_alive = if exists {
             let pid_cmd = format!(
                 "tmux list-panes -t {}:0 -F '#{{pane_pid}}' | head -1",
-                session_name
+                shq(session_name)
             );
             match run_ssh(config, node, pid_cmd, dry_run).await {
                 Ok(pid_str) => {
                     let pid = pid_str.trim();
                     if !pid.is_empty() && pid != "0" {
-                        let ps_cmd = format!("ps -p {} -o pid= 2>/dev/null", pid);
+                        let ps_cmd = format!("ps -p {} -o pid= 2>/dev/null", shq(pid));
                         run_ssh(config, node, ps_cmd, dry_run)
                             .await
                             .map(|o| !o.trim().is_empty())
@@ -179,7 +183,7 @@ impl TmuxManager {
         let _ = run_ssh(
             config,
             node,
-            format!("tmux send-keys -t {}:0 C-c", session_name),
+            format!("tmux send-keys -t {}:0 C-c", shq(session_name)),
             dry_run,
         )
         .await;
@@ -189,7 +193,7 @@ impl TmuxManager {
             sleep(Duration::from_secs(2)).await;
             let check_cmd = format!(
                 "tmux list-panes -t {}:0 -F '#{{pane_pid}}' 2>/dev/null | head -1",
-                session_name
+                shq(session_name)
             );
             match run_ssh(config, node, check_cmd, dry_run).await {
                 Ok(pid_str) => {
@@ -203,7 +207,7 @@ impl TmuxManager {
                         );
                         break;
                     }
-                    let ps_cmd = format!("ps -p {} -o pid= 2>/dev/null", pid);
+                    let ps_cmd = format!("ps -p {} -o pid= 2>/dev/null", shq(pid));
                     match run_ssh(config, node, ps_cmd, dry_run).await {
                         Ok(ps_out) if ps_out.trim().is_empty() => {
                             exited = true;
@@ -232,7 +236,7 @@ impl TmuxManager {
             let _ = run_ssh(
                 config,
                 node,
-                format!("tmux kill-session -t {}", session_name),
+                format!("tmux kill-session -t {}", shq(session_name)),
                 dry_run,
             )
             .await;
@@ -266,20 +270,22 @@ impl TmuxManager {
             return Ok(String::new());
         }
 
+        let sess_q = shq(session_name);
         let cmd = if follow {
+            // pipe-pane runs `cat > LOG` inside the pane's tmux shell; the
+            // redirect target is quoted so a session name with a quote/space
+            // can't break out. `tail -f` then streams the same file.
+            let log_name = session_name.replace('-', "_");
+            let log_path = format!("/tmp/sm121_pipe_{}_{}.log", log_name, window);
+            let log_q = shq(&log_path);
             format!(
-                "tmux pipe-pane -o -t {}:{} 'cat > /tmp/sm121_pipe_{}_{}.log' && tail -f /tmp/sm121_pipe_{}_{}.log",
-                session_name,
-                window,
-                session_name.replace('-', "_"),
-                window,
-                session_name.replace('-', "_"),
-                window
+                "tmux pipe-pane -o -t {}:{} \"cat > {}\" && tail -f {}",
+                sess_q, window, log_q, log_q
             )
         } else {
             format!(
                 "tmux capture-pane -t {}:{} -p -S -200 2>/dev/null || echo 'no output'",
-                session_name, window
+                sess_q, window
             )
         };
 
@@ -306,7 +312,7 @@ impl TmuxManager {
             "  Attaching to '{}' on {}... Press Ctrl+B D to detach.",
             session_name, node.name
         );
-        info!("  (russh does not support interactive PTY. Use the following command directly:)");
+        info!("  {}", fluent_i18n::t!("tmux-no-pty"));
         println!(
             "  ssh -t {}@{} tmux attach-session -t {}",
             config.ssh_user, node.ssh_ip, session_name