Bläddra i källkod

docs: restructure for end users: DEVELOPMENT.md + user-friendly README

- DEVELOPMENT.md: architecture, module layout, quality gates, dev workflow
- README.md: rewritten for end users (install, configure, use the binary)
- AGENTS.md: condensed to concise agent quick-reference card
Alexander Korolev 4 månader sedan
förälder
incheckning
30efdff974
3 ändrade filer med 375 tillägg och 701 borttagningar
  1. 69 318
      AGENTS.md
  2. 143 0
      DEVELOPMENT.md
  3. 163 383
      README.md

+ 69 - 318
AGENTS.md

@@ -1,351 +1,102 @@
-# AGENTS.md — vllm-spark-launcher
+# AGENTS.md — sm121-launcher
 
-Rust CLI tool for launching vLLM/SGLang containers across a DGX Spark cluster. SSH-based provisioning over `russh`. Supports two runtimes: **Docker** (default) and **tmux** (alternative).
+Rust CLI (`edition 2024`) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (`russh`). Docker default, tmux alternative.
 
-## Project Layout
+## Dev quick-ref
 
-- `Cargo.toml` — binary crate `vllm-spark-launcher`. Edition 2024.
-- `src/main.rs` — CLI entrypoint, command dispatch, i18n init.
-- `src/cli.rs` — `clap` derive-based CLI definitions (Commands, PowerAction, TmuxCommands).
-- `src/config.rs` — Config, Cluster, Node, Recipe, NcclConfig, HomeAssistantConfig deserialization; cluster resolution (exact + prefix match, auto-select smallest).
-- `src/ssh.rs` — SSH client session management, key loading, host key verification, retry logic.
-- `src/docker/mod.rs` — Docker command builder: serve, stop, health, logs, log streaming.
-- `src/docker/tests.rs` — Docker command builder tests (MP, SGLang, Ray, keep, port, env overrides).
-- `src/tmux/mod.rs` — tmux module root (session manager, serve, ops).
-- `src/tmux/serve.rs` — tmux serve logic (window/panel layout).
-- `src/tmux/ops.rs` — tmux stop/status/logs/attach operations.
-- `src/health.rs` — tmux health-check helpers.
-- `src/power.rs` — `power on/off/switch` flows (SSH shutdown + HA toggle).
-- `src/home_assistant.rs` — HA REST API client (turn_on/turn_off).
-- `src/host_keys.rs` — `init-host-keys` command (auto-write SSH host_keys).
-- `src/exec.rs` — `exec` subcommand: run commands on cluster/nodes, resolve aliases.
-- `src/doctor.rs` — Doctor mode: pre-flight checks and `--repair` (Dockerfile generation).
-- `src/pre_serve.rs` — BeforeServe/PreServeCommand types.
-- `src/before_serve.rs` — Pre-serve hook execution logic.
-- `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent i18n translation files.
-- `config.toml` — runtime config: SSH key, cluster definitions, per-cluster NCCL settings, HA, exec aliases.
-- `recipes/<name>/recipe.toml` — recipe files.
+| Action | Command |
+|--------|---------|
+| Build | `cargo build` |
+| All gates | `make check` (fmt → clippy → test → ≤500 lines) |
+| Dry-run | `cargo run -- -d serve --recipe deepseek-v4-flash` |
+| Tests | `cargo test` (65, all passing) |
+| l10n | `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent FTL files |
 
-## Commands
+See [`DEVELOPMENT.md`](DEVELOPMENT.md) for architecture, module layout, and dev workflow.
 
-### Docker runtime
-```
-cargo run serve --recipe glm51                             # Auto-cluster selection
-cargo run serve --recipe glm51 --cluster 4nodes-switch     # Explicit cluster
-cargo run serve --recipe glm51 --skip-before-serve         # Bypass pre-serve hooks
-cargo run -- -d serve --recipe glm51                       # Dry-run (print only)
-cargo run stop --recipe glm51                              # Stop containers on all nodes
-cargo run health                                           # Health check all clusters
-cargo run health --cluster 2nodes-dac                      # Single cluster
-cargo run health --fingerprint                             # Show host keys for config.toml
-cargo run init-host-keys                                   # Auto-write SSH host_keys
-cargo run exec --cluster 4nodes -- "..."                   # Run on all nodes
-cargo run exec --cluster 4nodes --node dgx1 -- "uptime"    # Run on one node
-cargo run exec --recipe glm51 -- "cat /tmp/foo"            # Use recipe's cluster
-cargo run exec --cluster 4nodes --alias gpu-lock           # Use exec alias
-cargo run logs --recipe glm51                              # Last 100 lines per node
-cargo run logs --recipe glm51 --follow                     # Follow head node logs
-cargo run doctor --recipe glm51                            # Check node prerequisites
-cargo run doctor --recipe glm51 --repair                   # Auto-fix (build repaired image)
+## Commands
 
-### Power management
 ```
-cargo run power off --cluster 4nodes-switch                # SSH shutdown + HA turn_off
-cargo run power off --cluster 4 --node dgx1                # Prefix matching + single node
-cargo run power on --cluster 4nodes-switch                 # HA turn_on (wake nodes)
-cargo run power switch --from 4 --to 2                     # Switch clusters (disable → enable)
-cargo run -- -d power off --cluster 4nodes-switch          # Dry-run
+serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys
 ```
 
-### SGLang recipe example
-```
-cargo run serve --recipe minimax-m3                        # SGLang, 4 nodes, TP=4
-cargo run -- -d serve --recipe minimax-m3                  # Dry-run (see the commands)
-cargo run stop --recipe minimax-m3                         # Stop SGLang containers
-cargo run logs --recipe minimax-m3                         # Check SGLang logs
-```
+Cluster: exact name, prefix (`4`→`4nodes-switch`), or `auto` (smallest with NCCL + matching `required_nodes`).
 
-### Tmux runtime
-```
-cargo run tmux serve --recipe glm51                        # Launch via tmux (no Docker)
-cargo run tmux stop --recipe glm51                         # Stop tmux sessions
-cargo run tmux status --recipe glm51                       # Detailed health per node
-cargo run tmux logs --recipe glm51                         # Capture pane output
-cargo run tmux logs --recipe glm51 --follow                # Stream pane output
-cargo run tmux attach --recipe glm51                       # Interactive attach (head node)
+## Recipe format (`recipes/<name>/recipe.toml`)
+
+```toml
+engine = "vllm"                     # or "sglang"
+docker_image = "img:tag"
+model = "org/model"
+port = 30000                        # default
+required_nodes = 2                  # optional
+tensor_parallel_size = 2
+pipeline_parallel_size = 1          # vLLM only, 0 = auto
+distributed_backend = "mp"          # or "ray"
+shm_size = "64gb"                   # optional, only set when non-empty
+entrypoint = ["vllm", "serve"]      # optional, after image
+before_serve = [{ command = "echo", on_failure = "abort" }]
+args = ["--kv-cache-dtype", "fp8"]
+[env]
+KEY = "val"                         # overrides launcher defaults
 ```
 
-All subprocess calls support `-d` / `--dry-run` as a global flag.
+## Env vars (auto-set)
+
+Cluster-driven: `NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX`, `GLOO_/TP_SOCKET_IFNAME`, `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include`
+Defaults: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN`
+Engine IP: `VLLM_HOST_IP` / `SGLANG_WORKER_IP`
+Head: `VLLM_LOGGING_COLOR=1`, `TERM=xterm-256color`, `-p <port>:<port>`
+Ray: `RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS`
+Recipe `[env]` wins (last wins in `-e`).
+
+## Flags auto-generated
 
-## config.toml
+- vLLM: `--tensor-parallel-size`, `--pipeline-parallel-size`, `--nnodes`, `--node-rank`, `--master-addr`, `--master-port`, `--host`, `--port`, `--headless` (workers)
+- SGLang: `--tp`, `--nnodes`, `--node-rank`, `--dist-init-addr`, `--host`, `--port`
+- Both: `--distributed-executor-backend` (ray only, mp is auto-detected)
 
-User-specific. Has hardcoded SSH key paths — won't work without editing. Structure:
+## Config fields
 
 ```toml
-ssh_user = "username"
-ssh_key_path = "/absolute/path/to/key"
+ssh_user = "u"
+ssh_key_path = "/path"              # PathBuf
 
 [[clusters]]
-name = "my-cluster"
-master_ib_ip = "1.2.3.4"
-nodes = [
-    # host_key = "SHA256:..."  # run 'cargo run health --fingerprint' to discover
-    { name = "dgx1.lan", ssh_ip = "10.0.0.1", ib_ip = "1.2.3.4", rank = 0, entity = "switch.dgx1" },
-    { name = "dgx2.lan", ssh_ip = "10.0.0.2", ib_ip = "1.2.3.5", rank = 1, entity = "switch.dgx2" },
-]
-extra_entities = ["switch.network_switch"]    # cluster-level switches (PDU, network)
-
-[nccl."my-cluster"]
-ib_hca = "rocep1s0f0,roceP2p1s0f0"
-socket_ifname = "enp1s0f0np0"
+name = "c"
+master_ib_ip = "10.0.0.1"
+nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }]
+extra_entities = ["switch.network"] # cluster-level HA entities
+
+[nccl."c"]
+ib_hca = "..."
+socket_ifname = "..."
 ib_gid_index = 3
 ib_disable = 0
-```
-
-### Home Assistant config
 
-Optional `[home_assistant]` section for power management:
-
-```toml
 [home_assistant]
-url = "http://homeassistant.local:8123"
-token = "long-lived-access-token"
+url = "..."
+token = "..."
 shutdown_delay_seconds = 30
-```
-
-### Exec aliases
-
-```toml
-[exec_aliases]
-gpu-lock = "sudo nvidia-smi -lgc 300,2300"
-shutdown = "sudo shutdown now"
-```
-
-Use: `cargo run exec --cluster 4 --alias gpu-lock`
 
-### Recipe-level cluster override
-
-A recipe can pin itself to a specific cluster (bypasses `--cluster` / auto-pick):
-
-```toml
-cluster = "4nodes-switch"
-```
-
-### Pre-serve hooks (`[before_serve."<cluster>"]`)
-
-Run shell commands on every node of a cluster **before** `serve` / `tmux serve`
-launches containers or tmux sessions.  Defined in `config.toml`:
-
-```toml
-[before_serve."4nodes-switch"]
-commands = [
-    # Default: abort the serve if this fails
-    "sudo nvidia-smi -lgc 300,2300",
-
-    # Explicit on_failure:
-    #   "abort"    — fail-fast (default)
-    #   "warn"     — log warning, continue
-    #   "continue" — stay silent, continue
-    { command = "sudo nvidia-smi -pm 1", on_failure = "warn" },
-]
-```
-
-Recipes can also declare pre-serve commands that **append** to the cluster
-config (so recipe-level commands run after cluster-level):
-
-```toml
-before_serve = [
-    { command = "echo recipe-level", on_failure = "abort" },
-]
-```
-
-Commands are run sequentially on each node (so order matters) but nodes
-run in parallel.  Use `--skip-before-serve` to bypass for a one-off
-serve.
-
-### `exec` aliases (`[exec_aliases]`)
-
-Map short names to full shell commands for use with
-`cargo run exec --alias`.  Defined in `config.toml`:
-
-```toml
 [exec_aliases]
-gpu-lock = "sudo nvidia-smi -lgc 300,2300"
-shutdown = "sudo shutdown now"
-reboot = "sudo reboot"
-```
-
-Then:
+name = "command"
 
-```bash
-cargo run exec --cluster 4nodes --alias gpu-lock
-cargo run exec --cluster 4nodes --alias shutdown
+[before_serve."c"]
+commands = ["cmd", { command = "cmd", on_failure = "warn" }]
 ```
 
-`--alias <name>` and a trailing command are mutually exclusive.  If the
-alias name is unknown, the error message lists the configured aliases.
-
-## recipes/
-
-TOML format. Directory name is intentionally misspelled. Each recipe lives in its own subdirectory:
-
-```toml
-engine = "vllm"                        # or "sglang" (default: "vllm")
-name = "Human Readable Name"
-docker_image = "my-image:tag"
-model = "org/model-name"
-port = 30000                              # default: 30000
-required_nodes = 4                        # optional, omitted = no check
-pipeline_parallel_size = 0                # 0 = auto (equals nnodes), vLLM only
-tensor_parallel_size = 1                  # per-node TP for vLLM, total TP for SGLang
-distributed_backend = "mp"                # "mp" (multiprocessing) or "ray"
-shm_size = "64gb"                         # optional, only added when non-empty
-cluster = "4nodes-switch"                 # optional: pin to a specific cluster
-
-args = [
-    "--kv-cache-dtype", "fp8",
-    "--max-model-len", "32768",
-]
-
-entrypoint = ["vllm", "serve"]             # optional, appended after image name
-
-before_serve = [                           # recipe-level pre-serve hooks (after cluster)
-    { command = "echo recipe", on_failure = "abort" },
-]
-
-[env]
-SOME_VAR = "value"
-```
-
-### Engine support
-
-Both `vllm` and `sglang` engines are supported. The engine field controls:
-
-**Host IP env var:**
-- `vllm` → `-e VLLM_HOST_IP=<node.ib_ip>`
-- `sglang` → `-e SGLANG_WORKER_IP=<node.ib_ip>`
-
-**Model argument:**
-- `vllm` → bare model name (e.g. `org/model`)
-- `sglang` → `--model-path org/model`
-
-**Distributed flags:**
-- `vllm` → `--tensor-parallel-size 1`, `--pipeline-parallel-size <pp_size>`, `--distributed-executor-backend mp|ray`
-- `sglang` → `--tp <tensor_parallel_size>`, no `--pipeline-parallel-size`, no `--distributed-executor-backend`
-
-Both engines use `--nnodes`, `--node-rank`, and `--master-addr`.
-
-## How Docker launch works
-
-`build_docker_command` in `src/docker.rs` constructs the `docker run` command for each node. It is engine-aware:
-
-### vLLM engine (default)
-
-Appends these flags:
-
-```
---tensor-parallel-size <recipe.tensor_parallel_size>
---pipeline-parallel-size <computed>
---nnodes <total>
---node-rank <rank>
---master-addr <cluster.master_ib_ip>
---master-port <config.init_port>          # default 25000
---distributed-executor-backend ray         # only for ray backend (mp is auto-detected)
---host 0.0.0.0
---port <recipe.port>
-```
-
-Non-head nodes also receive `--headless`.
-
-### SGLang engine
-
-Appends:
-
-```
---tp <tensor_parallel_size>
---nnodes <total>
---node-rank <rank>
---dist-init-addr <master_ib_ip>:<init_port>
---host 0.0.0.0
---port <recipe.port>
-```
-
-### Ray backend
-
-For vLLM Ray backend, additionally appends:
-
-```
--e RAY_ADDRESS=<master_ib_ip>:6379
--e RAY_PORT=<port>
--e RAY_NODE_IP_ADDRESS=<node.ib_ip>
--e RAY_OVERRIDE_NODE_IP_ADDRESS=<node.ib_ip>
-```
-
-### Env vars passed to every container
-
-**Cluster-driven** (from `[nccl."<name>"]`):
-`NCCL_IB_DISABLE`, `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`, `NCCL_IB_GID_INDEX`, `GLOO_SOCKET_IFNAME`, `TP_SOCKET_IFNAME`
-
-**Network interface** (from `nccl.socket_ifname`):
-`MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include`
-
-**Defaults**:
-`OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN`
-
-**Engine IP** (from `node.ib_ip`):
-`VLLM_HOST_IP` (vLLM) or `SGLANG_WORKER_IP` (SGLang)
-
-**Head node only**:
-`-e VLLM_LOGGING_COLOR=1 -e TERM=xterm-256color` (colored logs)
-
-Recipe `[env]` entries are appended last and override any default.
-
-### Common to all runs
-
-NCCL env vars come from the cluster's `[nccl."<name>"]` section. Port mapping `-p <port>:<port>` on the head node only. `HF_HOME=/root/.cache/huggingface` with volume mount.
-
-Containers are labelled:
-- `sm121.recipe=<name>`
-- `sm121.rank=<rank>`
-
-`--cluster auto` picks the **smallest** cluster (fewest nodes) that matches the recipe's `required_nodes` and has an NCCL config. Prefix matching works: `--cluster 4` matches `4nodes-switch`.
-
-## How tmux launch works
-
-`tmux serve` builds sessions on each node with a 3-window layout:
-
-| Window | Content |
-|--------|---------|
-| 0: `model` | Main inference process (vLLM/SGLang) |
-| 1: `gpu` | `nvidia-smi -l 5` GPU monitor |
-| 2: `shell` | Debug shell |
-
-For **Ray backend**, nodes start sequentially: head node first (polled for API readiness up to 150s), then workers. For **MP backend**, all nodes start in parallel.
+## Power
 
-## Doctor mode
+- `off`: SSH shutdown → wait → HA `turn_off` per-node entities + cluster extras
+- `on`: HA `turn_on` (no SSH, requires HA config)
+- `switch --from <src> --to <tgt>`: disable nodes/extras unique to `from`, enable all in `to`
 
-`cargo run doctor --recipe <name>` checks each node for:
-- Docker availability
-- NVIDIA GPU (nvidia-smi)
-- NCCL libraries
-- Engine version inside the image (`vllm version` / `sglang version`)
-- Ray availability (if backend is "ray")
-- Container entrypoint
+## i18n
 
-With `--repair`, generates a Dockerfile that installs missing packages and rebuilds the image with a `.repaired` tag. The repaired image is auto-detected on subsequent `serve` runs.
+`fluent-i18n` crate, auto-detect via `set_locale(None)`. FTL in `locales/`.
 
-## Dependencies
+## Constraints
 
-- `clap` 4.6 + `derive` — derive-based CLI
-- `russh` 0.61 + `russh-keys` 0.50-beta — SSH client
-- `tokio` full — async runtime
-- `toml` 1 + `serde` 1 — config/recipe deserialization
-- `tracing` + `tracing-subscriber` — logging
-- `anyhow` — error handling
-- `futures` 0.3 — `join_all` for concurrent SSH
-- `sha2` 0.11 — SHA-256 for host key fingerprints
-- `base64` 0.22 — Base64 encoding for fingerprints
-- `dirs` 6 — OS config dir for default config fallback
-- `reqwest` 0.13 — HTTP client for Home Assistant REST API
-- `serde_json` 1 — JSON for HA requests
-- `fluent-i18n` — i18n via Mozilla Fluent (locales in `locales/`)
+- Files ≤500 lines (`make check-lines`)
+- `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`

+ 143 - 0
DEVELOPMENT.md

@@ -0,0 +1,143 @@
+# Development — sm121-launcher
+
+Rust CLI tool for launching vLLM/SGLang containers across a DGX Spark cluster. SSH-based provisioning over `russh`.
+
+## Project Layout
+
+```
+Cargo.toml                 # Binary crate, Edition 2024
+config.toml                # User config (gitignored)
+locales/                   # Fluent i18n files (en, ru, es, it, fr, de)
+recipes/                   # Recipe directories (<name>/recipe.toml)
+src/
+├── main.rs                # CLI entrypoint, command dispatch, i18n init
+├── cli.rs                 # clap derive CLI definitions
+├── config.rs              # Config/Recipe deserialization, cluster resolution
+├── ssh.rs                 # SSH client, key loading, host key verification
+├── docker/
+│   ├── mod.rs             # Docker command builder: serve, stop, health, logs
+│   └── tests.rs           # Command builder tests
+├── tmux/
+│   ├── mod.rs             # Tmux module root
+│   ├── serve.rs           # Tmux serve (window/panel layout)
+│   └── ops.rs             # Tmux stop/status/logs/attach
+├── power.rs               # Power on/off/switch flows
+├── home_assistant.rs      # HA REST API client
+├── host_keys.rs           # init-host-keys command
+├── exec.rs                # Exec subcommand + alias resolution
+├── doctor.rs              # Doctor mode + --repair
+├── pre_serve.rs           # PreServeCommand types
+├── before_serve.rs        # Pre-serve hook execution
+└── health.rs              # Tmux health-check helpers
+```
+
+## Quality Gates
+
+```bash
+make check            # All gates: fmt → clippy → test → line-count
+make fmt              # cargo fmt
+make check-fmt        # cargo fmt --check
+make check-clippy     # cargo clippy --all-targets -- -D warnings
+make check-test       # cargo test
+make check-lines      # File-size ≤500 lines per src/**/*.rs
+```
+
+## Architecture
+
+### Docker launch flow
+
+1. `serve` reads the recipe TOML
+2. Resolves cluster (exact/prefix match → smallest with NCCL + matching node count)
+3. Runs pre-serve hooks (cluster-level + recipe-level)
+4. Checks for repaired image (via doctor --repair)
+5. For each node: builds `docker run` command via `build_docker_command()`, runs via SSH
+6. Waits 10s, then shows tail logs
+
+`build_docker_command()` in `src/docker/mod.rs` is engine-aware:
+
+| Aspect | vLLM | SGLang |
+|---|---|---|
+| Host IP env var | `VLLM_HOST_IP` | `SGLANG_WORKER_IP` |
+| Model argument | bare name | `--model-path` |
+| TP flag | `--tensor-parallel-size` | `--tp` |
+| PP flag | `--pipeline-parallel-size` | — |
+| Backend flag | `--distributed-executor-backend` (ray only) | — |
+| Init addr | `--master-addr` + `--master-port` | `--dist-init-addr` |
+| Workers | `--headless` | — |
+
+Auto-generated flags: `--host 0.0.0.0`, `--port <recipe.port>`, `--nnodes`, `--node-rank`.
+
+### Cluster selection
+
+- **Exact match**: `--cluster 4nodes-switch`
+- **Prefix match**: `--cluster 4` → resolves to `4nodes-switch` (unique prefix, fails on ambiguous)
+- **Auto**: `--cluster auto` (or omitted) → picks the **smallest** cluster with NCCL config matching `required_nodes`
+
+### Env var precedence
+
+1. Cluster-driven (from `[nccl]`): `NCCL_IB_DISABLE`, `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`, etc.
+2. Network interface (from `nccl.socket_ifname`): `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include`
+3. Defaults: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN`
+4. Engine IP: `VLLM_HOST_IP` / `SGLANG_WORKER_IP`
+5. Head node: `VLLM_LOGGING_COLOR=1`, `TERM=xterm-256color`, `-p <port>:<port>`
+6. Ray backend: `RAY_ADDRESS`, `RAY_PORT`, `RAY_NODE_IP_ADDRESS`, `RAY_OVERRIDE_NODE_IP_ADDRESS`
+7. Recipe `[env]`: appended last, overrides everything above
+
+### Power management
+
+- `power off`: SSH `sudo shutdown now` → wait `shutdown_delay_seconds` → HA `turn_off` per-node entities + cluster extras
+- `power on`: HA `turn_on` per-node entities + cluster extras (no SSH)
+- `power switch`: disable nodes/extras unique to `from` → enable all in `to`
+
+### i18n
+
+Mozilla Fluent via `fluent-i18n` crate. Locale auto-detected from system. FTL files in `locales/{lang}/main.ftl`.
+
+### Module size limit
+
+Every `src/**/*.rs` file must stay ≤500 lines. Enforced by `make check-lines`.
+
+## Dev Workflow
+
+```bash
+# Build
+cargo build
+
+# Run locally with dry-run (no SSH execution)
+cargo run -- -d serve --recipe deepseek-v4-flash
+
+# Run tests
+cargo test
+
+# All gates (must pass before commit)
+make check
+
+# Add a new recipe
+mkdir -p recipes/my-model/
+# create recipes/my-model/recipe.toml
+cargo run -- -d serve --recipe my-model --cluster <name>
+
+# Add a new locale
+cp locales/en/main.ftl locales/xx/main.ftl
+# translate strings, commit
+```
+
+## Dependencies
+
+| Crate | Version | Purpose |
+|---|---|---|
+| `clap` | 4.6 | CLI parsing (derive) |
+| `russh` | 0.61 | SSH client |
+| `russh-keys` | 0.50-beta | SSH key loading |
+| `tokio` | 1 (full) | Async runtime |
+| `toml` | 1 | Config deserialization |
+| `serde` | 1 | Serialization |
+| `tracing` | 0.1 | Logging |
+| `anyhow` | 1.0 | Error handling |
+| `futures` | 0.3 | Concurrent SSH |
+| `sha2` | 0.11 | Host key fingerprints |
+| `base64` | 0.22 | Base64 encoding |
+| `reqwest` | 0.13 | HTTP client (HA) |
+| `serde_json` | 1 | JSON (HA) |
+| `dirs` | 6 | OS config dir |
+| `fluent-i18n` | — | Mozilla Fluent i18n |

+ 163 - 383
README.md

@@ -1,159 +1,141 @@
 # sm121-launcher
 
-Rust CLI tool for launching vLLM / SGLang containers across a DGX Spark cluster via SSH.
+Launch vLLM / SGLang inference clusters across DGX Spark machines via SSH.
 
-## Features
-
-- Launch distributed LLM inference clusters on 2–4 DGX Spark machines
-- **vLLM** and **SGLang** engines — auto-generates the right distributed flags for each
-- Docker (default) or **tmux** (alternative) runtime
-- SSH-based provisioning via `russh` — no agent required on target nodes
-- Cluster health checks, container log tailing, and clean teardown
-- **Doctor mode** — pre-flight checks and auto-repair of broken container images
-- Dry-run (`-d`) to preview every SSH command without execution
-
-## Quality Gates
+## Quick Install
 
 ```bash
-make check            # Run all quality gates (fmt, test, clippy, line count)
-make check-fmt        # cargo fmt --check
-make check-test       # cargo test
-make check-clippy     # cargo clippy with -D warnings
-make check-lines      # Ensure no Rust file exceeds 500 lines
+# From source
+cargo install --path .
+
+# Or download a pre-built binary (requires Rust toolchain to build)
+cargo build --release
+# Binary at target/release/sm121-launcher
 ```
 
 ## Quick Start
 
 ```bash
-# Build
-cargo build --release
-
-# Launch a recipe (auto-selects cluster from config.toml)
-cargo run serve --recipe glm51                # vLLM — GLM-5.1 NVFP8
-cargo run serve --recipe minimax-m3            # SGLang — Minimax-M3 NVFP4
-
-# Launch on a specific cluster
-cargo run serve --recipe glm51 --cluster 4nodes-switch
-
-# Dry-run (print SSH commands without executing)
-cargo run -- -d serve --recipe minimax-m3
-
-# View cluster logs
-cargo run logs --recipe glm51                  # Last 100 lines per node
-cargo run logs --recipe minimax-m3 --follow    # Stream head node logs
+# Create config.toml (see Configuration below)
+# Launch a recipe
+sm121-launcher serve --recipe deepseek-v4-flash
+
+# Follow logs
+sm121-launcher logs --recipe deepseek-v4-flash --follow
+
+# Stop containers
+sm121-launcher stop --recipe deepseek-v4-flash
+```
+
+## Commands
+
+| Command | Description |
+|---------|-------------|
+| `serve --recipe <name>` | Launch recipe on cluster |
+| `logs --recipe <name> [--follow]` | Show container logs |
+| `stop --recipe <name>` | Stop and remove containers |
+| `health [--cluster <name>]` | Check cluster health |
+| `doctor --recipe <name> [--repair]` | Check/repair node prerequisites |
+| `exec --cluster <name> -- <command>` | Run command on cluster nodes |
+| `exec --cluster <name> --alias <name>` | Run predefined alias |
+| `power off --cluster <name> [--node <name>]` | Shutdown + HA turn_off |
+| `power on --cluster <name> [--node <name>]` | HA turn_on (wake nodes) |
+| `power switch --from <src> --to <tgt>` | Switch clusters |
+| `tmux serve --recipe <name>` | Launch via tmux (no Docker) |
+| `tmux status --recipe <name>` | Check tmux session health |
+| `tmux logs --recipe <name>` | Capture tmux pane output |
+| `init-host-keys` | Auto-write SSH host keys |
+| `-d` / `--dry-run` | Preview commands without executing |
+
+### Cluster selection
 
-# Stop all containers for a recipe
-cargo run stop --recipe glm51
-
-# Health check
-cargo run health                               # All clusters
-cargo run health --cluster 2nodes-dac          # Single cluster
-cargo run health --fingerprint                 # Discover host keys for config.toml
-
-# Doctor mode — check prerequisites
-cargo run doctor --recipe minimax-m3           # Check node prerequisites
-cargo run doctor --recipe minimax-m3 --repair  # Auto-fix (build repaired image)
-
-# Tmux runtime (no Docker)
-cargo run tmux serve --recipe glm51
-cargo run tmux status --recipe glm51
-cargo run tmux attach --recipe glm51
+```bash
+sm121-launcher serve --recipe deepseek-v4-flash              # auto (smallest match)
+sm121-launcher serve --recipe glm51 --cluster 4nodes-switch  # exact name
+sm121-launcher serve --recipe glm51 --cluster 4             # prefix match
+sm121-launcher serve --recipe glm51 --cluster 2             # picks 2nodes-dac
 ```
 
-## Available Recipes
+Prefix must be unambiguous — error if multiple clusters match.
 
-### [`deepseek-v4-flash`](recipes/deepseek-v4-flash/recipe.toml) — vLLM (DeepSeek V4 Flash MXFP8)
+## Recipes
 
-```toml
-engine = "vllm"
-docker_image = "aidendle94/sparkrun-vllm-ds4-gb10:production-v2"
-model = "deepseek-ai/DeepSeek-V4-Flash"
-required_nodes = 2
-tensor_parallel_size = 2
-pipeline_parallel_size = 1
-distributed_backend = "mp"
-```
+Recipes are TOML files in `recipes/<name>/recipe.toml`.
+The launcher looks for them in the current directory first,
+then in `~/.config/sm121-launcher/recipes/<name>/recipe.toml`.
 
-DeepSeek V4 Flash with 1M context, FP8 KV cache, MTP speculative decoding, and DeepSeek V4 reasoning/tool parsers. 2 nodes, TP=2 across nodes. Uses `sparkrun` DGX Spark image.
+### Available
 
-```bash
-cargo run serve --recipe deepseek-v4-flash          # auto-selects 2nodes-dac
-cargo run logs --recipe deepseek-v4-flash --follow
-```
+| Recipe | Engine | Nodes | Model |
+|--------|--------|-------|-------|
+| `deepseek-v4-flash` | vLLM | 2 | DeepSeek V4 Flash (MXFP8, 1M ctx, MTP) |
+| `minimax-m3-mxfp8` | vLLM | 4 | MiniMax M3 (MXFP8, 428B, MSA sparse attn) |
+| `glm51` | vLLM | 4 | GLM-5.1 NVFP8 |
+| `minimax-m3` | SGLang | 4 | MiniMax-M3-v0 NVFP4 |
+| `test-sleep` | vLLM | any | Heartbeat test container |
 
-### [`minimax-m3-mxfp8`](recipes/minimax-m3-mxfp8/recipe.toml) — vLLM (MiniMax M3 MXFP8)
+### Creating a recipe
 
 ```toml
-engine = "vllm"
-docker_image = "aidendle94/sparkrun-vllm-ds4-gb10:production-v2"
-model = "MiniMaxAI/MiniMax-M3-MXFP8"
-required_nodes = 4
-tensor_parallel_size = 4
-distributed_backend = "mp"
-```
+engine = "vllm"                        # or "sglang" (default: "vllm")
+name = "My Model"
+docker_image = "my-image:tag"
+model = "org/model-name"
+required_nodes = 2                     # optional
+tensor_parallel_size = 2               # TP across nodes
+pipeline_parallel_size = 1             # vLLM only, 0 = auto (equals nnodes)
+distributed_backend = "mp"             # "mp" (default) or "ray"
+shm_size = "64gb"                      # optional, only added when set
 
-MiniMax M3 vision-language MoE (428B params, 23B active) with MXFP8 quantization, MSA sparse attention (`--block-size 128` mandatory), expert parallelism, and `minimax_m3` parsers. Text-only mode skips the vision encoder. 4 nodes, TP=4.
+entrypoint = ["vllm", "serve"]         # optional, appended after image
 
-```bash
-cargo run serve --recipe minimax-m3-mxfp8
-```
+args = [
+    "--kv-cache-dtype", "fp8",
+    "--max-model-len", "32768",
+]
 
-### [`glm51`](recipes/glm51/recipe.toml) — vLLM (GLM-5.1 NVFP8)
+before_serve = [                       # recipe-level pre-serve hooks
+    { command = "echo ready", on_failure = "abort" },
+]
 
-```toml
-engine = "vllm"
-docker_image = "vllm/vllm-openai:glm51-cu130"
-model = "lukealonso/GLM-5.1-NVFP4"
-port = 30000
-required_nodes = 4
-distributed_backend = "mp"
+[env]
+VLLM_LOGGING_COLOR = "1"              # override any launcher default
 ```
 
-GLM-5.1 with vLLM across 4 nodes. Pipeline parallelism (PP=4, TP=1). MTP speculative decoding, TRITON_MLA attention, prefix caching, chunked prefill.
+## Configuration
 
-```bash
-cargo run serve --recipe glm51
-cargo run logs --recipe glm51 --follow
-```
+The launcher reads `config.toml` from the current directory,
+falling back to `~/.config/sm121-launcher/config.toml`.
 
-### [`minimax-m3`](recipes/minimax-m3/recipe.toml) — SGLang (Minimax-M3-v0 NVFP4)
+### Minimal
 
 ```toml
-engine = "sglang"
-docker_image = "scitrera/dgx-spark-sglang-mm:v0"
-model = "sparkarena/Minimax-M3-v0-NVFP4"
-port = 30000
-required_nodes = 4
-tensor_parallel_size = 4
-distributed_backend = "mp"
-entrypoint = ["sglang", "serve"]
-```
-
-Minimax-M3 with SGLang, 4 nodes, TP=4. NVFP4 quantization, `cutlass` fp4 gemm, `triton` MoE runner, `instanttensor` load format.
+ssh_user = "kilork"
+ssh_key_path = "/path/to/ssh/key"
 
-```bash
-cargo run serve --recipe minimax-m3
-cargo run logs --recipe minimax-m3
+[[clusters]]
+name = "4nodes-switch"
+master_ib_ip = "192.168.200.15"
+nodes = [
+    { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0 },
+    { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1 },
+    { name = "dgx3", ssh_ip = "10.0.0.3", ib_ip = "192.168.200.17", rank = 2 },
+    { name = "dgx4", ssh_ip = "10.0.0.4", ib_ip = "192.168.200.19", rank = 3 },
+]
 ```
 
-## Configuration
-
-### config.toml
-
-Create a `config.toml` in the project root (not tracked in git):
+### With Home Assistant (power management)
 
 ```toml
-ssh_user = "kilork"
-ssh_key_path = "/path/to/ssh/key"
-
 [[clusters]]
 name = "4nodes-switch"
 master_ib_ip = "192.168.200.15"
+extra_entities = ["switch.network_switch"]   # cluster-level switches
 nodes = [
-    { name = "dgx1", ssh_ip = "192.168.88.14", ib_ip = "192.168.200.15", rank = 0 },
-    { name = "dgx2", ssh_ip = "192.168.88.10", ib_ip = "192.168.200.13", rank = 1 },
-    { name = "dgx3", ssh_ip = "192.168.88.50", ib_ip = "192.168.200.17", rank = 2 },
-    { name = "dgx4", ssh_ip = "192.168.88.51", ib_ip = "192.168.200.19", rank = 3 },
+    { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0, entity = "switch.dgx1" },
+    { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1, entity = "switch.dgx2" },
+    { name = "dgx3", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.17", rank = 2, entity = "switch.dgx3" },
+    { name = "dgx4", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.19", rank = 3, entity = "switch.dgx4" },
 ]
 
 [nccl."4nodes-switch"]
@@ -161,305 +143,103 @@ ib_hca = "rocep1s0f0,roceP2p1s0f0"
 socket_ifname = "enp1s0f0np0"
 ib_gid_index = 3
 ib_disable = 0
-```
-
-| Field | Purpose |
-|---|---|
-| `ssh_user` / `ssh_key_path` | SSH credential for all nodes |
-| `clusters[].master_ib_ip` | InfiniBand IP of the head node (rank 0) |
-| `nodes[].ssh_ip` | SSH reachable IP (usually Ethernet) |
-| `nodes[].ib_ip` | InfiniBand / RoCE IP for NCCL communication |
-| `nodes[].rank` | Node rank (0 = head, 1..N = workers) |
-| `nodes[].host_key` | Optional SHA-256 fingerprint — run `cargo run health --fingerprint` to discover |
-| `nccl."<name>".*` | NCCL environment variables for the cluster's IB fabric |
-
-Multiple clusters can be defined; `--cluster auto` picks the first one with an NCCL section.
-
-### Creating a recipe
-
-Recipes live in `recipes/<name>/recipe.toml` (directory name intentionally misspelled).
-
-```toml
-engine = "vllm"                        # or "sglang" (default: "vllm")
-name = "Human Readable Name"
-docker_image = "my-image:tag"
-model = "org/model-name"
-port = 30000
-required_nodes = 4                     # optional, omitted = no check
-pipeline_parallel_size = 0             # 0 = auto (equals nnodes), vLLM only
-tensor_parallel_size = 1               # per-node TP for vLLM, total TP for SGLang
-distributed_backend = "mp"             # "mp" (multiprocessing) or "ray"
-cluster = "4nodes-switch"              # optional: pin to a specific cluster
 
-entrypoint = ["sglang", "serve"]       # optional, appended after image name
-
-args = [
-    "--kv-cache-dtype", "fp8",
-    "--max-model-len", "32768",
-]
-
-[env]
-SOME_VAR = "value"
-```
-
-#### Engine-specific behavior
-
-| Aspect | vLLM | SGLang |
-|---|---|---|
-| Host IP env var | `VLLM_HOST_IP` | `SGLANG_WORKER_IP` |
-| Model argument | bare model name | `--model-path <model>` |
-| TP flag | `--tensor-parallel-size 1` | `--tp <tensor_parallel_size>` |
-| PP flag | `--pipeline-parallel-size <pp_size>` | _(not used)_ |
-| Backend flag | `--distributed-executor-backend mp|ray` | _(not used)_ |
-| Common flags | `--nnodes`, `--node-rank`, `--master-addr` | same |
-| Port flag | _(recipe args)_ | `--port <port>` (auto-generated) |
-| Entrypoint | `python -m vllm.entrypoints.openai.api_server` | recipe-defined (e.g. `sglang serve`) |
-
-## Launch Behavior
-
-### Docker runtime
-
-For each node, the tool builds a `docker run` command via SSH:
-
-```
-docker run -d --gpus all --ipc=host --privileged --network=host \
-  -e NCCL_IB_DISABLE=<...> -e NCCL_IB_HCA=<...> ... \
-  -e SGLANG_WORKER_IP=<node.ib_ip> \       # or VLLM_HOST_IP
-  -e <recipe.env>... \
-  [-p <port>:<port>] \                       # head node only
-  --name sm121_<recipe>_<rank> \
-  --label sm121.recipe=<recipe> \
-  --label sm121.rank=<rank> \
-  <docker_image> \
-  <entrypoint>... \                          # e.g. sglang serve
-  [--model-path] <model> \                   # engine-specific
-  <recipe.args>... \                         # from recipe
-  --tp <tensor_parallel_size> \              # engine-specific flags
-  --nnodes <nnodes> \
-  --node-rank <rank> \
-  --master-addr <master_ib_ip>
-  [--port <port>]                            # SGLang only
-```
-
-NCCL env vars come from the cluster's `[nccl."<name>"]` section. All containers use `--network=host` for direct InfiniBand access.
-
-### Tmux runtime
-
-`cargo run tmux serve --recipe <name>` creates a 3-window tmux session on each node:
-
-| Window | Content |
-|--------|---------|
-| 0: `model` | Main inference process (vLLM/SGLang) |
-| 1: `gpu` | `nvidia-smi -l 5` GPU monitor |
-| 2: `shell` | Debug shell |
-
-For Ray backend, nodes start sequentially (head first with API readiness poll up to 150s). For MP backend, all nodes start in parallel.
-
-## Environment Variables
-
-The launcher passes the following env vars to every container.  Recipe-level
-`[env]` entries are appended after the defaults and can override them.
-
-### Cluster-driven (from `[nccl."<cluster>"]`)
-
-| Env var | Source | Purpose |
-|---|---|---|
-| `NCCL_IB_DISABLE` | `nccl.<cluster>.ib_disable` | Disable NCCL's InfiniBand path (0 = use IB, 1 = disable) |
-| `NCCL_IB_HCA` | `nccl.<cluster>.ib_hca` | RDMA devices for NCCL (comma-separated, e.g. `rocep1s0f0,roceP2p1s0f0`) |
-| `NCCL_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | Network interface NCCL uses for socket-based control (e.g. `enp1s0f0np0`) |
-| `NCCL_IB_GID_INDEX` | `nccl.<cluster>.ib_gid_index` | RoCE GID index for InfiniBand routing |
-| `GLOO_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | Network interface for Gloo (vLLM distributed + PyTorch CPU collectives) |
-| `TP_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | vLLM tensor-parallel coordinator interface |
-| `MN_IF_NAME` | `nccl.<cluster>.socket_ifname` | Multi-node interface name |
-| `UCX_NET_DEVICES` | `nccl.<cluster>.socket_ifname` | UCX communication devices |
-| `OMPI_MCA_btl_tcp_if_include` | `nccl.<cluster>.socket_ifname` | OpenMPI TCP interface |
-| `VLLM_LOGGING_COLOR` | `1` | Force colored vLLM logs (head node only) |
-
-### Hard-coded defaults
-
-| Env var | Value | Purpose |
-|---|---|---|
-| `NCCL_IGNORE_CPU_AFFINITY` | `1` | Don't pin NCCL threads to a specific CPU set (needed on DGX Spark) |
-| `NCCL_NET` | `IB` | Force NCCL to use the InfiniBand plugin (`IB` = `libnccl-net.so`) |
-| `NCCL_DEBUG` | `WARN` | NCCL log level (override with `[env]` for `INFO`/`TRACE` while debugging) |
-| `OMP_NUM_THREADS` | `4` | OpenMP thread limit per process |
-| `HF_HOME` | `/root/.cache/huggingface` | HuggingFace cache root (volume-mounted from `ssh_user`'s host cache) |
-
-### Engine-specific host IP
-
-| Env var | Engines | Value | Purpose |
-|---|---|---|---|
-| `VLLM_HOST_IP` | vLLM only | `node.ib_ip` | vLLM's own host-IP for distributed init |
-| `SGLANG_WORKER_IP` | SGLang only | `node.ib_ip` | SGLang's worker-IP for distributed init |
-
-### Ray-only (when `distributed_backend = "ray"`)
-
-| Env var | Value | Purpose |
-|---|---|---|
-| `RAY_ADDRESS` | `<master_ib_ip>:6379` | Ray head address (workers connect here) |
-| `RAY_PORT` | `recipe.port` | Ray gRPC port |
-
-### Recipe-level overrides
-
-Anything in `[env]` is appended after all of the above, so a recipe can
-override any default (e.g. `NCCL_CUMEM_ENABLE = "1"` to re-enable
-CUDA-managed memory).
-
-## Power Management (Home Assistant)
-
-Optional `[home_assistant]` section lets the launcher toggle smart
-plugs (or any switch domain) so you can shut down and wake nodes
-remotely:
-
-```toml
 [home_assistant]
 url = "http://homeassistant.local:8123"
 token = "long-lived-access-token"
-# Seconds to wait between `sudo shutdown now` and HA `turn_off` so
-# the OS has time to actually shut down.  Default: 30.
 shutdown_delay_seconds = 30
-```
-
-Each node can optionally carry an `entity` field for the plug that
-controls it:
-
-```toml
-[[clusters]]
-name = "4nodes-switch"
-master_ib_ip = "192.168.200.15"
 
-# Optional: extra switches to toggle alongside the nodes.  Useful for
-# a network switch that powers the cluster, or a PDU that backs the
-# rack.  Toggled in addition to any per-node entities.
-extra_entities = ["switch.network_switch_4nodes"]
+[exec_aliases]
+gpu-lock = "sudo nvidia-smi -lgc 300,2300"
+shutdown = "sudo shutdown now"
 
-nodes = [
-    { name = "dgx1", entity = "switch.dgx1", ssh_ip = "10.0.0.1", ib_ip = "1.2.3.4", rank = 0 },
-    # ...
+[before_serve."4nodes-switch"]
+commands = [
+    "sudo nvidia-smi -lgc 300,2300",
+    { command = "sudo nvidia-smi -pm 1", on_failure = "warn" },
 ]
 ```
 
-Without an `entity` the node is skipped (with a warning) — the SSH
-shutdown still happens, but Home Assistant is not consulted.
-Cluster-level `extra_entities` are always toggled (no opt-in).
+### NCCL config
 
-### `power off` flow
+Required for each cluster. Controls InfiniBand/RoCE communication:
 
-```
-1. ssh <node>: sudo shutdown now
-2. wait shutdown_delay_seconds (default 30s)
-3. HA POST /api/services/switch/turn_off
-   (entity: switch.dgx1)
-```
+| Field | Example | Purpose |
+|-------|---------|---------|
+| `ib_hca` | `rocep1s0f0,roceP2p1s0f0` | RDMA devices |
+| `socket_ifname` | `enp1s0f0np0` | Network interface |
+| `ib_gid_index` | `3` | RoCE GID index |
+| `ib_disable` | `0` | Keep IB enabled |
+
+## Power Management
+
+Requires `[home_assistant]` in config.toml.
 
 ```bash
+# Graceful shutdown + wait + turn off smart plugs
 sm121-launcher power off --cluster 4nodes-switch
-sm121-launcher power off --cluster 4nodes-switch --node dgx1
-sm121-launcher -d power off --cluster 4nodes-switch   # dry-run
-```
 
-### `power on` flow
-
-`power on` does **not** SSH — the smart plug wakes the machine:
+# Wake nodes via smart plugs
+sm121-launcher power on --cluster 4nodes-switch
 
-```
-1. HA POST /api/services/switch/turn_on
-   (entity: switch.dgx1)
-```
+# Switch clusters (disable 4 → enable 2)
+sm121-launcher power switch --from 4 --to 2
 
-Requires `[home_assistant]` to be configured (no other way to wake
-a node).
+# Single node
+sm121-launcher power off --cluster 4nodes-switch --node dgx1
 
-```bash
-sm121-launcher power on --cluster 4nodes-switch
-sm121-launcher power on --cluster 4nodes-switch --node dgx1
+# Dry-run
+sm121-launcher -d power off --cluster 4nodes-switch
 ```
 
-### `power switch` flow
+**Power-off flow**: SSH `sudo shutdown now` → wait `shutdown_delay_seconds` → HA `turn_off` entity → HA `turn_off` cluster extras.
 
-Switch between clusters: disable nodes/entities unique to `from`,
-enable everything in `to`.
+**Power-on flow**: HA `turn_on` (no SSH, assumes smart plug wakes the machine).
 
-```
-1. SSH sudo shutdown now on nodes in from but not to
-2. Wait shutdown_delay_seconds
-3. HA turn_off those nodes + extra_entities unique to from
-4. HA turn_on all nodes + extra_entities in to
-```
+**Switch flow**: disable nodes/extras only in source → enable all in target.
 
-```bash
-sm121-launcher power switch --from 4 --to 2   # prefix matching works
-```
+## Environment Variables
 
-## Doctor Mode
+The launcher sets these automatically. Recipe `[env]` overrides any.
 
-```bash
-cargo run doctor --recipe minimax-m3           # Check node prerequisites
-cargo run doctor --recipe minimax-m3 --repair  # Auto-fix issues
-```
+### Auto-set (always)
 
-Checks each node for:
-- Docker availability
-- NVIDIA GPU (`nvidia-smi`)
-- NCCL libraries
-- Engine version inside the image (`vllm version` / `sglang version`)
-- Ray availability (if backend is "ray")
+| Env var | Source | Description |
+|---------|--------|-------------|
+| `NCCL_IB_DISABLE` | `nccl.<name>.ib_disable` | InfiniBand on/off |
+| `NCCL_IB_HCA` | `nccl.<name>.ib_hca` | RDMA devices |
+| `NCCL_SOCKET_IFNAME` | `nccl.<name>.socket_ifname` | NCCL control interface |
+| `GLOO_SOCKET_IFNAME` | same | Gloo (PyTorch) interface |
+| `TP_SOCKET_IFNAME` | same | vLLM TP coordinator |
+| `MN_IF_NAME` | same | Multi-node interface |
+| `UCX_NET_DEVICES` | same | UCX devices |
+| `OMPI_MCA_btl_tcp_if_include` | same | OpenMPI TCP interface |
+| `NCCL_IB_GID_INDEX` | `nccl.<name>.ib_gid_index` | RoCE GID |
+| `VLLM_HOST_IP` / `SGLANG_WORKER_IP` | `node.ib_ip` | Engine host IP |
+| `HF_HOME` | `/root/.cache/huggingface` | HF cache (volume-mounted) |
+| `OMP_NUM_THREADS` | `4` | OpenMP threads |
+| `NCCL_IGNORE_CPU_AFFINITY` | `1` | Don't pin NCCL threads |
+| `NCCL_NET` | `IB` | NCCL IB plugin |
+| `NCCL_DEBUG` | `WARN` | NCCL log level |
+| `VLLM_LOGGING_COLOR` | `1` | Colored logs (head only) |
+| `TERM` | `xterm-256color` | Terminal type (head only) |
 
-With `--repair`, generates a Dockerfile that installs missing packages and rebuilds the image with a `.repaired` tag. The repaired image is auto-detected on subsequent `serve` runs.
+### Ray backend (when `distributed_backend = "ray"`)
 
-## Internationalization (i18n)
+`RAY_ADDRESS`, `RAY_PORT`, `RAY_NODE_IP_ADDRESS`, `RAY_OVERRIDE_NODE_IP_ADDRESS`
 
-Log messages are localized via Mozilla Fluent (`.ftl` files in `locales/`).
-Auto-detects system locale. Available languages:
+## Internationalization
 
-| Locale | File | Status |
-|--------|------|--------|
-| English | `locales/en/main.ftl` | Fallback |
-| Russian | `locales/ru/main.ftl` | Full |
-| Spanish | `locales/es/main.ftl` | Full |
-| Italian | `locales/it/main.ftl` | Full |
-| French | `locales/fr/main.ftl` | Full |
-| German | `locales/de/main.ftl` | Full |
+Messages are localized. Auto-detects system locale.
 
 ```bash
-# Override locale via environment
-LANG=ru_RU.UTF-8 cargo run serve --recipe deepseek-v4-flash
+# Override locale
+LANG=ru_RU.UTF-8 sm121-launcher serve --recipe deepseek-v4-flash
 ```
 
-## CLI Reference
-
-```text
-Usage: sm121-launcher [OPTIONS] <COMMAND>
-
-Commands:
-  serve    Launch a recipe on a cluster
-  logs     Show container logs
-  stop     Stop and remove containers for a recipe
-  health   Check cluster health (or fingerprint host keys)
-  doctor   Check and repair node prerequisites
-  tmux     Manage tmux sessions (alternative to Docker)
-  power    Power on/off nodes via Home Assistant
-  exec     Run an arbitrary shell command on a cluster or single node
-
-Global flags:
-  -c, --config <FILE>    Config file path (default: config.toml)
-  -d, --dry-run          Print commands without executing
-  -h, --help             Print help
-  -V, --version          Print version
-```
+Available: English (fallback), Russian, Spanish, Italian, French, German.
+
+## Project Links
 
-## Dependencies
-
-| Crate | Purpose |
-|---|---|
-| `clap` 4.6 + derive | CLI argument parsing |
-| `russh` 0.61 + `russh-keys` 0.50-beta | SSH client |
-| `tokio` full | Async runtime |
-| `toml` 1 + `serde` 1 | Config / recipe deserialization |
-| `tracing` + `tracing-subscriber` | Structured logging |
-| `anyhow` | Error handling |
-| `futures` 0.3 | Concurrent SSH (`join_all`) |
-| `sha2` 0.11 + `base64` 0.22 | Host key fingerprinting |
-| `reqwest` 0.13 | HTTP client for Home Assistant |
-| `dirs` 6 | OS config dir discovery |
-| `fluent-i18n` | Mozilla Fluent i18n (locales: en, ru, es, it, fr, de) |
+- **Development docs**: [`DEVELOPMENT.md`](DEVELOPMENT.md)
+- **Agent reference**: [`AGENTS.md`](AGENTS.md)