|
|
@@ -1,159 +1,141 @@
|
|
|
# sm121-launcher
|
|
|
|
|
|
-Rust CLI tool for launching vLLM / SGLang containers across a DGX Spark cluster via SSH.
|
|
|
+Launch vLLM / SGLang inference clusters across DGX Spark machines via SSH.
|
|
|
|
|
|
-## Features
|
|
|
-
|
|
|
-- Launch distributed LLM inference clusters on 2–4 DGX Spark machines
|
|
|
-- **vLLM** and **SGLang** engines — auto-generates the right distributed flags for each
|
|
|
-- Docker (default) or **tmux** (alternative) runtime
|
|
|
-- SSH-based provisioning via `russh` — no agent required on target nodes
|
|
|
-- Cluster health checks, container log tailing, and clean teardown
|
|
|
-- **Doctor mode** — pre-flight checks and auto-repair of broken container images
|
|
|
-- Dry-run (`-d`) to preview every SSH command without execution
|
|
|
-
|
|
|
-## Quality Gates
|
|
|
+## Quick Install
|
|
|
|
|
|
```bash
|
|
|
-make check # Run all quality gates (fmt, test, clippy, line count)
|
|
|
-make check-fmt # cargo fmt --check
|
|
|
-make check-test # cargo test
|
|
|
-make check-clippy # cargo clippy with -D warnings
|
|
|
-make check-lines # Ensure no Rust file exceeds 500 lines
|
|
|
+# From source
|
|
|
+cargo install --path .
|
|
|
+
|
|
|
+# Or download a pre-built binary (requires Rust toolchain to build)
|
|
|
+cargo build --release
|
|
|
+# Binary at target/release/sm121-launcher
|
|
|
```
|
|
|
|
|
|
## Quick Start
|
|
|
|
|
|
```bash
|
|
|
-# Build
|
|
|
-cargo build --release
|
|
|
-
|
|
|
-# Launch a recipe (auto-selects cluster from config.toml)
|
|
|
-cargo run serve --recipe glm51 # vLLM — GLM-5.1 NVFP8
|
|
|
-cargo run serve --recipe minimax-m3 # SGLang — Minimax-M3 NVFP4
|
|
|
-
|
|
|
-# Launch on a specific cluster
|
|
|
-cargo run serve --recipe glm51 --cluster 4nodes-switch
|
|
|
-
|
|
|
-# Dry-run (print SSH commands without executing)
|
|
|
-cargo run -- -d serve --recipe minimax-m3
|
|
|
-
|
|
|
-# View cluster logs
|
|
|
-cargo run logs --recipe glm51 # Last 100 lines per node
|
|
|
-cargo run logs --recipe minimax-m3 --follow # Stream head node logs
|
|
|
+# Create config.toml (see Configuration below)
|
|
|
+# Launch a recipe
|
|
|
+sm121-launcher serve --recipe deepseek-v4-flash
|
|
|
+
|
|
|
+# Follow logs
|
|
|
+sm121-launcher logs --recipe deepseek-v4-flash --follow
|
|
|
+
|
|
|
+# Stop containers
|
|
|
+sm121-launcher stop --recipe deepseek-v4-flash
|
|
|
+```
|
|
|
+
|
|
|
+## Commands
|
|
|
+
|
|
|
+| Command | Description |
|
|
|
+|---------|-------------|
|
|
|
+| `serve --recipe <name>` | Launch recipe on cluster |
|
|
|
+| `logs --recipe <name> [--follow]` | Show container logs |
|
|
|
+| `stop --recipe <name>` | Stop and remove containers |
|
|
|
+| `health [--cluster <name>]` | Check cluster health |
|
|
|
+| `doctor --recipe <name> [--repair]` | Check/repair node prerequisites |
|
|
|
+| `exec --cluster <name> -- <command>` | Run command on cluster nodes |
|
|
|
+| `exec --cluster <name> --alias <name>` | Run predefined alias |
|
|
|
+| `power off --cluster <name> [--node <name>]` | Shutdown + HA turn_off |
|
|
|
+| `power on --cluster <name> [--node <name>]` | HA turn_on (wake nodes) |
|
|
|
+| `power switch --from <src> --to <tgt>` | Switch clusters |
|
|
|
+| `tmux serve --recipe <name>` | Launch via tmux (no Docker) |
|
|
|
+| `tmux status --recipe <name>` | Check tmux session health |
|
|
|
+| `tmux logs --recipe <name>` | Capture tmux pane output |
|
|
|
+| `init-host-keys` | Auto-write SSH host keys |
|
|
|
+| `-d` / `--dry-run` | Preview commands without executing |
|
|
|
+
|
|
|
+### Cluster selection
|
|
|
|
|
|
-# Stop all containers for a recipe
|
|
|
-cargo run stop --recipe glm51
|
|
|
-
|
|
|
-# Health check
|
|
|
-cargo run health # All clusters
|
|
|
-cargo run health --cluster 2nodes-dac # Single cluster
|
|
|
-cargo run health --fingerprint # Discover host keys for config.toml
|
|
|
-
|
|
|
-# Doctor mode — check prerequisites
|
|
|
-cargo run doctor --recipe minimax-m3 # Check node prerequisites
|
|
|
-cargo run doctor --recipe minimax-m3 --repair # Auto-fix (build repaired image)
|
|
|
-
|
|
|
-# Tmux runtime (no Docker)
|
|
|
-cargo run tmux serve --recipe glm51
|
|
|
-cargo run tmux status --recipe glm51
|
|
|
-cargo run tmux attach --recipe glm51
|
|
|
+```bash
|
|
|
+sm121-launcher serve --recipe deepseek-v4-flash # auto (smallest match)
|
|
|
+sm121-launcher serve --recipe glm51 --cluster 4nodes-switch # exact name
|
|
|
+sm121-launcher serve --recipe glm51 --cluster 4 # prefix match
|
|
|
+sm121-launcher serve --recipe glm51 --cluster 2 # picks 2nodes-dac
|
|
|
```
|
|
|
|
|
|
-## Available Recipes
|
|
|
+Prefix must be unambiguous — error if multiple clusters match.
|
|
|
|
|
|
-### [`deepseek-v4-flash`](recipes/deepseek-v4-flash/recipe.toml) — vLLM (DeepSeek V4 Flash MXFP8)
|
|
|
+## Recipes
|
|
|
|
|
|
-```toml
|
|
|
-engine = "vllm"
|
|
|
-docker_image = "aidendle94/sparkrun-vllm-ds4-gb10:production-v2"
|
|
|
-model = "deepseek-ai/DeepSeek-V4-Flash"
|
|
|
-required_nodes = 2
|
|
|
-tensor_parallel_size = 2
|
|
|
-pipeline_parallel_size = 1
|
|
|
-distributed_backend = "mp"
|
|
|
-```
|
|
|
+Recipes are TOML files in `recipes/<name>/recipe.toml`.
|
|
|
+The launcher looks for them in the current directory first,
|
|
|
+then in `~/.config/sm121-launcher/recipes/<name>/recipe.toml`.
|
|
|
|
|
|
-DeepSeek V4 Flash with 1M context, FP8 KV cache, MTP speculative decoding, and DeepSeek V4 reasoning/tool parsers. 2 nodes, TP=2 across nodes. Uses `sparkrun` DGX Spark image.
|
|
|
+### Available
|
|
|
|
|
|
-```bash
|
|
|
-cargo run serve --recipe deepseek-v4-flash # auto-selects 2nodes-dac
|
|
|
-cargo run logs --recipe deepseek-v4-flash --follow
|
|
|
-```
|
|
|
+| Recipe | Engine | Nodes | Model |
|
|
|
+|--------|--------|-------|-------|
|
|
|
+| `deepseek-v4-flash` | vLLM | 2 | DeepSeek V4 Flash (MXFP8, 1M ctx, MTP) |
|
|
|
+| `minimax-m3-mxfp8` | vLLM | 4 | MiniMax M3 (MXFP8, 428B, MSA sparse attn) |
|
|
|
+| `glm51` | vLLM | 4 | GLM-5.1 NVFP8 |
|
|
|
+| `minimax-m3` | SGLang | 4 | MiniMax-M3-v0 NVFP4 |
|
|
|
+| `test-sleep` | vLLM | any | Heartbeat test container |
|
|
|
|
|
|
-### [`minimax-m3-mxfp8`](recipes/minimax-m3-mxfp8/recipe.toml) — vLLM (MiniMax M3 MXFP8)
|
|
|
+### Creating a recipe
|
|
|
|
|
|
```toml
|
|
|
-engine = "vllm"
|
|
|
-docker_image = "aidendle94/sparkrun-vllm-ds4-gb10:production-v2"
|
|
|
-model = "MiniMaxAI/MiniMax-M3-MXFP8"
|
|
|
-required_nodes = 4
|
|
|
-tensor_parallel_size = 4
|
|
|
-distributed_backend = "mp"
|
|
|
-```
|
|
|
+engine = "vllm" # or "sglang" (default: "vllm")
|
|
|
+name = "My Model"
|
|
|
+docker_image = "my-image:tag"
|
|
|
+model = "org/model-name"
|
|
|
+required_nodes = 2 # optional
|
|
|
+tensor_parallel_size = 2 # TP across nodes
|
|
|
+pipeline_parallel_size = 1 # vLLM only, 0 = auto (equals nnodes)
|
|
|
+distributed_backend = "mp" # "mp" (default) or "ray"
|
|
|
+shm_size = "64gb" # optional, only added when set
|
|
|
|
|
|
-MiniMax M3 vision-language MoE (428B params, 23B active) with MXFP8 quantization, MSA sparse attention (`--block-size 128` mandatory), expert parallelism, and `minimax_m3` parsers. Text-only mode skips the vision encoder. 4 nodes, TP=4.
|
|
|
+entrypoint = ["vllm", "serve"] # optional, appended after image
|
|
|
|
|
|
-```bash
|
|
|
-cargo run serve --recipe minimax-m3-mxfp8
|
|
|
-```
|
|
|
+args = [
|
|
|
+ "--kv-cache-dtype", "fp8",
|
|
|
+ "--max-model-len", "32768",
|
|
|
+]
|
|
|
|
|
|
-### [`glm51`](recipes/glm51/recipe.toml) — vLLM (GLM-5.1 NVFP8)
|
|
|
+before_serve = [ # recipe-level pre-serve hooks
|
|
|
+ { command = "echo ready", on_failure = "abort" },
|
|
|
+]
|
|
|
|
|
|
-```toml
|
|
|
-engine = "vllm"
|
|
|
-docker_image = "vllm/vllm-openai:glm51-cu130"
|
|
|
-model = "lukealonso/GLM-5.1-NVFP4"
|
|
|
-port = 30000
|
|
|
-required_nodes = 4
|
|
|
-distributed_backend = "mp"
|
|
|
+[env]
|
|
|
+VLLM_LOGGING_COLOR = "1" # override any launcher default
|
|
|
```
|
|
|
|
|
|
-GLM-5.1 with vLLM across 4 nodes. Pipeline parallelism (PP=4, TP=1). MTP speculative decoding, TRITON_MLA attention, prefix caching, chunked prefill.
|
|
|
+## Configuration
|
|
|
|
|
|
-```bash
|
|
|
-cargo run serve --recipe glm51
|
|
|
-cargo run logs --recipe glm51 --follow
|
|
|
-```
|
|
|
+The launcher reads `config.toml` from the current directory,
|
|
|
+falling back to `~/.config/sm121-launcher/config.toml`.
|
|
|
|
|
|
-### [`minimax-m3`](recipes/minimax-m3/recipe.toml) — SGLang (Minimax-M3-v0 NVFP4)
|
|
|
+### Minimal
|
|
|
|
|
|
```toml
|
|
|
-engine = "sglang"
|
|
|
-docker_image = "scitrera/dgx-spark-sglang-mm:v0"
|
|
|
-model = "sparkarena/Minimax-M3-v0-NVFP4"
|
|
|
-port = 30000
|
|
|
-required_nodes = 4
|
|
|
-tensor_parallel_size = 4
|
|
|
-distributed_backend = "mp"
|
|
|
-entrypoint = ["sglang", "serve"]
|
|
|
-```
|
|
|
-
|
|
|
-Minimax-M3 with SGLang, 4 nodes, TP=4. NVFP4 quantization, `cutlass` fp4 gemm, `triton` MoE runner, `instanttensor` load format.
|
|
|
+ssh_user = "kilork"
|
|
|
+ssh_key_path = "/path/to/ssh/key"
|
|
|
|
|
|
-```bash
|
|
|
-cargo run serve --recipe minimax-m3
|
|
|
-cargo run logs --recipe minimax-m3
|
|
|
+[[clusters]]
|
|
|
+name = "4nodes-switch"
|
|
|
+master_ib_ip = "192.168.200.15"
|
|
|
+nodes = [
|
|
|
+ { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0 },
|
|
|
+ { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1 },
|
|
|
+ { name = "dgx3", ssh_ip = "10.0.0.3", ib_ip = "192.168.200.17", rank = 2 },
|
|
|
+ { name = "dgx4", ssh_ip = "10.0.0.4", ib_ip = "192.168.200.19", rank = 3 },
|
|
|
+]
|
|
|
```
|
|
|
|
|
|
-## Configuration
|
|
|
-
|
|
|
-### config.toml
|
|
|
-
|
|
|
-Create a `config.toml` in the project root (not tracked in git):
|
|
|
+### With Home Assistant (power management)
|
|
|
|
|
|
```toml
|
|
|
-ssh_user = "kilork"
|
|
|
-ssh_key_path = "/path/to/ssh/key"
|
|
|
-
|
|
|
[[clusters]]
|
|
|
name = "4nodes-switch"
|
|
|
master_ib_ip = "192.168.200.15"
|
|
|
+extra_entities = ["switch.network_switch"] # cluster-level switches
|
|
|
nodes = [
|
|
|
- { name = "dgx1", ssh_ip = "192.168.88.14", ib_ip = "192.168.200.15", rank = 0 },
|
|
|
- { name = "dgx2", ssh_ip = "192.168.88.10", ib_ip = "192.168.200.13", rank = 1 },
|
|
|
- { name = "dgx3", ssh_ip = "192.168.88.50", ib_ip = "192.168.200.17", rank = 2 },
|
|
|
- { name = "dgx4", ssh_ip = "192.168.88.51", ib_ip = "192.168.200.19", rank = 3 },
|
|
|
+ { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0, entity = "switch.dgx1" },
|
|
|
+ { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1, entity = "switch.dgx2" },
|
|
|
+ { name = "dgx3", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.17", rank = 2, entity = "switch.dgx3" },
|
|
|
+ { name = "dgx4", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.19", rank = 3, entity = "switch.dgx4" },
|
|
|
]
|
|
|
|
|
|
[nccl."4nodes-switch"]
|
|
|
@@ -161,305 +143,103 @@ ib_hca = "rocep1s0f0,roceP2p1s0f0"
|
|
|
socket_ifname = "enp1s0f0np0"
|
|
|
ib_gid_index = 3
|
|
|
ib_disable = 0
|
|
|
-```
|
|
|
-
|
|
|
-| Field | Purpose |
|
|
|
-|---|---|
|
|
|
-| `ssh_user` / `ssh_key_path` | SSH credential for all nodes |
|
|
|
-| `clusters[].master_ib_ip` | InfiniBand IP of the head node (rank 0) |
|
|
|
-| `nodes[].ssh_ip` | SSH reachable IP (usually Ethernet) |
|
|
|
-| `nodes[].ib_ip` | InfiniBand / RoCE IP for NCCL communication |
|
|
|
-| `nodes[].rank` | Node rank (0 = head, 1..N = workers) |
|
|
|
-| `nodes[].host_key` | Optional SHA-256 fingerprint — run `cargo run health --fingerprint` to discover |
|
|
|
-| `nccl."<name>".*` | NCCL environment variables for the cluster's IB fabric |
|
|
|
-
|
|
|
-Multiple clusters can be defined; `--cluster auto` picks the first one with an NCCL section.
|
|
|
-
|
|
|
-### Creating a recipe
|
|
|
-
|
|
|
-Recipes live in `recipes/<name>/recipe.toml` (directory name intentionally misspelled).
|
|
|
-
|
|
|
-```toml
|
|
|
-engine = "vllm" # or "sglang" (default: "vllm")
|
|
|
-name = "Human Readable Name"
|
|
|
-docker_image = "my-image:tag"
|
|
|
-model = "org/model-name"
|
|
|
-port = 30000
|
|
|
-required_nodes = 4 # optional, omitted = no check
|
|
|
-pipeline_parallel_size = 0 # 0 = auto (equals nnodes), vLLM only
|
|
|
-tensor_parallel_size = 1 # per-node TP for vLLM, total TP for SGLang
|
|
|
-distributed_backend = "mp" # "mp" (multiprocessing) or "ray"
|
|
|
-cluster = "4nodes-switch" # optional: pin to a specific cluster
|
|
|
|
|
|
-entrypoint = ["sglang", "serve"] # optional, appended after image name
|
|
|
-
|
|
|
-args = [
|
|
|
- "--kv-cache-dtype", "fp8",
|
|
|
- "--max-model-len", "32768",
|
|
|
-]
|
|
|
-
|
|
|
-[env]
|
|
|
-SOME_VAR = "value"
|
|
|
-```
|
|
|
-
|
|
|
-#### Engine-specific behavior
|
|
|
-
|
|
|
-| Aspect | vLLM | SGLang |
|
|
|
-|---|---|---|
|
|
|
-| Host IP env var | `VLLM_HOST_IP` | `SGLANG_WORKER_IP` |
|
|
|
-| Model argument | bare model name | `--model-path <model>` |
|
|
|
-| TP flag | `--tensor-parallel-size 1` | `--tp <tensor_parallel_size>` |
|
|
|
-| PP flag | `--pipeline-parallel-size <pp_size>` | _(not used)_ |
|
|
|
-| Backend flag | `--distributed-executor-backend mp|ray` | _(not used)_ |
|
|
|
-| Common flags | `--nnodes`, `--node-rank`, `--master-addr` | same |
|
|
|
-| Port flag | _(recipe args)_ | `--port <port>` (auto-generated) |
|
|
|
-| Entrypoint | `python -m vllm.entrypoints.openai.api_server` | recipe-defined (e.g. `sglang serve`) |
|
|
|
-
|
|
|
-## Launch Behavior
|
|
|
-
|
|
|
-### Docker runtime
|
|
|
-
|
|
|
-For each node, the tool builds a `docker run` command via SSH:
|
|
|
-
|
|
|
-```
|
|
|
-docker run -d --gpus all --ipc=host --privileged --network=host \
|
|
|
- -e NCCL_IB_DISABLE=<...> -e NCCL_IB_HCA=<...> ... \
|
|
|
- -e SGLANG_WORKER_IP=<node.ib_ip> \ # or VLLM_HOST_IP
|
|
|
- -e <recipe.env>... \
|
|
|
- [-p <port>:<port>] \ # head node only
|
|
|
- --name sm121_<recipe>_<rank> \
|
|
|
- --label sm121.recipe=<recipe> \
|
|
|
- --label sm121.rank=<rank> \
|
|
|
- <docker_image> \
|
|
|
- <entrypoint>... \ # e.g. sglang serve
|
|
|
- [--model-path] <model> \ # engine-specific
|
|
|
- <recipe.args>... \ # from recipe
|
|
|
- --tp <tensor_parallel_size> \ # engine-specific flags
|
|
|
- --nnodes <nnodes> \
|
|
|
- --node-rank <rank> \
|
|
|
- --master-addr <master_ib_ip>
|
|
|
- [--port <port>] # SGLang only
|
|
|
-```
|
|
|
-
|
|
|
-NCCL env vars come from the cluster's `[nccl."<name>"]` section. All containers use `--network=host` for direct InfiniBand access.
|
|
|
-
|
|
|
-### Tmux runtime
|
|
|
-
|
|
|
-`cargo run tmux serve --recipe <name>` creates a 3-window tmux session on each node:
|
|
|
-
|
|
|
-| Window | Content |
|
|
|
-|--------|---------|
|
|
|
-| 0: `model` | Main inference process (vLLM/SGLang) |
|
|
|
-| 1: `gpu` | `nvidia-smi -l 5` GPU monitor |
|
|
|
-| 2: `shell` | Debug shell |
|
|
|
-
|
|
|
-For Ray backend, nodes start sequentially (head first with API readiness poll up to 150s). For MP backend, all nodes start in parallel.
|
|
|
-
|
|
|
-## Environment Variables
|
|
|
-
|
|
|
-The launcher passes the following env vars to every container. Recipe-level
|
|
|
-`[env]` entries are appended after the defaults and can override them.
|
|
|
-
|
|
|
-### Cluster-driven (from `[nccl."<cluster>"]`)
|
|
|
-
|
|
|
-| Env var | Source | Purpose |
|
|
|
-|---|---|---|
|
|
|
-| `NCCL_IB_DISABLE` | `nccl.<cluster>.ib_disable` | Disable NCCL's InfiniBand path (0 = use IB, 1 = disable) |
|
|
|
-| `NCCL_IB_HCA` | `nccl.<cluster>.ib_hca` | RDMA devices for NCCL (comma-separated, e.g. `rocep1s0f0,roceP2p1s0f0`) |
|
|
|
-| `NCCL_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | Network interface NCCL uses for socket-based control (e.g. `enp1s0f0np0`) |
|
|
|
-| `NCCL_IB_GID_INDEX` | `nccl.<cluster>.ib_gid_index` | RoCE GID index for InfiniBand routing |
|
|
|
-| `GLOO_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | Network interface for Gloo (vLLM distributed + PyTorch CPU collectives) |
|
|
|
-| `TP_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | vLLM tensor-parallel coordinator interface |
|
|
|
-| `MN_IF_NAME` | `nccl.<cluster>.socket_ifname` | Multi-node interface name |
|
|
|
-| `UCX_NET_DEVICES` | `nccl.<cluster>.socket_ifname` | UCX communication devices |
|
|
|
-| `OMPI_MCA_btl_tcp_if_include` | `nccl.<cluster>.socket_ifname` | OpenMPI TCP interface |
|
|
|
-| `VLLM_LOGGING_COLOR` | `1` | Force colored vLLM logs (head node only) |
|
|
|
-
|
|
|
-### Hard-coded defaults
|
|
|
-
|
|
|
-| Env var | Value | Purpose |
|
|
|
-|---|---|---|
|
|
|
-| `NCCL_IGNORE_CPU_AFFINITY` | `1` | Don't pin NCCL threads to a specific CPU set (needed on DGX Spark) |
|
|
|
-| `NCCL_NET` | `IB` | Force NCCL to use the InfiniBand plugin (`IB` = `libnccl-net.so`) |
|
|
|
-| `NCCL_DEBUG` | `WARN` | NCCL log level (override with `[env]` for `INFO`/`TRACE` while debugging) |
|
|
|
-| `OMP_NUM_THREADS` | `4` | OpenMP thread limit per process |
|
|
|
-| `HF_HOME` | `/root/.cache/huggingface` | HuggingFace cache root (volume-mounted from `ssh_user`'s host cache) |
|
|
|
-
|
|
|
-### Engine-specific host IP
|
|
|
-
|
|
|
-| Env var | Engines | Value | Purpose |
|
|
|
-|---|---|---|---|
|
|
|
-| `VLLM_HOST_IP` | vLLM only | `node.ib_ip` | vLLM's own host-IP for distributed init |
|
|
|
-| `SGLANG_WORKER_IP` | SGLang only | `node.ib_ip` | SGLang's worker-IP for distributed init |
|
|
|
-
|
|
|
-### Ray-only (when `distributed_backend = "ray"`)
|
|
|
-
|
|
|
-| Env var | Value | Purpose |
|
|
|
-|---|---|---|
|
|
|
-| `RAY_ADDRESS` | `<master_ib_ip>:6379` | Ray head address (workers connect here) |
|
|
|
-| `RAY_PORT` | `recipe.port` | Ray gRPC port |
|
|
|
-
|
|
|
-### Recipe-level overrides
|
|
|
-
|
|
|
-Anything in `[env]` is appended after all of the above, so a recipe can
|
|
|
-override any default (e.g. `NCCL_CUMEM_ENABLE = "1"` to re-enable
|
|
|
-CUDA-managed memory).
|
|
|
-
|
|
|
-## Power Management (Home Assistant)
|
|
|
-
|
|
|
-Optional `[home_assistant]` section lets the launcher toggle smart
|
|
|
-plugs (or any switch domain) so you can shut down and wake nodes
|
|
|
-remotely:
|
|
|
-
|
|
|
-```toml
|
|
|
[home_assistant]
|
|
|
url = "http://homeassistant.local:8123"
|
|
|
token = "long-lived-access-token"
|
|
|
-# Seconds to wait between `sudo shutdown now` and HA `turn_off` so
|
|
|
-# the OS has time to actually shut down. Default: 30.
|
|
|
shutdown_delay_seconds = 30
|
|
|
-```
|
|
|
-
|
|
|
-Each node can optionally carry an `entity` field for the plug that
|
|
|
-controls it:
|
|
|
-
|
|
|
-```toml
|
|
|
-[[clusters]]
|
|
|
-name = "4nodes-switch"
|
|
|
-master_ib_ip = "192.168.200.15"
|
|
|
|
|
|
-# Optional: extra switches to toggle alongside the nodes. Useful for
|
|
|
-# a network switch that powers the cluster, or a PDU that backs the
|
|
|
-# rack. Toggled in addition to any per-node entities.
|
|
|
-extra_entities = ["switch.network_switch_4nodes"]
|
|
|
+[exec_aliases]
|
|
|
+gpu-lock = "sudo nvidia-smi -lgc 300,2300"
|
|
|
+shutdown = "sudo shutdown now"
|
|
|
|
|
|
-nodes = [
|
|
|
- { name = "dgx1", entity = "switch.dgx1", ssh_ip = "10.0.0.1", ib_ip = "1.2.3.4", rank = 0 },
|
|
|
- # ...
|
|
|
+[before_serve."4nodes-switch"]
|
|
|
+commands = [
|
|
|
+ "sudo nvidia-smi -lgc 300,2300",
|
|
|
+ { command = "sudo nvidia-smi -pm 1", on_failure = "warn" },
|
|
|
]
|
|
|
```
|
|
|
|
|
|
-Without an `entity` the node is skipped (with a warning) — the SSH
|
|
|
-shutdown still happens, but Home Assistant is not consulted.
|
|
|
-Cluster-level `extra_entities` are always toggled (no opt-in).
|
|
|
+### NCCL config
|
|
|
|
|
|
-### `power off` flow
|
|
|
+Required for each cluster. Controls InfiniBand/RoCE communication:
|
|
|
|
|
|
-```
|
|
|
-1. ssh <node>: sudo shutdown now
|
|
|
-2. wait shutdown_delay_seconds (default 30s)
|
|
|
-3. HA POST /api/services/switch/turn_off
|
|
|
- (entity: switch.dgx1)
|
|
|
-```
|
|
|
+| Field | Example | Purpose |
|
|
|
+|-------|---------|---------|
|
|
|
+| `ib_hca` | `rocep1s0f0,roceP2p1s0f0` | RDMA devices |
|
|
|
+| `socket_ifname` | `enp1s0f0np0` | Network interface |
|
|
|
+| `ib_gid_index` | `3` | RoCE GID index |
|
|
|
+| `ib_disable` | `0` | Keep IB enabled |
|
|
|
+
|
|
|
+## Power Management
|
|
|
+
|
|
|
+Requires `[home_assistant]` in config.toml.
|
|
|
|
|
|
```bash
|
|
|
+# Graceful shutdown + wait + turn off smart plugs
|
|
|
sm121-launcher power off --cluster 4nodes-switch
|
|
|
-sm121-launcher power off --cluster 4nodes-switch --node dgx1
|
|
|
-sm121-launcher -d power off --cluster 4nodes-switch # dry-run
|
|
|
-```
|
|
|
|
|
|
-### `power on` flow
|
|
|
-
|
|
|
-`power on` does **not** SSH — the smart plug wakes the machine:
|
|
|
+# Wake nodes via smart plugs
|
|
|
+sm121-launcher power on --cluster 4nodes-switch
|
|
|
|
|
|
-```
|
|
|
-1. HA POST /api/services/switch/turn_on
|
|
|
- (entity: switch.dgx1)
|
|
|
-```
|
|
|
+# Switch clusters (disable 4 → enable 2)
|
|
|
+sm121-launcher power switch --from 4 --to 2
|
|
|
|
|
|
-Requires `[home_assistant]` to be configured (no other way to wake
|
|
|
-a node).
|
|
|
+# Single node
|
|
|
+sm121-launcher power off --cluster 4nodes-switch --node dgx1
|
|
|
|
|
|
-```bash
|
|
|
-sm121-launcher power on --cluster 4nodes-switch
|
|
|
-sm121-launcher power on --cluster 4nodes-switch --node dgx1
|
|
|
+# Dry-run
|
|
|
+sm121-launcher -d power off --cluster 4nodes-switch
|
|
|
```
|
|
|
|
|
|
-### `power switch` flow
|
|
|
+**Power-off flow**: SSH `sudo shutdown now` → wait `shutdown_delay_seconds` → HA `turn_off` entity → HA `turn_off` cluster extras.
|
|
|
|
|
|
-Switch between clusters: disable nodes/entities unique to `from`,
|
|
|
-enable everything in `to`.
|
|
|
+**Power-on flow**: HA `turn_on` (no SSH, assumes smart plug wakes the machine).
|
|
|
|
|
|
-```
|
|
|
-1. SSH sudo shutdown now on nodes in from but not to
|
|
|
-2. Wait shutdown_delay_seconds
|
|
|
-3. HA turn_off those nodes + extra_entities unique to from
|
|
|
-4. HA turn_on all nodes + extra_entities in to
|
|
|
-```
|
|
|
+**Switch flow**: disable nodes/extras only in source → enable all in target.
|
|
|
|
|
|
-```bash
|
|
|
-sm121-launcher power switch --from 4 --to 2 # prefix matching works
|
|
|
-```
|
|
|
+## Environment Variables
|
|
|
|
|
|
-## Doctor Mode
|
|
|
+The launcher sets these automatically. Recipe `[env]` overrides any.
|
|
|
|
|
|
-```bash
|
|
|
-cargo run doctor --recipe minimax-m3 # Check node prerequisites
|
|
|
-cargo run doctor --recipe minimax-m3 --repair # Auto-fix issues
|
|
|
-```
|
|
|
+### Auto-set (always)
|
|
|
|
|
|
-Checks each node for:
|
|
|
-- Docker availability
|
|
|
-- NVIDIA GPU (`nvidia-smi`)
|
|
|
-- NCCL libraries
|
|
|
-- Engine version inside the image (`vllm version` / `sglang version`)
|
|
|
-- Ray availability (if backend is "ray")
|
|
|
+| Env var | Source | Description |
|
|
|
+|---------|--------|-------------|
|
|
|
+| `NCCL_IB_DISABLE` | `nccl.<name>.ib_disable` | InfiniBand on/off |
|
|
|
+| `NCCL_IB_HCA` | `nccl.<name>.ib_hca` | RDMA devices |
|
|
|
+| `NCCL_SOCKET_IFNAME` | `nccl.<name>.socket_ifname` | NCCL control interface |
|
|
|
+| `GLOO_SOCKET_IFNAME` | same | Gloo (PyTorch) interface |
|
|
|
+| `TP_SOCKET_IFNAME` | same | vLLM TP coordinator |
|
|
|
+| `MN_IF_NAME` | same | Multi-node interface |
|
|
|
+| `UCX_NET_DEVICES` | same | UCX devices |
|
|
|
+| `OMPI_MCA_btl_tcp_if_include` | same | OpenMPI TCP interface |
|
|
|
+| `NCCL_IB_GID_INDEX` | `nccl.<name>.ib_gid_index` | RoCE GID |
|
|
|
+| `VLLM_HOST_IP` / `SGLANG_WORKER_IP` | `node.ib_ip` | Engine host IP |
|
|
|
+| `HF_HOME` | `/root/.cache/huggingface` | HF cache (volume-mounted) |
|
|
|
+| `OMP_NUM_THREADS` | `4` | OpenMP threads |
|
|
|
+| `NCCL_IGNORE_CPU_AFFINITY` | `1` | Don't pin NCCL threads |
|
|
|
+| `NCCL_NET` | `IB` | NCCL IB plugin |
|
|
|
+| `NCCL_DEBUG` | `WARN` | NCCL log level |
|
|
|
+| `VLLM_LOGGING_COLOR` | `1` | Colored logs (head only) |
|
|
|
+| `TERM` | `xterm-256color` | Terminal type (head only) |
|
|
|
|
|
|
-With `--repair`, generates a Dockerfile that installs missing packages and rebuilds the image with a `.repaired` tag. The repaired image is auto-detected on subsequent `serve` runs.
|
|
|
+### Ray backend (when `distributed_backend = "ray"`)
|
|
|
|
|
|
-## Internationalization (i18n)
|
|
|
+`RAY_ADDRESS`, `RAY_PORT`, `RAY_NODE_IP_ADDRESS`, `RAY_OVERRIDE_NODE_IP_ADDRESS`
|
|
|
|
|
|
-Log messages are localized via Mozilla Fluent (`.ftl` files in `locales/`).
|
|
|
-Auto-detects system locale. Available languages:
|
|
|
+## Internationalization
|
|
|
|
|
|
-| Locale | File | Status |
|
|
|
-|--------|------|--------|
|
|
|
-| English | `locales/en/main.ftl` | Fallback |
|
|
|
-| Russian | `locales/ru/main.ftl` | Full |
|
|
|
-| Spanish | `locales/es/main.ftl` | Full |
|
|
|
-| Italian | `locales/it/main.ftl` | Full |
|
|
|
-| French | `locales/fr/main.ftl` | Full |
|
|
|
-| German | `locales/de/main.ftl` | Full |
|
|
|
+Messages are localized. Auto-detects system locale.
|
|
|
|
|
|
```bash
|
|
|
-# Override locale via environment
|
|
|
-LANG=ru_RU.UTF-8 cargo run serve --recipe deepseek-v4-flash
|
|
|
+# Override locale
|
|
|
+LANG=ru_RU.UTF-8 sm121-launcher serve --recipe deepseek-v4-flash
|
|
|
```
|
|
|
|
|
|
-## CLI Reference
|
|
|
-
|
|
|
-```text
|
|
|
-Usage: sm121-launcher [OPTIONS] <COMMAND>
|
|
|
-
|
|
|
-Commands:
|
|
|
- serve Launch a recipe on a cluster
|
|
|
- logs Show container logs
|
|
|
- stop Stop and remove containers for a recipe
|
|
|
- health Check cluster health (or fingerprint host keys)
|
|
|
- doctor Check and repair node prerequisites
|
|
|
- tmux Manage tmux sessions (alternative to Docker)
|
|
|
- power Power on/off nodes via Home Assistant
|
|
|
- exec Run an arbitrary shell command on a cluster or single node
|
|
|
-
|
|
|
-Global flags:
|
|
|
- -c, --config <FILE> Config file path (default: config.toml)
|
|
|
- -d, --dry-run Print commands without executing
|
|
|
- -h, --help Print help
|
|
|
- -V, --version Print version
|
|
|
-```
|
|
|
+Available: English (fallback), Russian, Spanish, Italian, French, German.
|
|
|
+
|
|
|
+## Project Links
|
|
|
|
|
|
-## Dependencies
|
|
|
-
|
|
|
-| Crate | Purpose |
|
|
|
-|---|---|
|
|
|
-| `clap` 4.6 + derive | CLI argument parsing |
|
|
|
-| `russh` 0.61 + `russh-keys` 0.50-beta | SSH client |
|
|
|
-| `tokio` full | Async runtime |
|
|
|
-| `toml` 1 + `serde` 1 | Config / recipe deserialization |
|
|
|
-| `tracing` + `tracing-subscriber` | Structured logging |
|
|
|
-| `anyhow` | Error handling |
|
|
|
-| `futures` 0.3 | Concurrent SSH (`join_all`) |
|
|
|
-| `sha2` 0.11 + `base64` 0.22 | Host key fingerprinting |
|
|
|
-| `reqwest` 0.13 | HTTP client for Home Assistant |
|
|
|
-| `dirs` 6 | OS config dir discovery |
|
|
|
-| `fluent-i18n` | Mozilla Fluent i18n (locales: en, ru, es, it, fr, de) |
|
|
|
+- **Development docs**: [`DEVELOPMENT.md`](DEVELOPMENT.md)
|
|
|
+- **Agent reference**: [`AGENTS.md`](AGENTS.md)
|