# sm121-launcher Launch vLLM / SGLang inference clusters across DGX Spark machines via SSH. ## Quick Install ```bash # From source cargo install --path . # Or download a pre-built binary (requires Rust toolchain to build) cargo build --release # Binary at target/release/sm121-launcher ``` ## Quick Start ```bash # Create config.toml (see Configuration below) # Launch a recipe sm121-launcher serve --recipe deepseek-v4-flash # Follow logs sm121-launcher logs --recipe deepseek-v4-flash --follow # Stop containers sm121-launcher stop --recipe deepseek-v4-flash ``` ## Commands | Command | Description | |---------|-------------| | `serve --recipe ` | Launch recipe on cluster | | `logs --recipe [--follow]` | Show container logs | | `stop --recipe ` | Stop and remove containers | | `health [--cluster ]` | Check cluster health | | `doctor --recipe [--repair]` | Check/repair node prerequisites | | `exec --cluster -- ` | Run command on cluster nodes | | `exec --cluster --alias ` | Run predefined alias | | `power off --cluster [--node ]` | Shutdown + HA turn_off | | `power on --cluster [--node ]` | HA turn_on (wake nodes) | | `power switch --from --to ` | Switch clusters | | `tmux serve --recipe ` | Launch via tmux (no Docker) | | `tmux status --recipe ` | Check tmux session health | | `tmux logs --recipe ` | Capture tmux pane output | | `init-host-keys` | Auto-write SSH host keys | | `-d` / `--dry-run` | Preview commands without executing | ### Cluster selection ```bash sm121-launcher serve --recipe deepseek-v4-flash # auto (smallest match) sm121-launcher serve --recipe glm51 --cluster 4nodes-switch # exact name sm121-launcher serve --recipe glm51 --cluster 4 # prefix match sm121-launcher serve --recipe glm51 --cluster 2 # picks 2nodes-dac ``` Prefix must be unambiguous — error if multiple clusters match. ## Recipes Recipes are TOML files in `recipes//recipe.toml`. The launcher looks for them in the current directory first, then in `~/.config/sm121-launcher/recipes//recipe.toml`. ### Available | Recipe | Engine | Nodes | Model | |--------|--------|-------|-------| | `deepseek-v4-flash` | vLLM | 2 | DeepSeek V4 Flash (MXFP8, 1M ctx, MTP) | | `minimax-m3-mxfp8` | vLLM | 4 | MiniMax M3 (MXFP8, 428B, MSA sparse attn) | | `glm51` | vLLM | 4 | GLM-5.1 NVFP8 | | `minimax-m3` | SGLang | 4 | MiniMax-M3-v0 NVFP4 | | `test-sleep` | vLLM | any | Heartbeat test container | ### Creating a recipe ```toml engine = "vllm" # or "sglang" (default: "vllm") name = "My Model" docker_image = "my-image:tag" model = "org/model-name" required_nodes = 2 # optional tensor_parallel_size = 2 # TP across nodes pipeline_parallel_size = 1 # vLLM only, 0 = auto (equals nnodes) distributed_backend = "mp" # "mp" (default) or "ray" shm_size = "64gb" # optional, only added when set entrypoint = ["vllm", "serve"] # optional, appended after image args = [ "--kv-cache-dtype", "fp8", "--max-model-len", "32768", ] before_serve = [ # recipe-level pre-serve hooks { command = "echo ready", on_failure = "abort" }, ] [env] VLLM_LOGGING_COLOR = "1" # override any launcher default ``` ## Configuration The launcher reads `config.toml` from the current directory, falling back to `~/.config/sm121-launcher/config.toml`. ### Minimal ```toml ssh_user = "kilork" ssh_key_path = "/path/to/ssh/key" [[clusters]] name = "4nodes-switch" master_ib_ip = "192.168.200.15" nodes = [ { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0 }, { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1 }, { name = "dgx3", ssh_ip = "10.0.0.3", ib_ip = "192.168.200.17", rank = 2 }, { name = "dgx4", ssh_ip = "10.0.0.4", ib_ip = "192.168.200.19", rank = 3 }, ] ``` ### With Home Assistant (power management) ```toml [[clusters]] name = "4nodes-switch" master_ib_ip = "192.168.200.15" extra_entities = ["switch.network_switch"] # cluster-level switches nodes = [ { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0, entity = "switch.dgx1" }, { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1, entity = "switch.dgx2" }, { name = "dgx3", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.17", rank = 2, entity = "switch.dgx3" }, { name = "dgx4", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.19", rank = 3, entity = "switch.dgx4" }, ] [nccl."4nodes-switch"] ib_hca = "rocep1s0f0,roceP2p1s0f0" socket_ifname = "enp1s0f0np0" ib_gid_index = 3 ib_disable = 0 [home_assistant] url = "http://homeassistant.local:8123" token = "long-lived-access-token" shutdown_delay_seconds = 30 [exec_aliases] gpu-lock = "sudo nvidia-smi -lgc 300,2300" shutdown = "sudo shutdown now" [before_serve."4nodes-switch"] commands = [ "sudo nvidia-smi -lgc 300,2300", { command = "sudo nvidia-smi -pm 1", on_failure = "warn" }, ] ``` ### NCCL config Required for each cluster. Controls InfiniBand/RoCE communication: | Field | Example | Purpose | |-------|---------|---------| | `ib_hca` | `rocep1s0f0,roceP2p1s0f0` | RDMA devices | | `socket_ifname` | `enp1s0f0np0` | Network interface | | `ib_gid_index` | `3` | RoCE GID index | | `ib_disable` | `0` | Keep IB enabled | ## Power Management Requires `[home_assistant]` in config.toml. ```bash # Graceful shutdown + wait + turn off smart plugs sm121-launcher power off --cluster 4nodes-switch # Wake nodes via smart plugs sm121-launcher power on --cluster 4nodes-switch # Switch clusters (disable 4 → enable 2) sm121-launcher power switch --from 4 --to 2 # Single node sm121-launcher power off --cluster 4nodes-switch --node dgx1 # Dry-run sm121-launcher -d power off --cluster 4nodes-switch ``` **Power-off flow**: SSH `sudo shutdown now` → wait `shutdown_delay_seconds` → HA `turn_off` entity → HA `turn_off` cluster extras. **Power-on flow**: HA `turn_on` (no SSH, assumes smart plug wakes the machine). **Switch flow**: disable nodes/extras only in source → enable all in target. ## Environment Variables The launcher sets these automatically. Recipe `[env]` overrides any. ### Auto-set (always) | Env var | Source | Description | |---------|--------|-------------| | `NCCL_IB_DISABLE` | `nccl..ib_disable` | InfiniBand on/off | | `NCCL_IB_HCA` | `nccl..ib_hca` | RDMA devices | | `NCCL_SOCKET_IFNAME` | `nccl..socket_ifname` | NCCL control interface | | `GLOO_SOCKET_IFNAME` | same | Gloo (PyTorch) interface | | `TP_SOCKET_IFNAME` | same | vLLM TP coordinator | | `MN_IF_NAME` | same | Multi-node interface | | `UCX_NET_DEVICES` | same | UCX devices | | `OMPI_MCA_btl_tcp_if_include` | same | OpenMPI TCP interface | | `NCCL_IB_GID_INDEX` | `nccl..ib_gid_index` | RoCE GID | | `VLLM_HOST_IP` / `SGLANG_WORKER_IP` | `node.ib_ip` | Engine host IP | | `HF_HOME` | `/root/.cache/huggingface` | HF cache (volume-mounted) | | `OMP_NUM_THREADS` | `4` | OpenMP threads | | `NCCL_IGNORE_CPU_AFFINITY` | `1` | Don't pin NCCL threads | | `NCCL_NET` | `IB` | NCCL IB plugin | | `NCCL_DEBUG` | `WARN` | NCCL log level | | `VLLM_LOGGING_COLOR` | `1` | Colored logs (head only) | | `TERM` | `xterm-256color` | Terminal type (head only) | ### Ray backend (when `distributed_backend = "ray"`) `RAY_ADDRESS`, `RAY_PORT`, `RAY_NODE_IP_ADDRESS`, `RAY_OVERRIDE_NODE_IP_ADDRESS` ## Internationalization Messages are localized. Auto-detects system locale. ```bash # Override locale (works on both macOS and Linux) LANG=ru_RU.UTF-8 RUST_LOG=info sm121-launcher serve --recipe deepseek-v4-flash ``` Available: English (fallback), Russian, Spanish, Italian, French, German. ## Project Links - **Development docs**: [`DEVELOPMENT.md`](DEVELOPMENT.md) - **Agent reference**: [`AGENTS.md`](AGENTS.md)