# sm121-launcher

Rust CLI tool for launching vLLM / SGLang containers across a DGX Spark cluster via SSH.

## Features

- Launch distributed LLM inference clusters on 2–4 DGX Spark machines
- **vLLM** and **SGLang** engines — auto-generates the right distributed flags for each
- Docker (default) or **tmux** (alternative) runtime
- SSH-based provisioning via `russh` — no agent required on target nodes
- Cluster health checks, container log tailing, and clean teardown
- **Doctor mode** — pre-flight checks and auto-repair of broken container images
- Dry-run (`-d`) to preview every SSH command without execution

## Quality Gates

```bash
make check            # Run all quality gates (fmt, test, clippy, line count)
make check-fmt        # cargo fmt --check
make check-test       # cargo test
make check-clippy     # cargo clippy with -D warnings
make check-lines      # Ensure no Rust file exceeds 500 lines
```

## Quick Start

```bash
# Build
cargo build --release

# Launch a recipe (auto-selects cluster from config.toml)
cargo run serve --recipe glm51                # vLLM — GLM-5.1 NVFP8
cargo run serve --recipe minimax-m3            # SGLang — Minimax-M3 NVFP4

# Launch on a specific cluster
cargo run serve --recipe glm51 --cluster 4nodes-switch

# Dry-run (print SSH commands without executing)
cargo run -- -d serve --recipe minimax-m3

# View cluster logs
cargo run logs --recipe glm51                  # Last 100 lines per node
cargo run logs --recipe minimax-m3 --follow    # Stream head node logs

# Stop all containers for a recipe
cargo run stop --recipe glm51

# Health check
cargo run health                               # All clusters
cargo run health --cluster 2nodes-dac          # Single cluster
cargo run health --fingerprint                 # Discover host keys for config.toml

# Doctor mode — check prerequisites
cargo run doctor --recipe minimax-m3           # Check node prerequisites
cargo run doctor --recipe minimax-m3 --repair  # Auto-fix (build repaired image)

# Tmux runtime (no Docker)
cargo run tmux serve --recipe glm51
cargo run tmux status --recipe glm51
cargo run tmux attach --recipe glm51
```

## Available Recipes

### [`deepseek-v4-flash`](recipes/deepseek-v4-flash/recipe.toml) — vLLM (DeepSeek V4 Flash MXFP8)

```toml
engine = "vllm"
docker_image = "aidendle94/sparkrun-vllm-ds4-gb10:production-v2"
model = "deepseek-ai/DeepSeek-V4-Flash"
required_nodes = 2
tensor_parallel_size = 2
pipeline_parallel_size = 1
distributed_backend = "mp"
```

DeepSeek V4 Flash with 1M context, FP8 KV cache, MTP speculative decoding, and DeepSeek V4 reasoning/tool parsers. 2 nodes, TP=2 across nodes. Uses `sparkrun` DGX Spark image.

```bash
cargo run serve --recipe deepseek-v4-flash          # auto-selects 2nodes-dac
cargo run logs --recipe deepseek-v4-flash --follow
```

### [`minimax-m3-mxfp8`](recipes/minimax-m3-mxfp8/recipe.toml) — vLLM (MiniMax M3 MXFP8)

```toml
engine = "vllm"
docker_image = "aidendle94/sparkrun-vllm-ds4-gb10:production-v2"
model = "MiniMaxAI/MiniMax-M3-MXFP8"
required_nodes = 4
tensor_parallel_size = 4
distributed_backend = "mp"
```

MiniMax M3 vision-language MoE (428B params, 23B active) with MXFP8 quantization, MSA sparse attention (`--block-size 128` mandatory), expert parallelism, and `minimax_m3` parsers. Text-only mode skips the vision encoder. 4 nodes, TP=4.

```bash
cargo run serve --recipe minimax-m3-mxfp8
```

### [`glm51`](recipes/glm51/recipe.toml) — vLLM (GLM-5.1 NVFP8)

```toml
engine = "vllm"
docker_image = "vllm/vllm-openai:glm51-cu130"
model = "lukealonso/GLM-5.1-NVFP4"
port = 30000
required_nodes = 4
distributed_backend = "mp"
```

GLM-5.1 with vLLM across 4 nodes. Pipeline parallelism (PP=4, TP=1). MTP speculative decoding, TRITON_MLA attention, prefix caching, chunked prefill.

```bash
cargo run serve --recipe glm51
cargo run logs --recipe glm51 --follow
```

### [`minimax-m3`](recipes/minimax-m3/recipe.toml) — SGLang (Minimax-M3-v0 NVFP4)

```toml
engine = "sglang"
docker_image = "scitrera/dgx-spark-sglang-mm:v0"
model = "sparkarena/Minimax-M3-v0-NVFP4"
port = 30000
required_nodes = 4
tensor_parallel_size = 4
distributed_backend = "mp"
entrypoint = ["sglang", "serve"]
```

Minimax-M3 with SGLang, 4 nodes, TP=4. NVFP4 quantization, `cutlass` fp4 gemm, `triton` MoE runner, `instanttensor` load format.

```bash
cargo run serve --recipe minimax-m3
cargo run logs --recipe minimax-m3
```

## Configuration

### config.toml

Create a `config.toml` in the project root (not tracked in git):

```toml
ssh_user = "kilork"
ssh_key_path = "/path/to/ssh/key"

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"
nodes = [
    { name = "dgx1", ssh_ip = "192.168.88.14", ib_ip = "192.168.200.15", rank = 0 },
    { name = "dgx2", ssh_ip = "192.168.88.10", ib_ip = "192.168.200.13", rank = 1 },
    { name = "dgx3", ssh_ip = "192.168.88.50", ib_ip = "192.168.200.17", rank = 2 },
    { name = "dgx4", ssh_ip = "192.168.88.51", ib_ip = "192.168.200.19", rank = 3 },
]

[nccl."4nodes-switch"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3
ib_disable = 0
```

| Field | Purpose |
|---|---|
| `ssh_user` / `ssh_key_path` | SSH credential for all nodes |
| `clusters[].master_ib_ip` | InfiniBand IP of the head node (rank 0) |
| `nodes[].ssh_ip` | SSH reachable IP (usually Ethernet) |
| `nodes[].ib_ip` | InfiniBand / RoCE IP for NCCL communication |
| `nodes[].rank` | Node rank (0 = head, 1..N = workers) |
| `nodes[].host_key` | Optional SHA-256 fingerprint — run `cargo run health --fingerprint` to discover |
| `nccl."<name>".*` | NCCL environment variables for the cluster's IB fabric |

Multiple clusters can be defined; `--cluster auto` picks the first one with an NCCL section.

### Creating a recipe

Recipes live in `recipes/<name>/recipe.toml` (directory name intentionally misspelled).

```toml
engine = "vllm"                        # or "sglang" (default: "vllm")
name = "Human Readable Name"
docker_image = "my-image:tag"
model = "org/model-name"
port = 30000
required_nodes = 4                     # optional, omitted = no check
pipeline_parallel_size = 0             # 0 = auto (equals nnodes), vLLM only
tensor_parallel_size = 1               # per-node TP for vLLM, total TP for SGLang
distributed_backend = "mp"             # "mp" (multiprocessing) or "ray"
cluster = "4nodes-switch"              # optional: pin to a specific cluster

entrypoint = ["sglang", "serve"]       # optional, appended after image name

args = [
    "--kv-cache-dtype", "fp8",
    "--max-model-len", "32768",
]

[env]
SOME_VAR = "value"
```

#### Engine-specific behavior

| Aspect | vLLM | SGLang |
|---|---|---|
| Host IP env var | `VLLM_HOST_IP` | `SGLANG_WORKER_IP` |
| Model argument | bare model name | `--model-path <model>` |
| TP flag | `--tensor-parallel-size 1` | `--tp <tensor_parallel_size>` |
| PP flag | `--pipeline-parallel-size <pp_size>` | _(not used)_ |
| Backend flag | `--distributed-executor-backend mp|ray` | _(not used)_ |
| Common flags | `--nnodes`, `--node-rank`, `--master-addr` | same |
| Port flag | _(recipe args)_ | `--port <port>` (auto-generated) |
| Entrypoint | `python -m vllm.entrypoints.openai.api_server` | recipe-defined (e.g. `sglang serve`) |

## Launch Behavior

### Docker runtime

For each node, the tool builds a `docker run` command via SSH:

```
docker run -d --gpus all --ipc=host --privileged --network=host \
  -e NCCL_IB_DISABLE=<...> -e NCCL_IB_HCA=<...> ... \
  -e SGLANG_WORKER_IP=<node.ib_ip> \       # or VLLM_HOST_IP
  -e <recipe.env>... \
  [-p <port>:<port>] \                       # head node only
  --name sm121_<recipe>_<rank> \
  --label sm121.recipe=<recipe> \
  --label sm121.rank=<rank> \
  <docker_image> \
  <entrypoint>... \                          # e.g. sglang serve
  [--model-path] <model> \                   # engine-specific
  <recipe.args>... \                         # from recipe
  --tp <tensor_parallel_size> \              # engine-specific flags
  --nnodes <nnodes> \
  --node-rank <rank> \
  --master-addr <master_ib_ip>
  [--port <port>]                            # SGLang only
```

NCCL env vars come from the cluster's `[nccl."<name>"]` section. All containers use `--network=host` for direct InfiniBand access.

### Tmux runtime

`cargo run tmux serve --recipe <name>` creates a 3-window tmux session on each node:

| Window | Content |
|--------|---------|
| 0: `model` | Main inference process (vLLM/SGLang) |
| 1: `gpu` | `nvidia-smi -l 5` GPU monitor |
| 2: `shell` | Debug shell |

For Ray backend, nodes start sequentially (head first with API readiness poll up to 150s). For MP backend, all nodes start in parallel.

## Environment Variables

The launcher passes the following env vars to every container.  Recipe-level
`[env]` entries are appended after the defaults and can override them.

### Cluster-driven (from `[nccl."<cluster>"]`)

| Env var | Source | Purpose |
|---|---|---|
| `NCCL_IB_DISABLE` | `nccl.<cluster>.ib_disable` | Disable NCCL's InfiniBand path (0 = use IB, 1 = disable) |
| `NCCL_IB_HCA` | `nccl.<cluster>.ib_hca` | RDMA devices for NCCL (comma-separated, e.g. `rocep1s0f0,roceP2p1s0f0`) |
| `NCCL_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | Network interface NCCL uses for socket-based control (e.g. `enp1s0f0np0`) |
| `NCCL_IB_GID_INDEX` | `nccl.<cluster>.ib_gid_index` | RoCE GID index for InfiniBand routing |
| `GLOO_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | Network interface for Gloo (vLLM distributed + PyTorch CPU collectives) |
| `TP_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | vLLM tensor-parallel coordinator interface |
| `MN_IF_NAME` | `nccl.<cluster>.socket_ifname` | Multi-node interface name |
| `UCX_NET_DEVICES` | `nccl.<cluster>.socket_ifname` | UCX communication devices |
| `OMPI_MCA_btl_tcp_if_include` | `nccl.<cluster>.socket_ifname` | OpenMPI TCP interface |
| `VLLM_LOGGING_COLOR` | `1` | Force colored vLLM logs (head node only) |

### Hard-coded defaults

| Env var | Value | Purpose |
|---|---|---|
| `NCCL_IGNORE_CPU_AFFINITY` | `1` | Don't pin NCCL threads to a specific CPU set (needed on DGX Spark) |
| `NCCL_NET` | `IB` | Force NCCL to use the InfiniBand plugin (`IB` = `libnccl-net.so`) |
| `NCCL_DEBUG` | `WARN` | NCCL log level (override with `[env]` for `INFO`/`TRACE` while debugging) |
| `OMP_NUM_THREADS` | `4` | OpenMP thread limit per process |
| `HF_HOME` | `/root/.cache/huggingface` | HuggingFace cache root (volume-mounted from `ssh_user`'s host cache) |

### Engine-specific host IP

| Env var | Engines | Value | Purpose |
|---|---|---|---|
| `VLLM_HOST_IP` | vLLM only | `node.ib_ip` | vLLM's own host-IP for distributed init |
| `SGLANG_WORKER_IP` | SGLang only | `node.ib_ip` | SGLang's worker-IP for distributed init |

### Ray-only (when `distributed_backend = "ray"`)

| Env var | Value | Purpose |
|---|---|---|
| `RAY_ADDRESS` | `<master_ib_ip>:6379` | Ray head address (workers connect here) |
| `RAY_PORT` | `recipe.port` | Ray gRPC port |

### Recipe-level overrides

Anything in `[env]` is appended after all of the above, so a recipe can
override any default (e.g. `NCCL_CUMEM_ENABLE = "1"` to re-enable
CUDA-managed memory).

## Power Management (Home Assistant)

Optional `[home_assistant]` section lets the launcher toggle smart
plugs (or any switch domain) so you can shut down and wake nodes
remotely:

```toml
[home_assistant]
url = "http://homeassistant.local:8123"
token = "long-lived-access-token"
# Seconds to wait between `sudo shutdown now` and HA `turn_off` so
# the OS has time to actually shut down.  Default: 30.
shutdown_delay_seconds = 30
```

Each node can optionally carry an `entity` field for the plug that
controls it:

```toml
[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"

# Optional: extra switches to toggle alongside the nodes.  Useful for
# a network switch that powers the cluster, or a PDU that backs the
# rack.  Toggled in addition to any per-node entities.
extra_entities = ["switch.network_switch_4nodes"]

nodes = [
    { name = "dgx1", entity = "switch.dgx1", ssh_ip = "10.0.0.1", ib_ip = "1.2.3.4", rank = 0 },
    # ...
]
```

Without an `entity` the node is skipped (with a warning) — the SSH
shutdown still happens, but Home Assistant is not consulted.
Cluster-level `extra_entities` are always toggled (no opt-in).

### `power off` flow

```
1. ssh <node>: sudo shutdown now
2. wait shutdown_delay_seconds (default 30s)
3. HA POST /api/services/switch/turn_off
   (entity: switch.dgx1)
```

```bash
sm121-launcher power off --cluster 4nodes-switch
sm121-launcher power off --cluster 4nodes-switch --node dgx1
sm121-launcher -d power off --cluster 4nodes-switch   # dry-run
```

### `power on` flow

`power on` does **not** SSH — the smart plug wakes the machine:

```
1. HA POST /api/services/switch/turn_on
   (entity: switch.dgx1)
```

Requires `[home_assistant]` to be configured (no other way to wake
a node).

```bash
sm121-launcher power on --cluster 4nodes-switch
sm121-launcher power on --cluster 4nodes-switch --node dgx1
```

### `power switch` flow

Switch between clusters: disable nodes/entities unique to `from`,
enable everything in `to`.

```
1. SSH sudo shutdown now on nodes in from but not to
2. Wait shutdown_delay_seconds
3. HA turn_off those nodes + extra_entities unique to from
4. HA turn_on all nodes + extra_entities in to
```

```bash
sm121-launcher power switch --from 4 --to 2   # prefix matching works
```

## Doctor Mode

```bash
cargo run doctor --recipe minimax-m3           # Check node prerequisites
cargo run doctor --recipe minimax-m3 --repair  # Auto-fix issues
```

Checks each node for:
- Docker availability
- NVIDIA GPU (`nvidia-smi`)
- NCCL libraries
- Engine version inside the image (`vllm version` / `sglang version`)
- Ray availability (if backend is "ray")

With `--repair`, generates a Dockerfile that installs missing packages and rebuilds the image with a `.repaired` tag. The repaired image is auto-detected on subsequent `serve` runs.

## Internationalization (i18n)

Log messages are localized via Mozilla Fluent (`.ftl` files in `locales/`).
Auto-detects system locale. Available languages:

| Locale | File | Status |
|--------|------|--------|
| English | `locales/en/main.ftl` | Fallback |
| Russian | `locales/ru/main.ftl` | Full |
| Spanish | `locales/es/main.ftl` | Full |
| Italian | `locales/it/main.ftl` | Full |
| French | `locales/fr/main.ftl` | Full |
| German | `locales/de/main.ftl` | Full |

```bash
# Override locale via environment
LANG=ru_RU.UTF-8 cargo run serve --recipe deepseek-v4-flash
```

## CLI Reference

```text
Usage: sm121-launcher [OPTIONS] <COMMAND>

Commands:
  serve    Launch a recipe on a cluster
  logs     Show container logs
  stop     Stop and remove containers for a recipe
  health   Check cluster health (or fingerprint host keys)
  doctor   Check and repair node prerequisites
  tmux     Manage tmux sessions (alternative to Docker)
  power    Power on/off nodes via Home Assistant
  exec     Run an arbitrary shell command on a cluster or single node

Global flags:
  -c, --config <FILE>    Config file path (default: config.toml)
  -d, --dry-run          Print commands without executing
  -h, --help             Print help
  -V, --version          Print version
```

## Dependencies

| Crate | Purpose |
|---|---|
| `clap` 4.6 + derive | CLI argument parsing |
| `russh` 0.61 + `russh-keys` 0.50-beta | SSH client |
| `tokio` full | Async runtime |
| `toml` 1 + `serde` 1 | Config / recipe deserialization |
| `tracing` + `tracing-subscriber` | Structured logging |
| `anyhow` | Error handling |
| `futures` 0.3 | Concurrent SSH (`join_all`) |
| `sha2` 0.11 + `base64` 0.22 | Host key fingerprinting |
| `reqwest` 0.13 | HTTP client for Home Assistant |
| `dirs` 6 | OS config dir discovery |
| `fluent-i18n` | Mozilla Fluent i18n (locales: en, ru, es, it, fr, de) |