# sm121-launcher

Rust CLI tool for launching vLLM / SGLang containers across a DGX Spark cluster via SSH.

## Features

- Launch distributed LLM inference clusters on 2–4 DGX Spark machines
- **vLLM** and **SGLang** engines — auto-generates the right distributed flags for each
- Docker (default) or **tmux** (alternative) runtime
- SSH-based provisioning via `russh` — no agent required on target nodes
- Cluster health checks, container log tailing, and clean teardown
- **Doctor mode** — pre-flight checks and auto-repair of broken container images
- Dry-run (`-d`) to preview every SSH command without execution

## Quality Gates

```bash
make check            # Run all quality gates (fmt, test, clippy, line count)
make check-fmt        # cargo fmt --check
make check-test       # cargo test
make check-clippy     # cargo clippy with -D warnings
make check-lines      # Ensure no Rust file exceeds 500 lines
```

## Quick Start

```bash
# Build
cargo build --release

# Launch a recipe (auto-selects cluster from config.toml)
cargo run serve --recipe glm51                # vLLM — GLM-5.1 NVFP8
cargo run serve --recipe minimax-m3            # SGLang — Minimax-M3 NVFP4

# Launch on a specific cluster
cargo run serve --recipe glm51 --cluster 4nodes-switch

# Dry-run (print SSH commands without executing)
cargo run -- -d serve --recipe minimax-m3

# View cluster logs
cargo run logs --recipe glm51                  # Last 100 lines per node
cargo run logs --recipe minimax-m3 --follow    # Stream head node logs

# Stop all containers for a recipe
cargo run stop --recipe glm51

# Health check
cargo run health                               # All clusters
cargo run health --cluster 2nodes-dac          # Single cluster
cargo run health --fingerprint                 # Discover host keys for config.toml

# Doctor mode — check prerequisites
cargo run doctor --recipe minimax-m3           # Check node prerequisites
cargo run doctor --recipe minimax-m3 --repair  # Auto-fix (build repaired image)

# Tmux runtime (no Docker)
cargo run tmux serve --recipe glm51
cargo run tmux status --recipe glm51
cargo run tmux attach --recipe glm51
```

## Available Recipes

### [`glm51`](recipes/glm51/recipe.toml) — vLLM (GLM-5.1 NVFP8)

```toml
engine = "vllm"
docker_image = "vllm/vllm-openai:glm51-cu130"
model = "lukealonso/GLM-5.1-NVFP4"
port = 30000
required_nodes = 4
distributed_backend = "mp"
```

Launches GLM-5.1 with vLLM across 4 nodes. Uses pipeline parallelism (PP=4, TP=1 per node) with the `mp` distributed backend. Extra flags enable MTP speculative decoding, TRITON_MLA attention, prefix caching, and chunked prefill.

```bash
cargo run serve --recipe glm51
cargo run logs --recipe glm51 --follow
```

### [`minimax-m3`](recipes/minimax-m3/recipe.toml) — SGLang (Minimax-M3-v0 NVFP4)

```toml
engine = "sglang"
docker_image = "scitrera/dgx-spark-sglang-mm:v0"
model = "sparkarena/Minimax-M3-v0-NVFP4"
port = 30000
required_nodes = 4
tensor_parallel_size = 4
distributed_backend = "mp"
entrypoint = ["sglang", "serve"]
```

Launches Minimax-M3 with SGLang across 4 nodes. Uses tensor parallelism across all GPUs (TP=4, 1 GPU per node). NVFP4 quantization with the `cutlass` fp4 gemm backend and `triton` MoE runner. Uses `instanttensor` load format for fast model loading.

```bash
cargo run serve --recipe minimax-m3
cargo run logs --recipe minimax-m3
```

## Configuration

### config.toml

Create a `config.toml` in the project root (not tracked in git):

```toml
ssh_user = "kilork"
ssh_key_path = "/path/to/ssh/key"

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"
nodes = [
    { name = "dgx1", ssh_ip = "192.168.88.14", ib_ip = "192.168.200.15", rank = 0 },
    { name = "dgx2", ssh_ip = "192.168.88.10", ib_ip = "192.168.200.13", rank = 1 },
    { name = "dgx3", ssh_ip = "192.168.88.50", ib_ip = "192.168.200.17", rank = 2 },
    { name = "dgx4", ssh_ip = "192.168.88.51", ib_ip = "192.168.200.19", rank = 3 },
]

[nccl."4nodes-switch"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3
ib_disable = 0
```

| Field | Purpose |
|---|---|
| `ssh_user` / `ssh_key_path` | SSH credential for all nodes |
| `clusters[].master_ib_ip` | InfiniBand IP of the head node (rank 0) |
| `nodes[].ssh_ip` | SSH reachable IP (usually Ethernet) |
| `nodes[].ib_ip` | InfiniBand / RoCE IP for NCCL communication |
| `nodes[].rank` | Node rank (0 = head, 1..N = workers) |
| `nodes[].host_key` | Optional SHA-256 fingerprint — run `cargo run health --fingerprint` to discover |
| `nccl."<name>".*` | NCCL environment variables for the cluster's IB fabric |

Multiple clusters can be defined; `--cluster auto` picks the first one with an NCCL section.

### Creating a recipe

Recipes live in `recipes/<name>/recipe.toml` (directory name intentionally misspelled).

```toml
engine = "vllm"                        # or "sglang" (default: "vllm")
name = "Human Readable Name"
docker_image = "my-image:tag"
model = "org/model-name"
port = 30000
required_nodes = 4                     # optional, omitted = no check
pipeline_parallel_size = 0             # 0 = auto (equals nnodes), vLLM only
tensor_parallel_size = 1               # per-node TP for vLLM, total TP for SGLang
distributed_backend = "mp"             # "mp" (multiprocessing) or "ray"
cluster = "4nodes-switch"              # optional: pin to a specific cluster

entrypoint = ["sglang", "serve"]       # optional, appended after image name

args = [
    "--kv-cache-dtype", "fp8",
    "--max-model-len", "32768",
]

[env]
SOME_VAR = "value"
```

#### Engine-specific behavior

| Aspect | vLLM | SGLang |
|---|---|---|
| Host IP env var | `VLLM_HOST_IP` | `SGLANG_WORKER_IP` |
| Model argument | bare model name | `--model-path <model>` |
| TP flag | `--tensor-parallel-size 1` | `--tp <tensor_parallel_size>` |
| PP flag | `--pipeline-parallel-size <pp_size>` | _(not used)_ |
| Backend flag | `--distributed-executor-backend mp|ray` | _(not used)_ |
| Common flags | `--nnodes`, `--node-rank`, `--master-addr` | same |
| Port flag | _(recipe args)_ | `--port <port>` (auto-generated) |
| Entrypoint | `python -m vllm.entrypoints.openai.api_server` | recipe-defined (e.g. `sglang serve`) |

## Launch Behavior

### Docker runtime

For each node, the tool builds a `docker run` command via SSH:

```
docker run -d --gpus all --ipc=host --privileged --network=host \
  -e NCCL_IB_DISABLE=<...> -e NCCL_IB_HCA=<...> ... \
  -e SGLANG_WORKER_IP=<node.ib_ip> \       # or VLLM_HOST_IP
  -e <recipe.env>... \
  [-p <port>:<port>] \                       # head node only
  --name sm121_<recipe>_<rank> \
  --label sm121.recipe=<recipe> \
  --label sm121.rank=<rank> \
  <docker_image> \
  <entrypoint>... \                          # e.g. sglang serve
  [--model-path] <model> \                   # engine-specific
  <recipe.args>... \                         # from recipe
  --tp <tensor_parallel_size> \              # engine-specific flags
  --nnodes <nnodes> \
  --node-rank <rank> \
  --master-addr <master_ib_ip>
  [--port <port>]                            # SGLang only
```

NCCL env vars come from the cluster's `[nccl."<name>"]` section. All containers use `--network=host` for direct InfiniBand access.

### Tmux runtime

`cargo run tmux serve --recipe <name>` creates a 3-window tmux session on each node:

| Window | Content |
|--------|---------|
| 0: `model` | Main inference process (vLLM/SGLang) |
| 1: `gpu` | `nvidia-smi -l 5` GPU monitor |
| 2: `shell` | Debug shell |

For Ray backend, nodes start sequentially (head first with API readiness poll up to 150s). For MP backend, all nodes start in parallel.

## Environment Variables

The launcher passes the following env vars to every container.  Recipe-level
`[env]` entries are appended after the defaults and can override them.

### Cluster-driven (from `[nccl."<cluster>"]`)

| Env var | Source | Purpose |
|---|---|---|
| `NCCL_IB_DISABLE` | `nccl.<cluster>.ib_disable` | Disable NCCL's InfiniBand path (0 = use IB, 1 = disable) |
| `NCCL_IB_HCA` | `nccl.<cluster>.ib_hca` | RDMA devices for NCCL (comma-separated, e.g. `rocep1s0f0,roceP2p1s0f0`) |
| `NCCL_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | Network interface NCCL uses for socket-based control (e.g. `enp1s0f0np0`) |
| `NCCL_IB_GID_INDEX` | `nccl.<cluster>.ib_gid_index` | RoCE GID index for InfiniBand routing |
| `GLOO_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | Network interface for Gloo (vLLM distributed + PyTorch CPU collectives) |
| `TP_SOCKET_IFNAME` | `nccl.<cluster>.socket_ifname` | vLLM tensor-parallel coordinator interface |

### Hard-coded defaults

| Env var | Value | Purpose |
|---|---|---|
| `NCCL_CUMEM_ENABLE` | `0` | Disable CUDA-managed memory in NCCL (use system allocator) |
| `NCCL_IGNORE_CPU_AFFINITY` | `1` | Don't pin NCCL threads to a specific CPU set (needed on DGX Spark) |
| `NCCL_NET` | `IB` | Force NCCL to use the InfiniBand plugin (`IB` = `libnccl-net.so`) |
| `NCCL_CROSS_NIC` | `1` | Allow traffic across multiple RDMA NICs (multi-rail) |
| `NCCL_DEBUG` | `WARN` | NCCL log level (override with `[env]` for `INFO`/`TRACE` while debugging) |
| `OMP_NUM_THREADS` | `4` | OpenMP thread limit per process |
| `HF_HOME` | `/cache/huggingface` | HuggingFace cache root (volume-mounted from `ssh_user`'s host cache) |

### Engine-specific host IP

| Env var | Engines | Value | Purpose |
|---|---|---|---|
| `VLLM_HOST_IP` | vLLM only | `node.ib_ip` | vLLM's own host-IP for distributed init |
| `SGLANG_WORKER_IP` | SGLang only | `node.ib_ip` | SGLang's worker-IP for distributed init |

### Ray-only (when `distributed_backend = "ray"`)

| Env var | Value | Purpose |
|---|---|---|
| `RAY_ADDRESS` | `<master_ib_ip>:6379` | Ray head address (workers connect here) |
| `RAY_PORT` | `recipe.port` | Ray gRPC port |

### Recipe-level overrides

Anything in `[env]` is appended after all of the above, so a recipe can
override any default (e.g. `NCCL_CUMEM_ENABLE = "1"` to re-enable
CUDA-managed memory).

## Doctor Mode

```bash
cargo run doctor --recipe minimax-m3           # Check node prerequisites
cargo run doctor --recipe minimax-m3 --repair  # Auto-fix issues
```

Checks each node for:
- Docker availability
- NVIDIA GPU (`nvidia-smi`)
- NCCL libraries
- Engine version inside the image (`vllm version` / `sglang version`)
- Ray availability (if backend is "ray")

With `--repair`, generates a Dockerfile that installs missing packages and rebuilds the image with a `.repaired` tag. The repaired image is auto-detected on subsequent `serve` runs.

## CLI Reference

```text
Usage: sm121-launcher [OPTIONS] <COMMAND>

Commands:
  serve    Launch a recipe on a cluster
  logs     Show container logs
  stop     Stop and remove containers for a recipe
  health   Check cluster health (or fingerprint host keys)
  doctor   Check and repair node prerequisites
  tmux     Manage tmux sessions (alternative to Docker)

Global flags:
  -c, --config <FILE>    Config file path (default: config.toml)
  -d, --dry-run          Print commands without executing
  -h, --help             Print help
  -V, --version          Print version
```

## Dependencies

| Crate | Purpose |
|---|---|
| `clap` 4.6 + derive | CLI argument parsing |
| `russh` 0.60 | SSH client |
| `tokio` full | Async runtime |
| `toml` 1 + `serde` 1 | Config / recipe deserialization |
| `tracing` + `tracing-subscriber` | Structured logging |
| `anyhow` | Error handling |
| `futures` 0.3 | Concurrent SSH (`join_all`) |
| `sha2` 0.10 + `base64` 0.22 | Host key fingerprinting |