launcher

sm121-launcher

Rust CLI tool for launching vLLM / SGLang containers across a DGX Spark cluster via SSH.

Features

• Launch distributed LLM inference clusters on 2–4 DGX Spark machines
• vLLM and SGLang engines — auto-generates the right distributed flags for each
• Docker (default) or tmux (alternative) runtime
• SSH-based provisioning via russh — no agent required on target nodes
• Cluster health checks, container log tailing, and clean teardown
• Doctor mode — pre-flight checks and auto-repair of broken container images
• Dry-run (-d) to preview every SSH command without execution

Quality Gates

make check            # Run all quality gates (fmt, test, clippy, line count)
make check-fmt        # cargo fmt --check
make check-test       # cargo test
make check-clippy     # cargo clippy with -D warnings
make check-lines      # Ensure no Rust file exceeds 500 lines

Quick Start

# Build
cargo build --release

# Launch a recipe (auto-selects cluster from config.toml)
cargo run serve --recipe glm51                # vLLM — GLM-5.1 NVFP8
cargo run serve --recipe minimax-m3            # SGLang — Minimax-M3 NVFP4

# Launch on a specific cluster
cargo run serve --recipe glm51 --cluster 4nodes-switch

# Dry-run (print SSH commands without executing)
cargo run -- -d serve --recipe minimax-m3

# View cluster logs
cargo run logs --recipe glm51                  # Last 100 lines per node
cargo run logs --recipe minimax-m3 --follow    # Stream head node logs

# Stop all containers for a recipe
cargo run stop --recipe glm51

# Health check
cargo run health                               # All clusters
cargo run health --cluster 2nodes-dac          # Single cluster
cargo run health --fingerprint                 # Discover host keys for config.toml

# Doctor mode — check prerequisites
cargo run doctor --recipe minimax-m3           # Check node prerequisites
cargo run doctor --recipe minimax-m3 --repair  # Auto-fix (build repaired image)

# Tmux runtime (no Docker)
cargo run tmux serve --recipe glm51
cargo run tmux status --recipe glm51
cargo run tmux attach --recipe glm51

Available Recipes

glm51 — vLLM (GLM-5.1 NVFP8)

engine = "vllm"
docker_image = "vllm/vllm-openai:glm51-cu130"
model = "lukealonso/GLM-5.1-NVFP4"
port = 30000
required_nodes = 4
distributed_backend = "mp"

Launches GLM-5.1 with vLLM across 4 nodes. Uses pipeline parallelism (PP=4, TP=1 per node) with the mp distributed backend. Extra flags enable MTP speculative decoding, TRITON_MLA attention, prefix caching, and chunked prefill.

cargo run serve --recipe glm51
cargo run logs --recipe glm51 --follow

minimax-m3 — SGLang (Minimax-M3-v0 NVFP4)

engine = "sglang"
docker_image = "scitrera/dgx-spark-sglang-mm:v0"
model = "sparkarena/Minimax-M3-v0-NVFP4"
port = 30000
required_nodes = 4
tensor_parallel_size = 4
distributed_backend = "mp"
entrypoint = ["sglang", "serve"]

Launches Minimax-M3 with SGLang across 4 nodes. Uses tensor parallelism across all GPUs (TP=4, 1 GPU per node). NVFP4 quantization with the cutlass fp4 gemm backend and triton MoE runner. Uses instanttensor load format for fast model loading.

cargo run serve --recipe minimax-m3
cargo run logs --recipe minimax-m3

Configuration

config.toml

Create a config.toml in the project root (not tracked in git):

ssh_user = "kilork"
ssh_key_path = "/path/to/ssh/key"

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"
nodes = [
    { name = "dgx1", ssh_ip = "192.168.88.14", ib_ip = "192.168.200.15", rank = 0 },
    { name = "dgx2", ssh_ip = "192.168.88.10", ib_ip = "192.168.200.13", rank = 1 },
    { name = "dgx3", ssh_ip = "192.168.88.50", ib_ip = "192.168.200.17", rank = 2 },
    { name = "dgx4", ssh_ip = "192.168.88.51", ib_ip = "192.168.200.19", rank = 3 },
]

[nccl."4nodes-switch"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3
ib_disable = 0

Field	Purpose
ssh_user / ssh_key_path	SSH credential for all nodes
clusters[].master_ib_ip	InfiniBand IP of the head node (rank 0)
nodes[].ssh_ip	SSH reachable IP (usually Ethernet)
nodes[].ib_ip	InfiniBand / RoCE IP for NCCL communication
nodes[].rank	Node rank (0 = head, 1..N = workers)
nodes[].host_key	Optional SHA-256 fingerprint — run cargo run health --fingerprint to discover
nccl."<name>".*	NCCL environment variables for the cluster's IB fabric

Multiple clusters can be defined; --cluster auto picks the first one with an NCCL section.

Creating a recipe

Recipes live in recipes/<name>/recipe.toml (directory name intentionally misspelled).

engine = "vllm"                        # or "sglang" (default: "vllm")
name = "Human Readable Name"
docker_image = "my-image:tag"
model = "org/model-name"
port = 30000
required_nodes = 4                     # optional, omitted = no check
pipeline_parallel_size = 0             # 0 = auto (equals nnodes), vLLM only
tensor_parallel_size = 1               # per-node TP for vLLM, total TP for SGLang
distributed_backend = "mp"             # "mp" (multiprocessing) or "ray"
cluster = "4nodes-switch"              # optional: pin to a specific cluster

entrypoint = ["sglang", "serve"]       # optional, appended after image name

args = [
    "--kv-cache-dtype", "fp8",
    "--max-model-len", "32768",
]

[env]
SOME_VAR = "value"

Engine-specific behavior

Aspect	vLLM	SGLang
Host IP env var	VLLM_HOST_IP	SGLANG_WORKER_IP
Model argument	bare model name	--model-path <model>
TP flag	--tensor-parallel-size 1	--tp <tensor_parallel_size>
PP flag	--pipeline-parallel-size <pp_size>	(not used)
Backend flag	`--distributed-executor-backend mp	ray`
Common flags	--nnodes, --node-rank, --master-addr	same
Port flag	(recipe args)	--port <port> (auto-generated)
Entrypoint	python -m vllm.entrypoints.openai.api_server	recipe-defined (e.g. sglang serve)

Launch Behavior

Docker runtime

For each node, the tool builds a docker run command via SSH:

docker run -d --gpus all --ipc=host --privileged --network=host \
  -e NCCL_IB_DISABLE=<...> -e NCCL_IB_HCA=<...> ... \
  -e SGLANG_WORKER_IP=<node.ib_ip> \       # or VLLM_HOST_IP
  -e <recipe.env>... \
  [-p <port>:<port>] \                       # head node only
  --name sm121_<recipe>_<rank> \
  --label sm121.recipe=<recipe> \
  --label sm121.rank=<rank> \
  <docker_image> \
  <entrypoint>... \                          # e.g. sglang serve
  [--model-path] <model> \                   # engine-specific
  <recipe.args>... \                         # from recipe
  --tp <tensor_parallel_size> \              # engine-specific flags
  --nnodes <nnodes> \
  --node-rank <rank> \
  --master-addr <master_ib_ip>
  [--port <port>]                            # SGLang only

NCCL env vars come from the cluster's [nccl."<name>"] section. All containers use --network=host for direct InfiniBand access.

Tmux runtime

cargo run tmux serve --recipe <name> creates a 3-window tmux session on each node:

Window	Content
0: model	Main inference process (vLLM/SGLang)
1: gpu	nvidia-smi -l 5 GPU monitor
2: shell	Debug shell

For Ray backend, nodes start sequentially (head first with API readiness poll up to 150s). For MP backend, all nodes start in parallel.

Doctor Mode

cargo run doctor --recipe minimax-m3           # Check node prerequisites
cargo run doctor --recipe minimax-m3 --repair  # Auto-fix issues

Checks each node for:

• Docker availability
• NVIDIA GPU (nvidia-smi)
• NCCL libraries
• Engine version inside the image (vllm version / sglang version)
• Ray availability (if backend is "ray")

With --repair, generates a Dockerfile that installs missing packages and rebuilds the image with a .repaired tag. The repaired image is auto-detected on subsequent serve runs.

CLI Reference

Usage: sm121-launcher [OPTIONS] <COMMAND>

Commands:
  serve    Launch a recipe on a cluster
  logs     Show container logs
  stop     Stop and remove containers for a recipe
  health   Check cluster health (or fingerprint host keys)
  doctor   Check and repair node prerequisites
  tmux     Manage tmux sessions (alternative to Docker)

Global flags:
  -c, --config <FILE>    Config file path (default: config.toml)
  -d, --dry-run          Print commands without executing
  -h, --help             Print help
  -V, --version          Print version

Dependencies

Crate	Purpose
clap 4.6 + derive	CLI argument parsing
russh 0.60	SSH client
tokio full	Async runtime
toml 1 + serde 1	Config / recipe deserialization
tracing + tracing-subscriber	Structured logging
anyhow	Error handling
futures 0.3	Concurrent SSH (join_all)
sha2 0.10 + base64 0.22	Host key fingerprinting