launcher

sm121-launcher

Rust CLI tool for launching vLLM / SGLang containers across a DGX Spark cluster via SSH.

Features

• Launch distributed LLM inference clusters on 2–4 DGX Spark machines
• vLLM and SGLang engines — auto-generates the right distributed flags for each
• Docker (default) or tmux (alternative) runtime
• SSH-based provisioning via russh — no agent required on target nodes
• Cluster health checks, container log tailing, and clean teardown
• Doctor mode — pre-flight checks and auto-repair of broken container images
• Dry-run (-d) to preview every SSH command without execution

Quality Gates

make check            # Run all quality gates (fmt, test, clippy, line count)
make check-fmt        # cargo fmt --check
make check-test       # cargo test
make check-clippy     # cargo clippy with -D warnings
make check-lines      # Ensure no Rust file exceeds 500 lines

Quick Start

# Build
cargo build --release

# Launch a recipe (auto-selects cluster from config.toml)
cargo run serve --recipe glm51                # vLLM — GLM-5.1 NVFP8
cargo run serve --recipe minimax-m3            # SGLang — Minimax-M3 NVFP4

# Launch on a specific cluster
cargo run serve --recipe glm51 --cluster 4nodes-switch

# Dry-run (print SSH commands without executing)
cargo run -- -d serve --recipe minimax-m3

# View cluster logs
cargo run logs --recipe glm51                  # Last 100 lines per node
cargo run logs --recipe minimax-m3 --follow    # Stream head node logs

# Stop all containers for a recipe
cargo run stop --recipe glm51

# Health check
cargo run health                               # All clusters
cargo run health --cluster 2nodes-dac          # Single cluster
cargo run health --fingerprint                 # Discover host keys for config.toml

# Doctor mode — check prerequisites
cargo run doctor --recipe minimax-m3           # Check node prerequisites
cargo run doctor --recipe minimax-m3 --repair  # Auto-fix (build repaired image)

# Tmux runtime (no Docker)
cargo run tmux serve --recipe glm51
cargo run tmux status --recipe glm51
cargo run tmux attach --recipe glm51

Available Recipes

glm51 — vLLM (GLM-5.1 NVFP8)

engine = "vllm"
docker_image = "vllm/vllm-openai:glm51-cu130"
model = "lukealonso/GLM-5.1-NVFP4"
port = 30000
required_nodes = 4
distributed_backend = "mp"

Launches GLM-5.1 with vLLM across 4 nodes. Uses pipeline parallelism (PP=4, TP=1 per node) with the mp distributed backend. Extra flags enable MTP speculative decoding, TRITON_MLA attention, prefix caching, and chunked prefill.

cargo run serve --recipe glm51
cargo run logs --recipe glm51 --follow

minimax-m3 — SGLang (Minimax-M3-v0 NVFP4)

engine = "sglang"
docker_image = "scitrera/dgx-spark-sglang-mm:v0"
model = "sparkarena/Minimax-M3-v0-NVFP4"
port = 30000
required_nodes = 4
tensor_parallel_size = 4
distributed_backend = "mp"
entrypoint = ["sglang", "serve"]

Launches Minimax-M3 with SGLang across 4 nodes. Uses tensor parallelism across all GPUs (TP=4, 1 GPU per node). NVFP4 quantization with the cutlass fp4 gemm backend and triton MoE runner. Uses instanttensor load format for fast model loading.

cargo run serve --recipe minimax-m3
cargo run logs --recipe minimax-m3

Configuration

config.toml

Create a config.toml in the project root (not tracked in git):

ssh_user = "kilork"
ssh_key_path = "/path/to/ssh/key"

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"
nodes = [
    { name = "dgx1", ssh_ip = "192.168.88.14", ib_ip = "192.168.200.15", rank = 0 },
    { name = "dgx2", ssh_ip = "192.168.88.10", ib_ip = "192.168.200.13", rank = 1 },
    { name = "dgx3", ssh_ip = "192.168.88.50", ib_ip = "192.168.200.17", rank = 2 },
    { name = "dgx4", ssh_ip = "192.168.88.51", ib_ip = "192.168.200.19", rank = 3 },
]

[nccl."4nodes-switch"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3
ib_disable = 0

Field	Purpose
ssh_user / ssh_key_path	SSH credential for all nodes
clusters[].master_ib_ip	InfiniBand IP of the head node (rank 0)
nodes[].ssh_ip	SSH reachable IP (usually Ethernet)
nodes[].ib_ip	InfiniBand / RoCE IP for NCCL communication
nodes[].rank	Node rank (0 = head, 1..N = workers)
nodes[].host_key	Optional SHA-256 fingerprint — run cargo run health --fingerprint to discover
nccl."<name>".*	NCCL environment variables for the cluster's IB fabric

Multiple clusters can be defined; --cluster auto picks the first one with an NCCL section.

Creating a recipe

Recipes live in recipes/<name>/recipe.toml (directory name intentionally misspelled).

engine = "vllm"                        # or "sglang" (default: "vllm")
name = "Human Readable Name"
docker_image = "my-image:tag"
model = "org/model-name"
port = 30000
required_nodes = 4                     # optional, omitted = no check
pipeline_parallel_size = 0             # 0 = auto (equals nnodes), vLLM only
tensor_parallel_size = 1               # per-node TP for vLLM, total TP for SGLang
distributed_backend = "mp"             # "mp" (multiprocessing) or "ray"
cluster = "4nodes-switch"              # optional: pin to a specific cluster

entrypoint = ["sglang", "serve"]       # optional, appended after image name

args = [
    "--kv-cache-dtype", "fp8",
    "--max-model-len", "32768",
]

[env]
SOME_VAR = "value"

Engine-specific behavior

Aspect	vLLM	SGLang
Host IP env var	VLLM_HOST_IP	SGLANG_WORKER_IP
Model argument	bare model name	--model-path <model>
TP flag	--tensor-parallel-size 1	--tp <tensor_parallel_size>
PP flag	--pipeline-parallel-size <pp_size>	(not used)
Backend flag	`--distributed-executor-backend mp	ray`
Common flags	--nnodes, --node-rank, --master-addr	same
Port flag	(recipe args)	--port <port> (auto-generated)
Entrypoint	python -m vllm.entrypoints.openai.api_server	recipe-defined (e.g. sglang serve)

Launch Behavior

Docker runtime

For each node, the tool builds a docker run command via SSH:

docker run -d --gpus all --ipc=host --privileged --network=host \
  -e NCCL_IB_DISABLE=<...> -e NCCL_IB_HCA=<...> ... \
  -e SGLANG_WORKER_IP=<node.ib_ip> \       # or VLLM_HOST_IP
  -e <recipe.env>... \
  [-p <port>:<port>] \                       # head node only
  --name sm121_<recipe>_<rank> \
  --label sm121.recipe=<recipe> \
  --label sm121.rank=<rank> \
  <docker_image> \
  <entrypoint>... \                          # e.g. sglang serve
  [--model-path] <model> \                   # engine-specific
  <recipe.args>... \                         # from recipe
  --tp <tensor_parallel_size> \              # engine-specific flags
  --nnodes <nnodes> \
  --node-rank <rank> \
  --master-addr <master_ib_ip>
  [--port <port>]                            # SGLang only

NCCL env vars come from the cluster's [nccl."<name>"] section. All containers use --network=host for direct InfiniBand access.

Tmux runtime

cargo run tmux serve --recipe <name> creates a 3-window tmux session on each node:

Window	Content
0: model	Main inference process (vLLM/SGLang)
1: gpu	nvidia-smi -l 5 GPU monitor
2: shell	Debug shell

For Ray backend, nodes start sequentially (head first with API readiness poll up to 150s). For MP backend, all nodes start in parallel.

Environment Variables

The launcher passes the following env vars to every container. Recipe-level [env] entries are appended after the defaults and can override them.

Cluster-driven (from [nccl."<cluster>"])

Env var	Source	Purpose
NCCL_IB_DISABLE	nccl.<cluster>.ib_disable	Disable NCCL's InfiniBand path (0 = use IB, 1 = disable)
NCCL_IB_HCA	nccl.<cluster>.ib_hca	RDMA devices for NCCL (comma-separated, e.g. rocep1s0f0,roceP2p1s0f0)
NCCL_SOCKET_IFNAME	nccl.<cluster>.socket_ifname	Network interface NCCL uses for socket-based control (e.g. enp1s0f0np0)
NCCL_IB_GID_INDEX	nccl.<cluster>.ib_gid_index	RoCE GID index for InfiniBand routing
GLOO_SOCKET_IFNAME	nccl.<cluster>.socket_ifname	Network interface for Gloo (vLLM distributed + PyTorch CPU collectives)
TP_SOCKET_IFNAME	nccl.<cluster>.socket_ifname	vLLM tensor-parallel coordinator interface

Hard-coded defaults

Env var	Value	Purpose
NCCL_CUMEM_ENABLE	0	Disable CUDA-managed memory in NCCL (use system allocator)
NCCL_IGNORE_CPU_AFFINITY	1	Don't pin NCCL threads to a specific CPU set (needed on DGX Spark)
NCCL_NET	IB	Force NCCL to use the InfiniBand plugin (IB = libnccl-net.so)
NCCL_CROSS_NIC	1	Allow traffic across multiple RDMA NICs (multi-rail)
NCCL_DEBUG	WARN	NCCL log level (override with [env] for INFO/TRACE while debugging)
OMP_NUM_THREADS	4	OpenMP thread limit per process
HF_HOME	/root/.cache/huggingface	HuggingFace cache root (volume-mounted from ssh_user's host cache)

Engine-specific host IP

Env var	Engines	Value	Purpose
VLLM_HOST_IP	vLLM only	node.ib_ip	vLLM's own host-IP for distributed init
SGLANG_WORKER_IP	SGLang only	node.ib_ip	SGLang's worker-IP for distributed init

Ray-only (when distributed_backend = "ray")

Env var	Value	Purpose
RAY_ADDRESS	<master_ib_ip>:6379	Ray head address (workers connect here)
RAY_PORT	recipe.port	Ray gRPC port

Recipe-level overrides

Anything in [env] is appended after all of the above, so a recipe can override any default (e.g. NCCL_CUMEM_ENABLE = "1" to re-enable CUDA-managed memory).

Power Management (Home Assistant)

Optional [home_assistant] section lets the launcher toggle smart plugs (or any switch domain) so you can shut down and wake nodes remotely:

[home_assistant]
url = "http://homeassistant.local:8123"
token = "long-lived-access-token"
# Seconds to wait between `sudo shutdown now` and HA `turn_off` so
# the OS has time to actually shut down.  Default: 30.
shutdown_delay_seconds = 30

Each node can optionally carry an entity field for the plug that controls it:

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"

# Optional: extra switches to toggle alongside the nodes.  Useful for
# a network switch that powers the cluster, or a PDU that backs the
# rack.  Toggled in addition to any per-node entities.
extra_entities = ["switch.network_switch_4nodes"]

nodes = [
    { name = "dgx1", entity = "switch.dgx1", ssh_ip = "10.0.0.1", ib_ip = "1.2.3.4", rank = 0 },
    # ...
]

Without an entity the node is skipped (with a warning) — the SSH shutdown still happens, but Home Assistant is not consulted. Cluster-level extra_entities are always toggled (no opt-in).

power off flow

1. ssh <node>: sudo shutdown now
2. wait shutdown_delay_seconds (default 30s)
3. HA POST /api/services/switch/turn_off
   (entity: switch.dgx1)

sm121-launcher power off --cluster 4nodes-switch
sm121-launcher power off --cluster 4nodes-switch --node dgx1
sm121-launcher -d power off --cluster 4nodes-switch   # dry-run

power on flow

power on does not SSH — the smart plug wakes the machine:

1. HA POST /api/services/switch/turn_on
   (entity: switch.dgx1)

Requires [home_assistant] to be configured (no other way to wake a node).

sm121-launcher power on --cluster 4nodes-switch
sm121-launcher power on --cluster 4nodes-switch --node dgx1

Doctor Mode

cargo run doctor --recipe minimax-m3           # Check node prerequisites
cargo run doctor --recipe minimax-m3 --repair  # Auto-fix issues

Checks each node for:

• Docker availability
• NVIDIA GPU (nvidia-smi)
• NCCL libraries
• Engine version inside the image (vllm version / sglang version)
• Ray availability (if backend is "ray")

With --repair, generates a Dockerfile that installs missing packages and rebuilds the image with a .repaired tag. The repaired image is auto-detected on subsequent serve runs.

CLI Reference

Usage: sm121-launcher [OPTIONS] <COMMAND>

Commands:
  serve    Launch a recipe on a cluster
  logs     Show container logs
  stop     Stop and remove containers for a recipe
  health   Check cluster health (or fingerprint host keys)
  doctor   Check and repair node prerequisites
  tmux     Manage tmux sessions (alternative to Docker)
  power    Power on/off nodes via Home Assistant
  exec     Run an arbitrary shell command on a cluster or single node

Global flags:
  -c, --config <FILE>    Config file path (default: config.toml)
  -d, --dry-run          Print commands without executing
  -h, --help             Print help
  -V, --version          Print version

Dependencies

Crate	Purpose
clap 4.6 + derive	CLI argument parsing
russh 0.60	SSH client
tokio full	Async runtime
toml 1 + serde 1	Config / recipe deserialization
tracing + tracing-subscriber	Structured logging
anyhow	Error handling
futures 0.3	Concurrent SSH (join_all)
sha2 0.10 + base64 0.22	Host key fingerprinting