launcher

sm121-launcher

Rust CLI tool for launching vLLM/SGLang containers across a DGX Spark cluster via SSH.

Features

• Launch distributed LLM inference clusters with a single command
• Support for vLLM and SGLang engines
• Multi-cluster support with auto-selection
• SSH-based provisioning — no agent required on target nodes
• Docker-in-Docker deployment via SSH exec
• Health checks, log tailing, and clean cluster teardown
• Dry-run mode to preview commands without execution

Quick Start

# Build
cargo build --release

# Launch a recipe with auto-cluster selection
cargo run serve --recipe glm51

# Launch on a specific cluster
cargo run serve --recipe glm51 --cluster 4nodes-switch

# Dry-run (print SSH commands without executing)
cargo run -- -d serve --recipe glm51

# View cluster logs
cargo run logs --recipe glm51                # Last 100 lines per node
cargo run logs --recipe glm51 --follow       # Follow head node logs

# Stop all containers on the cluster
cargo run stop --recipe glm51

# Health check
cargo run health                                     # All clusters
cargo run health --cluster 2nodes-dac               # Single cluster

Configuration

config.toml

ssh_user = "kilork"
ssh_key_path = "/path/to/ssh/key"

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"
nodes = [
    { name = "dgx1.lan", ssh_ip = "192.168.88.14", ib_ip = "192.168.200.15", rank = 0 },
    { name = "dgx2.lan", ssh_ip = "192.168.88.10", ib_ip = "192.168.200.13", rank = 1 },
]

[nccl."4nodes-switch"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3
ib_disable = 0

Recipes

Recipes live in recipies/<name>/recipe.toml (note the directory spelling).

engine = "vllm"
name = "GLM-5.1 NVFP8"
docker_image = "vllm/vllm-openai:glm51-cu130"
model = "lukealonso/GLM-5.1-NVFP4"
port = 30000
required_nodes = 4
pipeline_parallel_size = 0    # 0 = auto (equals nnodes)

args = [
    "--kv-cache-dtype", "fp8",
    "--max-model-len", "32768",
]

[env]
NCCL_DEBUG = "WARN"

Launch Behavior

For each node, the tool builds and executes a docker run command via SSH with:

• Full GPU access (--gpus all --ipc=host --privileged --network=host)
• NCCL env vars from the cluster's [nccl."<name>"] section
• Recipe-defined env vars and args
• Appended multi-node flags: --tensor-parallel-size 1, --pipeline-parallel-size, --nnodes, --node-rank, --master-addr, --distributed-executor-backend mp

--cluster auto selects the first cluster in config with an NCCL section.

Dependencies

Crate	Purpose
clap 4.6	CLI argument parsing (derive)
russh 0.60	SSH client
tokio full	Async runtime
toml 1 + serde 1	Config/recipe deserialization
tracing	Structured logging
anyhow	Error handling