launcher

sm121-launcher

Launch vLLM / SGLang inference clusters across DGX Spark machines via SSH.

Quick Install

# From source
cargo install --path .

# Or download a pre-built binary (requires Rust toolchain to build)
cargo build --release
# Binary at target/release/sm121-launcher

Quick Start

# Create config.toml (see Configuration below)
# Launch a recipe
sm121-launcher serve --recipe deepseek-v4-flash

# Follow logs
sm121-launcher logs --recipe deepseek-v4-flash --follow

# Stop containers
sm121-launcher stop --recipe deepseek-v4-flash

Commands

Command	Description
serve --recipe <name>	Launch recipe on cluster
logs --recipe <name> [--follow]	Show container logs
stop --recipe <name>	Stop and remove containers
health [--cluster <name>]	Check cluster health
doctor --recipe <name> [--repair]	Check/repair node prerequisites
exec --cluster <name> -- <command>	Run command on cluster nodes
exec --cluster <name> --alias <name>	Run predefined alias
power off --cluster <name> [--node <name>]	Shutdown + HA turn_off
power on --cluster <name> [--node <name>]	HA turn_on (wake nodes)
power switch --from <src> --to <tgt>	Switch clusters
tmux serve --recipe <name>	Launch via tmux (no Docker)
tmux status --recipe <name>	Check tmux session health
tmux logs --recipe <name>	Capture tmux pane output
init-host-keys	Auto-write SSH host keys
-d / --dry-run	Preview commands without executing

Cluster selection

sm121-launcher serve --recipe deepseek-v4-flash              # auto (smallest match)
sm121-launcher serve --recipe glm51 --cluster 4nodes-switch  # exact name
sm121-launcher serve --recipe glm51 --cluster 4             # prefix match
sm121-launcher serve --recipe glm51 --cluster 2             # picks 2nodes-dac

Prefix must be unambiguous — error if multiple clusters match.

Recipes

Recipes are TOML files in recipes/<name>/recipe.toml. The launcher looks for them in the current directory first, then in ~/.config/sm121-launcher/recipes/<name>/recipe.toml.

Available

Recipe	Engine	Nodes	Model
deepseek-v4-flash	vLLM	2	DeepSeek V4 Flash (MXFP8, 1M ctx, MTP)
minimax-m3-mxfp8	vLLM	4	MiniMax M3 (MXFP8, 428B, MSA sparse attn)
glm51	vLLM	4	GLM-5.1 NVFP8
minimax-m3	SGLang	4	MiniMax-M3-v0 NVFP4
test-sleep	vLLM	any	Heartbeat test container

Creating a recipe

engine = "vllm"                        # or "sglang" (default: "vllm")
name = "My Model"
docker_image = "my-image:tag"
model = "org/model-name"
required_nodes = 2                     # optional
tensor_parallel_size = 2               # TP across nodes
pipeline_parallel_size = 1             # vLLM only, 0 = auto (equals nnodes)
distributed_backend = "mp"             # "mp" (default) or "ray"
shm_size = "64gb"                      # optional, only added when set

entrypoint = ["vllm", "serve"]         # optional, appended after image

args = [
    "--kv-cache-dtype", "fp8",
    "--max-model-len", "32768",
]

before_serve = [                       # recipe-level pre-serve hooks
    { command = "echo ready", on_failure = "abort" },
]

[env]
VLLM_LOGGING_COLOR = "1"              # override any launcher default

Configuration

The launcher reads config.toml from the current directory, falling back to ~/.config/sm121-launcher/config.toml.

Minimal

ssh_user = "kilork"
ssh_key_path = "/path/to/ssh/key"

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"
nodes = [
    { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0 },
    { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1 },
    { name = "dgx3", ssh_ip = "10.0.0.3", ib_ip = "192.168.200.17", rank = 2 },
    { name = "dgx4", ssh_ip = "10.0.0.4", ib_ip = "192.168.200.19", rank = 3 },
]

With Home Assistant (power management)

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"
extra_entities = ["switch.network_switch"]   # cluster-level switches
nodes = [
    { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0, entity = "switch.dgx1" },
    { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1, entity = "switch.dgx2" },
    { name = "dgx3", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.17", rank = 2, entity = "switch.dgx3" },
    { name = "dgx4", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.19", rank = 3, entity = "switch.dgx4" },
]

[nccl."4nodes-switch"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3
ib_disable = 0

[home_assistant]
url = "http://homeassistant.local:8123"
token = "long-lived-access-token"
shutdown_delay_seconds = 30

[exec_aliases]
gpu-lock = "sudo nvidia-smi -lgc 300,2300"
shutdown = "sudo shutdown now"

[before_serve."4nodes-switch"]
commands = [
    "sudo nvidia-smi -lgc 300,2300",
    { command = "sudo nvidia-smi -pm 1", on_failure = "warn" },
]

NCCL config

Required for each cluster. Controls InfiniBand/RoCE communication:

Field	Example	Purpose
ib_hca	rocep1s0f0,roceP2p1s0f0	RDMA devices
socket_ifname	enp1s0f0np0	Network interface
ib_gid_index	3	RoCE GID index
ib_disable	0	Keep IB enabled

Power Management

Requires [home_assistant] in config.toml.

# Graceful shutdown + wait + turn off smart plugs
sm121-launcher power off --cluster 4nodes-switch

# Wake nodes via smart plugs
sm121-launcher power on --cluster 4nodes-switch

# Switch clusters (disable 4 → enable 2)
sm121-launcher power switch --from 4 --to 2

# Single node
sm121-launcher power off --cluster 4nodes-switch --node dgx1

# Dry-run
sm121-launcher -d power off --cluster 4nodes-switch

Power-off flow: SSH sudo shutdown now → wait shutdown_delay_seconds → HA turn_off entity → HA turn_off cluster extras.

Power-on flow: HA turn_on (no SSH, assumes smart plug wakes the machine).

Switch flow: disable nodes/extras only in source → enable all in target.

Environment Variables

The launcher sets these automatically. Recipe [env] overrides any.

Auto-set (always)

Env var	Source	Description
NCCL_IB_DISABLE	nccl.<name>.ib_disable	InfiniBand on/off
NCCL_IB_HCA	nccl.<name>.ib_hca	RDMA devices
NCCL_SOCKET_IFNAME	nccl.<name>.socket_ifname	NCCL control interface
GLOO_SOCKET_IFNAME	same	Gloo (PyTorch) interface
TP_SOCKET_IFNAME	same	vLLM TP coordinator
MN_IF_NAME	same	Multi-node interface
UCX_NET_DEVICES	same	UCX devices
OMPI_MCA_btl_tcp_if_include	same	OpenMPI TCP interface
NCCL_IB_GID_INDEX	nccl.<name>.ib_gid_index	RoCE GID
VLLM_HOST_IP / SGLANG_WORKER_IP	node.ib_ip	Engine host IP
HF_HOME	/root/.cache/huggingface	HF cache (volume-mounted)
OMP_NUM_THREADS	4	OpenMP threads
NCCL_IGNORE_CPU_AFFINITY	1	Don't pin NCCL threads
NCCL_NET	IB	NCCL IB plugin
NCCL_DEBUG	WARN	NCCL log level
VLLM_LOGGING_COLOR	1	Colored logs (head only)
TERM	xterm-256color	Terminal type (head only)

Ray backend (when distributed_backend = "ray")

RAY_ADDRESS, RAY_PORT, RAY_NODE_IP_ADDRESS, RAY_OVERRIDE_NODE_IP_ADDRESS

Internationalization

Messages are localized. Auto-detects system locale.

# Override locale (works on both macOS and Linux)
LANG=ru_RU.UTF-8 sm121-launcher serve --recipe deepseek-v4-flash

Available: English (fallback), Russian, Spanish, Italian, French, German.

Project Links

• Development docs: DEVELOPMENT.md
• Agent reference: AGENTS.md