launcher

sm121

Launch vLLM / SGLang inference clusters across a DGX Spark fleet over SSH. Docker is the default deployment target; tmux is a first-class alternative (deploy = "tmux" in the recipe).

Install

cargo install --path .
# or: cargo build --release   → target/release/sm121

Quick start

sm121 config init-global            # create the global config if missing
sm121 -d serve --recipe deepseek-v4-flash   # dry-run: print the commands
sm121 serve --recipe deepseek-v4-flash      # launch (waits for the engine)
sm121 logs --recipe deepseek-v4-flash --follow
sm121 bench --recipe deepseek-v4-flash --label smoke   # tool-eval-bench
sm121 stop --recipe deepseek-v4-flash  # stops ALL nodes, verifies each one
sm121 health                          # cluster-level health (nodes + switches)

-d/--dry-run and -c/--config <path> are top-level flags: sm121 -d serve ..., sm121 -c /other/config.toml serve ....

Commands

Command	Description
serve --recipe <name>	Launch; records the run in the ledger, waits for the engine (/v1/models, up to 30 min; --no-wait skips)
stop --recipe <name>	Stop all nodes in parallel, verify each (60 s poll); --no-wait opts out
logs --recipe <name> [--follow]	Container logs; follow = liveness watch (30 s probes, 2 misses = died)
bench --recipe <name>	tool-eval-bench against a live serve; reports auto-land in recipes/<lane>/reports/ (+ run package: recipe copy + manifest)
report [--serve]	Table of tested recipes; --serve = interactive HTML (SSE, in-TOML navigation)
health [--cluster <name>]	Cluster-level health: UP only when all nodes reachable and blocking extras (HA switches, MikroTik routers) are on
doctor --recipe <name> [--repair]	Check/repair node prerequisites
exec --cluster <name> -- <cmd>	Run a shell command on cluster nodes; --alias <name> runs a predefined one
build --recipe <name> --build-node <n>	Build the recipe's image from docker-images/<dir>/ on a node (clone-at-pin)
publish --recipe <name> [--push]	Self-contained recipe+image bundle; --push also pushes the image under a versioned tag
registry	Browse/install recipes from the public registry
runlock list / close <recipe>	Offline run-ledger management (per-recipe .lock)
web	Local dashboard (axum + SSE): recipes, topology map, LED board, jobs, serve/stop jobs
power off/on/switch	Power management via Home Assistant + MikroTik RouterOS (see below)
tmux serve/status/logs	tmux deployment target (no Docker)
init-host-keys	Auto-discover SSH host keys into config
config path / init-global	Config resolution helpers
-d / --dry-run	Print instead of execute (all commands)

serve flags (cluster selection & overrides)

sm121 serve --recipe R --cluster auto            # smallest cluster that fits
sm121 serve --recipe R --cluster 4               # exact name, prefix, or auto
sm121 serve --recipe R --solo                    # first free node, single-node run
sm121 serve --recipe R --force                   # skip occupancy check (co-serve)
sm121 serve --recipe R --fit=2                   # scale to N nodes, TP tracks
sm121 serve --recipe R --mirror                  # per-node local HF mirror (no NAS at load)
sm121 serve --recipe R --hf-cache ~/hf-local     # override the host-side HF cache dir
sm121 serve --recipe R --clear                   # rm -rf host extra_mounts dirs first
sm121 serve --recipe R --build-node dgx1         # build the image first (if stale)
sm121 serve --recipe R --sync-image              # push the image to launch nodes first

Cluster selection: exact name, unambiguous prefix (4 → 4nodes-switch), or auto (smallest cluster with NCCL config + matching required_nodes).

Recipes

Recipes are TOML: recipes/<name>/recipe.toml, resolved from ./recipes first, then each recipe_sources entry in the config. This repo is the source of truth: a recipe and its image build context (docker-images/<dir>/) are versioned together — the served image must match the recipe revision. sm121 publish flattens both into a self-contained bundle; the public registry https://git.kilork.org/sm121/recipes stores those bundles only.

Core fields

engine = "vllm"                        # or "sglang"
name = "My Model"                      # display name
docker_image = "jovian-judgement:r35-glm53"
docker_image_dir = "jovian-judgement"  # build dir under docker-images/ (default: recipe name)
model = "org/model-name"
parent_model = "org/base-model"        # optional; report grouping
port = 30000
required_nodes = 2                     # optional; also caps the launch set
solo = false                           # optional; single-node on first free node
deploy = "docker"                      # or "tmux"
tensor_parallel_size = 2
pipeline_parallel_size = 1             # vLLM only, 0 = auto
distributed_backend = "mp"             # "mp" (default) or "ray"
shm_size = "64gb"                      # optional
entrypoint = ["vllm", "serve"]         # optional; used by docker and tmux

args = ["--kv-cache-dtype", "fp8"]     # engine args, passed through verbatim

[env]
KEY = "value"                          # wins over launcher defaults (last -e wins)
PORT = { from = "port" }               # value resolved from a recipe field at serve time

before_serve = [
    "sudo nvidia-smi -lgc",            # bare string = raw command
    { command = "echo hi", on_failure = "warn" },
    { script = "stop_dgx_services" },  # built-in host script alias (embedded in binary)
]
after_stop = [{ script = "restore_dgx_services" }]

Full spec — extra_mounts (aliases, HF repo path refs, local_files hardlinks), composed_model, [mods] (in-container patches before the engine), venv/venv_setup (tmux), build_profile + [build.versions] (image pin overrides) — is in AGENTS.md.

Image builds

A recipe's docker_image is built on demand (build or serve --build-node): the image dir is uploaded to the node, built in a tmux session, and stamped with a sm121.checksum label (recursive SHA-256 of the dir); later serves skip the build when the label matches. Builds are clone-at-pin: every input is a pinned KEY=VAL in versions.env, staged by fetch-deps.sh and verified against the committed locks/deps.lock (commit SHAs / gitlink SHAs / blob fingerprint) — any drift aborts the build.

Pins can be overridden without editing the image dir: a build_profile references profiles/<name>.env in the image dir; precedence: versions.env < profile < recipe [build.versions].

With [registry] in the config, a successful build pushes the image under a versioned tag ({base}/{image}:<checksum>, + :latest); a local publication ledger skips duplicate pushes unless --force. publish --push pushes without a build. See AGENTS.md.

Configuration

Config resolution: -c/--config <path> > ./config.toml > the global config (~/.config/sm121/config.toml on Linux, ~/Library/Application Support/sm121/config.toml on macOS — sm121 config path). The global config exists so sm121 works from any cwd; it may carry no clusters of its own — cluster_sources files provide them (merged by name, file wins).

ssh_user = "kilork"
ssh_key_path = "/path/to/key"

recipe_sources = ["/path/to/more/recipes"]    # extra recipe dirs (after ./recipes)
cluster_sources = ["/path/to/clusters.toml"]  # extra cluster defs

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"               # optional; default: rank-0 node's ib_ip
nodes = [
    { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0, entity = "switch.dgx1" },
    { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1, entity = "switch.dgx2" },
]
extra_entities = [
    "switch.network",                          # bare string = blocking (off → cluster down)
    { entity = "switch.fan", blocks = false }, # toggled + shown, never gates up/down
]

[nccl."4nodes-switch"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3          # optional: omit → auto-detect RoCE-v2 IPv4 GID per node at serve
ib_disable = 0

[home_assistant]
url = "http://ha.lan:8123"
token = "long-lived-access-token"
shutdown_delay_seconds = 30

[mikrotik]                 # RouterOS devices probed via API (health) + graceful shutdown
address = "192.168.88.39"
name = "SW2-MUC"           # must match a cluster entity name
username = "sm121"
password = "..."
on_entity = "switch.sw6_mt"   # optional HA relay used for power on; absent → manual reminder

[registry]                 # optional: image push target (see Image builds above)
base = "ghcr.io/kilork"

[exec_aliases]
gpu-lock = "sudo nvidia-smi -lgc 300,2300"

[mount_aliases]            # host paths by name for recipe [[extra_mounts]]
jit_cache = "/mnt/nas/ai/jit"

[before_serve."*"]         # global: every cluster; per-cluster key supplements
commands = ["sudo nvidia-smi -lgc 0,2000"]

Power management

power off: SSH sudo shutdown now → wait shutdown_delay_seconds → HA turn_off per-node entities + cluster extras; MikroTik devices get a graceful RouterOS system shutdown instead of a plain relay cut (and are probed by health via a RouterOS API login — a dead router takes the cluster down like any dead switch). power on: HA turn_on (no SSH; MikroTik powers up through its on_entity relay, or a manual reminder). power switch --from A --to B: disable only what A has, enable all of B.

⚠️ A cluster may be serving live traffic (possibly the model serving this very session). Always health and confirm before stop / power off.

Web dashboard

sm121 web — local axum + SSE dashboard: the recipes panel (the report --serve table, with run-ledger state and in-TOML navigation), a physical topology map (grid, 200G/10G fabric, HA glyphs, cluster frames), a cluster-level LED board, and serve/stop/power jobs with live status.

Internationalization

Messages are localized (Fluent FTL in locales/{en,ru,es,it,fr,de}/), auto-detected from the system locale:

LANG=ru_RU.UTF-8 sm121 serve --recipe deepseek-v4-flash

Further reading

• AGENTS.md — full operational reference: recipe format, image builds/pins, env vars, mods, composed models, registry, NCCL auto-detect, power, gotchas.
• DEVELOPMENT.md — architecture, module layout, dev workflow.
• specs/tech-architecture/ — measured lane states and findings.
• history/night-shifts/ — shift logs.