# sm121

Launch vLLM / SGLang inference clusters across a DGX Spark fleet over SSH.
Docker is the default deployment target; tmux is a first-class alternative
(`deploy = "tmux"` in the recipe).

## Install

```bash
cargo install --path .
# or: cargo build --release   → target/release/sm121
```

## Quick start

```bash
sm121 config init-global            # create the global config if missing
sm121 -d serve --recipe deepseek-v4-flash   # dry-run: print the commands
sm121 serve --recipe deepseek-v4-flash      # launch (waits for the engine)
sm121 logs --recipe deepseek-v4-flash --follow
sm121 bench --recipe deepseek-v4-flash --label smoke   # tool-eval-bench
sm121 stop --recipe deepseek-v4-flash  # stops ALL nodes, verifies each one
sm121 health                          # cluster-level health (nodes + switches)
```

`-d`/`--dry-run` and `-c/--config <path>` are top-level flags:
`sm121 -d serve ...`, `sm121 -c /other/config.toml serve ...`.

## Commands

| Command | Description |
|---------|-------------|
| `serve --recipe <name>` | Launch; records the run in the ledger, waits for the engine (`/v1/models`, up to 30 min; `--no-wait` skips) |
| `stop --recipe <name>` | Stop **all** nodes in parallel, verify each (60 s poll); `--no-wait` opts out |
| `logs --recipe <name> [--follow]` | Container logs; follow = liveness watch (30 s probes, 2 misses = died) |
| `bench --recipe <name>` | `tool-eval-bench` against a live serve; reports auto-land in `recipes/<lane>/reports/` (+ run package: recipe copy + manifest) |
| `report [--serve]` | Table of tested recipes; `--serve` = interactive HTML (SSE, in-TOML navigation) |
| `health [--cluster <name>]` | Cluster-level health: UP only when all nodes reachable **and** blocking extras (HA switches, MikroTik routers) are on |
| `doctor --recipe <name> [--repair]` | Check/repair node prerequisites |
| `exec --cluster <name> -- <cmd>` | Run a shell command on cluster nodes; `--alias <name>` runs a predefined one |
| `build --recipe <name> --build-node <n>` | Build the recipe's image from `docker-images/<dir>/` on a node (clone-at-pin) |
| `publish --recipe <name> [--push]` | Self-contained recipe+image bundle; `--push` also pushes the image under a versioned tag |
| `registry` | Browse/install recipes from the public registry |
| `runlock list / close <recipe>` | Offline run-ledger management (per-recipe `.lock`) |
| `web` | Local dashboard (axum + SSE): recipes, topology map, LED board, jobs, serve/stop jobs |
| `power off/on/switch` | Power management via Home Assistant + MikroTik RouterOS (see below) |
| `tmux serve/status/logs` | tmux deployment target (no Docker) |
| `init-host-keys` | Auto-discover SSH host keys into config |
| `config path / init-global` | Config resolution helpers |
| `-d` / `--dry-run` | Print instead of execute (all commands) |

### `serve` flags (cluster selection & overrides)

```bash
sm121 serve --recipe R --cluster auto            # smallest cluster that fits
sm121 serve --recipe R --cluster 4               # exact name, prefix, or auto
sm121 serve --recipe R --solo                    # first free node, single-node run
sm121 serve --recipe R --force                   # skip occupancy check (co-serve)
sm121 serve --recipe R --fit=2                   # scale to N nodes, TP tracks
sm121 serve --recipe R --mirror                  # per-node local HF mirror (no NAS at load)
sm121 serve --recipe R --hf-cache ~/hf-local     # override the host-side HF cache dir
sm121 serve --recipe R --clear                   # rm -rf host extra_mounts dirs first
sm121 serve --recipe R --build-node dgx1         # build the image first (if stale)
sm121 serve --recipe R --sync-image              # push the image to launch nodes first
```

Cluster selection: exact name, unambiguous prefix (`4` → `4nodes-switch`), or
`auto` (smallest cluster with NCCL config + matching `required_nodes`).

## Recipes

Recipes are TOML: `recipes/<name>/recipe.toml`, resolved from `./recipes`
first, then each `recipe_sources` entry in the config. This repo is the
source of truth: a recipe and its image build context (`docker-images/<dir>/`)
are versioned together — the served image must match the recipe revision.
`sm121 publish` flattens both into a self-contained bundle; the public
registry <https://git.kilork.org/sm121/recipes> stores those bundles only.

### Core fields

```toml
engine = "vllm"                        # or "sglang"
name = "My Model"                      # display name
docker_image = "jovian-judgement:r35-glm53"
docker_image_dir = "jovian-judgement"  # build dir under docker-images/ (default: recipe name)
model = "org/model-name"
parent_model = "org/base-model"        # optional; report grouping
port = 30000
required_nodes = 2                     # optional; also caps the launch set
solo = false                           # optional; single-node on first free node
deploy = "docker"                      # or "tmux"
tensor_parallel_size = 2
pipeline_parallel_size = 1             # vLLM only, 0 = auto
distributed_backend = "mp"             # "mp" (default) or "ray"
shm_size = "64gb"                      # optional
entrypoint = ["vllm", "serve"]         # optional; used by docker and tmux

args = ["--kv-cache-dtype", "fp8"]     # engine args, passed through verbatim

[env]
KEY = "value"                          # wins over launcher defaults (last -e wins)
PORT = { from = "port" }               # value resolved from a recipe field at serve time

before_serve = [
    "sudo nvidia-smi -lgc",            # bare string = raw command
    { command = "echo hi", on_failure = "warn" },
    { script = "stop_dgx_services" },  # built-in host script alias (embedded in binary)
]
after_stop = [{ script = "restore_dgx_services" }]
```

Full spec — `extra_mounts` (aliases, HF `repo` path refs, `local_files`
hardlinks), `composed_model`, `[mods]` (in-container patches before the
engine), `venv`/`venv_setup` (tmux), `build_profile` + `[build.versions]`
(image pin overrides) — is in [`AGENTS.md`](AGENTS.md#recipe-format-recipesnamerecipetoml).

## Image builds

A recipe's `docker_image` is built **on demand** (`build` or
`serve --build-node`): the image dir is uploaded to the node, built in a tmux
session, and stamped with a `sm121.checksum` label (recursive SHA-256 of the
dir); later serves skip the build when the label matches. Builds are
clone-at-pin: every input is a pinned `KEY=VAL` in `versions.env`, staged by
`fetch-deps.sh` and verified against the committed `locks/deps.lock`
(commit SHAs / gitlink SHAs / blob fingerprint) — any drift aborts the build.

Pins can be overridden without editing the image dir: a `build_profile`
references `profiles/<name>.env` in the image dir; precedence:
`versions.env` < profile < recipe `[build.versions]`.

With `[registry]` in the config, a successful build pushes the image under a
versioned tag (`{base}/{image}:<checksum>`, + `:latest`); a local publication
ledger skips duplicate pushes unless `--force`. `publish --push` pushes
without a build. See [`AGENTS.md`](AGENTS.md#image-registry-pushregistry).

## Configuration

Config resolution: `-c/--config <path>` > `./config.toml` > the global
config (`~/.config/sm121/config.toml` on Linux,
`~/Library/Application Support/sm121/config.toml` on macOS —
`sm121 config path`). The global config exists so `sm121` works from any
cwd; it may carry no clusters of its own — `cluster_sources` files provide
them (merged by name, file wins).

```toml
ssh_user = "kilork"
ssh_key_path = "/path/to/key"

recipe_sources = ["/path/to/more/recipes"]    # extra recipe dirs (after ./recipes)
cluster_sources = ["/path/to/clusters.toml"]  # extra cluster defs

[[clusters]]
name = "4nodes-switch"
master_ib_ip = "192.168.200.15"               # optional; default: rank-0 node's ib_ip
nodes = [
    { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0, entity = "switch.dgx1" },
    { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1, entity = "switch.dgx2" },
]
extra_entities = [
    "switch.network",                          # bare string = blocking (off → cluster down)
    { entity = "switch.fan", blocks = false }, # toggled + shown, never gates up/down
]

[nccl."4nodes-switch"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3          # optional: omit → auto-detect RoCE-v2 IPv4 GID per node at serve
ib_disable = 0

[home_assistant]
url = "http://ha.lan:8123"
token = "long-lived-access-token"
shutdown_delay_seconds = 30

[mikrotik]                 # RouterOS devices probed via API (health) + graceful shutdown
address = "192.168.88.39"
name = "SW2-MUC"           # must match a cluster entity name
username = "sm121"
password = "..."
on_entity = "switch.sw6_mt"   # optional HA relay used for power on; absent → manual reminder

[registry]                 # optional: image push target (see Image builds above)
base = "ghcr.io/kilork"

[exec_aliases]
gpu-lock = "sudo nvidia-smi -lgc 300,2300"

[mount_aliases]            # host paths by name for recipe [[extra_mounts]]
jit_cache = "/mnt/nas/ai/jit"

[before_serve."*"]         # global: every cluster; per-cluster key supplements
commands = ["sudo nvidia-smi -lgc 0,2000"]
```

## Power management

`power off`: SSH `sudo shutdown now` → wait `shutdown_delay_seconds` →
HA `turn_off` per-node entities + cluster extras; MikroTik devices get a
graceful RouterOS `system shutdown` instead of a plain relay cut (and are
probed by `health` via a RouterOS API login — a dead router takes the
cluster down like any dead switch). `power on`: HA `turn_on` (no SSH;
MikroTik powers up through its `on_entity` relay, or a manual reminder).
`power switch --from A --to B`: disable only what `A` has, enable all of `B`.

> ⚠️ A cluster may be serving live traffic (possibly the model serving this
> very session). Always `health` and confirm before `stop` / `power off`.

## Web dashboard

`sm121 web` — local axum + SSE dashboard: the recipes panel (the `report
--serve` table, with run-ledger state and in-TOML navigation), a physical
topology map (grid, 200G/10G fabric, HA glyphs, cluster frames), a
cluster-level LED board, and serve/stop/power jobs with live status.

## Internationalization

Messages are localized (Fluent FTL in `locales/{en,ru,es,it,fr,de}/`),
auto-detected from the system locale:

```bash
LANG=ru_RU.UTF-8 sm121 serve --recipe deepseek-v4-flash
```

## Further reading

- **[`AGENTS.md`](AGENTS.md)** — full operational reference: recipe format,
  image builds/pins, env vars, mods, composed models, registry, NCCL
  auto-detect, power, gotchas.
- **[`DEVELOPMENT.md`](DEVELOPMENT.md)** — architecture, module layout, dev workflow.
- **`specs/tech-architecture/`** — measured lane states and findings.
- **`history/night-shifts/`** — shift logs.