launcher/DEVELOPMENT.md

Development — sm121-launcher

Rust CLI tool for launching vLLM/SGLang containers across a DGX Spark cluster. SSH-based provisioning over russh.

Project Layout

Cargo.toml                 # Binary crate, Edition 2024
config.toml                # User config (gitignored)
locales/                   # Fluent i18n files (en, ru, es, it, fr, de)
recipes/                   # Recipe directories (<name>/recipe.toml)
src/
├── main.rs                # CLI entrypoint, command dispatch, i18n init
├── cli.rs                 # clap derive CLI definitions
├── config.rs              # Config deserialization, cluster resolution;
│                          #   config/types.rs (Node/Cluster/Config),
│                          #   config/recipe.rs (Recipe + [env] EnvValue refs)
├── ssh.rs                 # SSH client, key loading, host key verification
├── docker/
│   ├── mod.rs             # Docker command builder: serve, stop, health, logs
│   └── tests.rs           # Command builder tests
├── tmux/
│   ├── mod.rs             # Tmux module root
│   ├── serve.rs           # Tmux serve (window/panel layout)
│   └── ops.rs             # Tmux stop/status/logs/attach
├── power.rs               # Power on/off/switch flows
├── home_assistant.rs      # HA REST API client
├── host_keys.rs           # init-host-keys command
├── exec.rs                # Exec subcommand + alias resolution
├── doctor.rs              # Doctor mode + --repair
├── pre_serve.rs           # PreServeCommand types
├── before_serve.rs        # Pre-serve hook execution
├── health.rs              # Tmux health-check helpers
├── recipe_repos.rs        # [[recipe_repos]] git sync (clone/pull into
│                          #   ~/.cache/sm121/recipe-repos/<sha256(url)>),
│                          #   the repos.json archive registry, sm121 repos/
│                          #   archive/unarchive
├── promote.rs             # sm121 promote (cross-repo git mv: target add +
│                          #   push first, source remove + push second)
└── docker/registry_pull.rs  # --image-source registry (pull the prebuilt
                             #   {base}/{name}:{checksum} tag, verify the
                             #   sm121.checksum label)

Quality Gates

make check            # All gates: fmt → clippy → test → line-count
make fmt              # cargo fmt
make check-fmt        # cargo fmt --check
make check-clippy     # cargo clippy --all-targets -- -D warnings
make check-test       # cargo test
make check-lines      # File-size ≤500 lines per src/**/*.rs

Architecture

Docker launch flow

1. serve reads the recipe TOML
2. Resolves cluster (exact/prefix match → smallest with NCCL + matching node count)
3. Runs pre-serve hooks (cluster-level + recipe-level)
4. Checks for repaired image (via doctor --repair)
5. For each node: builds docker run command via build_docker_command(), runs via SSH
6. Waits 10s, then shows tail logs

build_docker_command() in src/docker/mod.rs is engine-aware:

Aspect	vLLM	SGLang
Host IP env var	VLLM_HOST_IP	SGLANG_WORKER_IP
Model argument	bare name	--model-path
TP flag	--tensor-parallel-size	--tp
PP flag	--pipeline-parallel-size	—
Backend flag	--distributed-executor-backend (ray only)	—
Init addr	--master-addr + --master-port	--dist-init-addr
Workers	--headless	—

Auto-generated flags: --host 0.0.0.0, --port <recipe.port>, --nnodes, --node-rank.

Cluster selection

• Exact match: --cluster 4nodes-switch
• Prefix match: --cluster 4 → resolves to 4nodes-switch (unique prefix, fails on ambiguous)
• Auto: --cluster auto (or omitted) → picks the smallest cluster with NCCL config matching required_nodes

Env var precedence

1. Cluster-driven (from [nccl]): NCCL_IB_DISABLE, NCCL_IB_HCA, NCCL_SOCKET_IFNAME, etc.
2. Network interface (from nccl.socket_ifname): MN_IF_NAME, UCX_NET_DEVICES, OMPI_MCA_btl_tcp_if_include
3. Defaults: OMP_NUM_THREADS=4, NCCL_IGNORE_CPU_AFFINITY=1, NCCL_NET=IB, NCCL_DEBUG=WARN
4. Engine IP: VLLM_HOST_IP / SGLANG_WORKER_IP
5. Head node: VLLM_LOGGING_COLOR=1, TERM=xterm-256color, -p <port>:<port>
6. Ray backend: RAY_ADDRESS, RAY_PORT, RAY_NODE_IP_ADDRESS, RAY_OVERRIDE_NODE_IP_ADDRESS
7. Recipe [env]: appended last, overrides everything above

Power management

• power off: SSH sudo shutdown now → wait shutdown_delay_seconds → HA turn_off per-node entities + cluster extras
• power on: HA turn_on per-node entities + cluster extras (no SSH)
• power switch: disable nodes/extras unique to from → enable all in to

i18n

Mozilla Fluent via fluent-i18n crate. Locale auto-detected from system. FTL files in locales/{lang}/main.ftl.

Recipe repos & the HA add-on

Full design: specs/tech-architecture/ha-addon-recipe-repos.md.

• [[recipe_repos]] (ordered list; first = highest priority on a recipe-name collision): git repos holding recipes/ (+ docker-images/), cloned/pulled by recipe_repos::sync_recipe_repos into ~/.cache/sm121/recipe-repos/<sha256(url)16>. Best-effort: an unreachable repo warns + is skipped (a dead playground must not take down production serving). The synced roots are appended to recipe_sources (as their recipes/ dirs) and docker_image_sources (as repo roots) in list order — the existing search-order semantics do the shadowing. Skipped in dry-run (no git I/O).
• Archive registry (repos.json next to the global config): separate from the config, so a repo can be archived without editing it (sm121 archive <url> / unarchive). Missing → fresh; corrupt → .bak + fresh + warning.
• sm121 promote <recipe> --from <url> --to <url> [--apply]: decomposed git mv — target checkout: cp -a + git add/commit/push FIRST; then source: git rm -r + commit + push. A failure in between leaves the recipe duplicated (safe under priority semantics), never deleted. Collision in the target is refused up front. --dry-run (the global -d) prints the exact commands.
• --image-source {repo,registry} (serve/build): repo (default) = the current on-demand build from docker-images/; registry pulls the prebuilt {base}/{name}:{checksum12} from the [registry] base on every launch node, tags it as the recipe's docker_image, and verifies the sm121.checksum label (a missing tag names the tag + points at build/push or the repo fallback). Mutually exclusive with --build-node.
• The add-on image (docker-images/sm121-addon/): multi-stage (sm121 is compiled in-container — the binary always matches the container arch), debian-slim + git + jq, entrypoint writes the effective config from env (the HA add-on options), drops the SSH key (base64 option), sm121 repos sync, then sm121 web --port 3100 --bind 0.0.0.0 with a re-sync supervisor. make addon-image / addon-push; smoke: scripts/smoke-addon.sh. The HA add-on manifest lives in the sibling sm121-ha-integration/ repo (its custom_components/ prototype is superseded — entities are a phase-2 nicety over the add-on's REST API).
• Single-launcher rule: the per-recipe run ledger (.lock in the recipe repo) serializes serving — run ONE launcher as the source of truth (the add-on); the Mac launcher is for dev.

Module size limit

Every src/**/*.rs file must stay ≤500 lines. Enforced by make check-lines.

Dev Workflow

# Build
cargo build

# Run locally with dry-run (no SSH execution)
cargo run -- -d serve --recipe deepseek-v4-flash

# Run tests
cargo test

# All gates (must pass before commit)
make check

# Add a new recipe
mkdir -p recipes/my-model/
# create recipes/my-model/recipe.toml
cargo run -- -d serve --recipe my-model --cluster <name>

# Add a new locale
cp locales/en/main.ftl locales/xx/main.ftl
# translate strings, commit

Dependencies

Crate	Version	Purpose
clap	4.6	CLI parsing (derive)
russh	0.61	SSH client
russh-keys	0.50-beta	SSH key loading
tokio	1 (full)	Async runtime
toml	1	Config deserialization
serde	1	Serialization
tracing	0.1	Logging
anyhow	1.0	Error handling
futures	0.3	Concurrent SSH
sha2	0.11	Host key fingerprints
base64	0.22	Base64 encoding
reqwest	0.13	HTTP client (HA)
serde_json	1	JSON (HA)
dirs	6	OS config dir
fluent-i18n	—	Mozilla Fluent i18n