# Development — sm121-launcher

Rust CLI tool for launching vLLM/SGLang containers across a DGX Spark cluster. SSH-based provisioning over `russh`.

## Project Layout

```
Cargo.toml                 # Binary crate, Edition 2024
config.toml                # User config (gitignored)
locales/                   # Fluent i18n files (en, ru, es, it, fr, de)
recipes/                   # Recipe directories (<name>/recipe.toml)
src/
├── main.rs                # CLI entrypoint, command dispatch, i18n init
├── cli.rs                 # clap derive CLI definitions
├── config.rs              # Config deserialization, cluster resolution;
│                          #   config/types.rs (Node/Cluster/Config),
│                          #   config/recipe.rs (Recipe + [env] EnvValue refs)
├── ssh.rs                 # SSH client, key loading, host key verification
├── docker/
│   ├── mod.rs             # Docker command builder: serve, stop, health, logs
│   └── tests.rs           # Command builder tests
├── tmux/
│   ├── mod.rs             # Tmux module root
│   ├── serve.rs           # Tmux serve (window/panel layout)
│   └── ops.rs             # Tmux stop/status/logs/attach
├── power.rs               # Power on/off/switch flows
├── home_assistant.rs      # HA REST API client
├── host_keys.rs           # init-host-keys command
├── exec.rs                # Exec subcommand + alias resolution
├── doctor.rs              # Doctor mode + --repair
├── pre_serve.rs           # PreServeCommand types
├── before_serve.rs        # Pre-serve hook execution
└── health.rs              # Tmux health-check helpers
```

## Quality Gates

```bash
make check            # All gates: fmt → clippy → test → line-count
make fmt              # cargo fmt
make check-fmt        # cargo fmt --check
make check-clippy     # cargo clippy --all-targets -- -D warnings
make check-test       # cargo test
make check-lines      # File-size ≤500 lines per src/**/*.rs
```

## Architecture

### Docker launch flow

1. `serve` reads the recipe TOML
2. Resolves cluster (exact/prefix match → smallest with NCCL + matching node count)
3. Runs pre-serve hooks (cluster-level + recipe-level)
4. Checks for repaired image (via doctor --repair)
5. For each node: builds `docker run` command via `build_docker_command()`, runs via SSH
6. Waits 10s, then shows tail logs

`build_docker_command()` in `src/docker/mod.rs` is engine-aware:

| Aspect | vLLM | SGLang |
|---|---|---|
| Host IP env var | `VLLM_HOST_IP` | `SGLANG_WORKER_IP` |
| Model argument | bare name | `--model-path` |
| TP flag | `--tensor-parallel-size` | `--tp` |
| PP flag | `--pipeline-parallel-size` | — |
| Backend flag | `--distributed-executor-backend` (ray only) | — |
| Init addr | `--master-addr` + `--master-port` | `--dist-init-addr` |
| Workers | `--headless` | — |

Auto-generated flags: `--host 0.0.0.0`, `--port <recipe.port>`, `--nnodes`, `--node-rank`.

### Cluster selection

- **Exact match**: `--cluster 4nodes-switch`
- **Prefix match**: `--cluster 4` → resolves to `4nodes-switch` (unique prefix, fails on ambiguous)
- **Auto**: `--cluster auto` (or omitted) → picks the **smallest** cluster with NCCL config matching `required_nodes`

### Env var precedence

1. Cluster-driven (from `[nccl]`): `NCCL_IB_DISABLE`, `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`, etc.
2. Network interface (from `nccl.socket_ifname`): `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include`
3. Defaults: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN`
4. Engine IP: `VLLM_HOST_IP` / `SGLANG_WORKER_IP`
5. Head node: `VLLM_LOGGING_COLOR=1`, `TERM=xterm-256color`, `-p <port>:<port>`
6. Ray backend: `RAY_ADDRESS`, `RAY_PORT`, `RAY_NODE_IP_ADDRESS`, `RAY_OVERRIDE_NODE_IP_ADDRESS`
7. Recipe `[env]`: appended last, overrides everything above

### Power management

- `power off`: SSH `sudo shutdown now` → wait `shutdown_delay_seconds` → HA `turn_off` per-node entities + cluster extras
- `power on`: HA `turn_on` per-node entities + cluster extras (no SSH)
- `power switch`: disable nodes/extras unique to `from` → enable all in `to`

### i18n

Mozilla Fluent via `fluent-i18n` crate. Locale auto-detected from system. FTL files in `locales/{lang}/main.ftl`.

### Module size limit

Every `src/**/*.rs` file must stay ≤500 lines. Enforced by `make check-lines`.

## Dev Workflow

```bash
# Build
cargo build

# Run locally with dry-run (no SSH execution)
cargo run -- -d serve --recipe deepseek-v4-flash

# Run tests
cargo test

# All gates (must pass before commit)
make check

# Add a new recipe
mkdir -p recipes/my-model/
# create recipes/my-model/recipe.toml
cargo run -- -d serve --recipe my-model --cluster <name>

# Add a new locale
cp locales/en/main.ftl locales/xx/main.ftl
# translate strings, commit
```

## Dependencies

| Crate | Version | Purpose |
|---|---|---|
| `clap` | 4.6 | CLI parsing (derive) |
| `russh` | 0.61 | SSH client |
| `russh-keys` | 0.50-beta | SSH key loading |
| `tokio` | 1 (full) | Async runtime |
| `toml` | 1 | Config deserialization |
| `serde` | 1 | Serialization |
| `tracing` | 0.1 | Logging |
| `anyhow` | 1.0 | Error handling |
| `futures` | 0.3 | Concurrent SSH |
| `sha2` | 0.11 | Host key fingerprints |
| `base64` | 0.22 | Base64 encoding |
| `reqwest` | 0.13 | HTTP client (HA) |
| `serde_json` | 1 | JSON (HA) |
| `dirs` | 6 | OS config dir |
| `fluent-i18n` | — | Mozilla Fluent i18n |