# Project Context — sm121-launcher

> Derived by `map-codebase` from a cold scan of the codebase. This is the
> project's long-term memory: stack, architecture, conventions, and the
> signals that influence planning. Refresh after significant changes.

## Stack

- **Language:** Rust, edition 2024 (strict typing, no `unsafe`)
- **Binary:** `sm121` — CLI for launching vLLM/SGLang inference containers
  across a DGX Spark cluster via SSH
- **Core dependencies:**
  - `clap` 4.6 (derive) — CLI argument parsing
  - `tokio` 1 (full) — async runtime; all SSH/Docker ops are async
  - `russh` 0.61 + `russh-keys` 0.50.0-beta.7 — SSH client (cluster ops)
  - `anyhow` 1.0 — error handling (`?` + `.context()` + `bail!`)
  - `serde` + `serde_json` + `toml` + `toml_edit` — config (TOML) + recipe files
  - `tracing` + `tracing-subscriber` (env-filter) — structured logging
  - `sha2` 0.11 — Docker image checksum labels
  - `base64` 0.22 — file-content encoding for SSH upload
  - `walkdir` 2 — recursive docker-dir checksumming
  - `reqwest` 0.13 (rustls, no default-features) — Home Assistant REST API
  - `fluent-i18n` 0.1.0 — localization (6 locales: en/ru/es/it/fr/de)
  - `dirs` 6 — config-dir discovery

## Architecture

### Entry points
- `src/main.rs` — `main()`: locale init, tracing init, dispatch on `Commands`
- `src/cli.rs` — `Commands` enum (clap derive): `serve`, `stop`, `logs`,
  `health`, `doctor`, `exec`, `power`, `tmux`, `init-host-keys`, `build`

### Primary data flow (serve)
```
main.rs → cli dispatch → orchestrate/mod.rs::serve()
  ├─ cluster_state.rs::select_cluster()   (auto: smallest with NCCL + required_nodes)
  ├─ orchestrate/mod.rs                  (power-on via home_assistant.rs if --on)
  ├─ before_serve.rs                     (cluster + recipe hooks, parallel SSH)
  ├─ docker/build.rs::handle_build()      (ONLY if --build-node; checksum-gated)
  ├─ docker/mod.rs::serve()
  │    ├─ doctor.rs::try_repaired_image()
  │    ├─ docker/stale.rs::warn_if_stale()  (checksum label vs dir, non-blocking)
  │    ├─ docker/sync.rs                  (digest-based image sync to peers)
  │    ├─ docker/command.rs               (build `docker run` cmd per node)
  │    │    ├─ docker/compose.rs          (composed_model dir: symlinks + files)
  │    │    ├─ docker/mods.rs             (entrypoint wrapping for [mods])
  │    │    └─ resolve extra_mounts + path_refs
  │    └─ ssh.rs::run_ssh() per node      (tokio::spawn, parallel)
  └─ docker/logs.rs                      (follow logs from head node)
```

### Module layout
- `config/` — `Config`, `Cluster`, `Node`, `Recipe`, `ComposedModel` types +
  TOML loading (`config.toml` + `recipes/<name>/recipe.toml`)
- `docker/` — container lifecycle: `build`, `sync`, `command`, `compose`,
  `mods`, `health`, `logs`, `deploy`, `stale`, `tail`
- `orchestrate/` — high-level serve/stop/power orchestration
- `tmux/` — alternative launcher (tmux sessions instead of docker)
- `ssh.rs` — russh wrapper: `run_ssh`, `run_ssh_with_retry`, `upload_script`,
  `upload_dir`, `fetch_host_key`, `load_secret_key`
- `power.rs`, `home_assistant.rs` — HA entity control (power on/off/switch)
- `before_serve.rs`, `after_stop.rs` — hook execution (parallel across nodes)
- `cluster_state.rs` — occupancy tracking + auto-cluster selection
- `builtin_scripts.rs` — `include_str!` of `host-scripts/*.sh` (stop/restore
  DGX services), referenced by alias from recipes
- `host_keys.rs` — SSH host key pinning into `config.toml`
- `doctor.rs` — `.repaired` image tag variant detection

### Config discovery (main.rs:38-39)
1. `~/.config/sm121/config.toml`
2. `./config.toml` in cwd
3. `<default_config_dir>/config.toml`

### External artifacts (repo-rooted)
- `recipes/<name>/recipe.toml` — 22 recipes
- `docker-images/<dir>/` — 4 image build dirs (Dockerfile + build.sh +
  versions.env)
- `host-scripts/*.sh` — 2 built-in scripts (embedded at compile time)
- `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent translations

## Conventions (Observed)

- **Error handling:** anyhow throughout. 47 `bail!` (validation), 21
  `.context()` (error enrichment), 15 `anyhow!` (ad-hoc). Errors bubble with
  context; no custom error types. SSH failures include command + output in
  the error message.
- **Observability:** `tracing` macros — 85 `info!`, 36 `warn!`, 1 `debug!`.
  `tracing-subscriber` with env-filter (default `info`). No structured JSON
  logging; human-readable with ANSI colors.
- **Type safety:** strict Rust, no `unsafe`, no `#[allow(...)]` (clippy
  warnings fixed by refactoring, never suppressed). `#![deny(clippy::wildcard_imports)]`.
- **Testing:** 128 tests across 18 files. Inline `#[cfg(test)] mod tests` +
  co-located `tests.rs` siblings (e.g. `config/tests.rs`, `docker/tests.rs`,
  `orchestrate/tests.rs`). Async tests use `#[tokio::test]`.
- **File-size discipline:** ≤500 lines per `.rs` file, enforced by
  `make check-lines`. Submodules extracted when a file approaches the limit.
- **Shell-quoting:** `shq()` wraps strings in single quotes for safe SSH
  interpolation. A known footgun: `shq` output inside `"..."` makes the
  single quotes literal (caused the composed-dir zero-symlinks bug, fixed
  in `0208fc1`). Convention: keep `shq()` bare, quote only `$vars`.
- **No TODO/FIXME markers** in the codebase (clean).
- **Commit style:** Conventional Commits (`feat(...)`, `fix(...)`, no
  `Co-authored-by` footers — P1 rule).

## Signals / Active Considerations

1. **Debt hotspots — 4 files at/near the 500-line limit:**
   - `power.rs` (500), `docker/mod.rs` (499), `docker/command.rs` (499),
     `docker/build.rs` (488)
   - Any new feature touching these requires refactoring first (extract
     submodules). This is the tightest constraint on where work can land.

2. **i18n thread-local bug (systemic, pre-existing):** `fluent_i18n::set_locale`
   is thread-local; `tokio::spawn` worker threads don't inherit the locale
   set on main. 16 spawn sites affected. Non-en translations only render on
   the main thread; en (fallback) renders everywhere. Fixing requires either
   a process-global locale store or per-task `set_locale` calls.

3. **i18n key drift:** en has 127 keys; other locales have 91 (36 keys
   missing). Non-en translations are incomplete — newly added keys
   (composed-dir-*, stale warnings) need translation.

4. **Stop-flow cleanup gap:** `composed_model` dirs in `/tmp` aren't cleaned
   if containers exited with `--rm` (stop_cluster only runs when running
   containers exist). Harmless symlinks accumulate. Minor.

5. **Docker image staleness:** `serve` without `--build-node` now WARNs on
   checksum mismatch (`63b5518`), but proceeds. Stale images can still run
   — the check is non-blocking by design (operator may want the current
   image). Documented in AGENTS.md.

6. **Cluster safety (operational, not code):** The cluster may be serving
   live traffic at any time. `stop`/`power off`/`power switch` MUST be
   preceded by `health` + user confirmation. A stray `stop` has killed
   running agents mid-task before. Enforced by discipline, not code.

7. **Two source-of-truth files (post-bigpowers adoption):** AGENTS.md
   (cluster/recipe/DGX ops, the original SoT) now coexists with
   CLAUDE.md + CONVENTIONS.md + specs/ (dev workflow, bigpowers). Split-
   domains decision: AGENTS.md owns cluster/recipe technical reference;
   CLAUDE.md/CONVENTIONS.md own dev workflow + hard gates. Both must stay
   consistent on the gates that overlap (≤500 lines, make check, no
   #[allow]).

8. **Composed-model mechanism is new and feature-rich:** `[[composed_model]]`
   with recursive symlink defaults, subdir globs, generated files, and
   container-side absolute symlinks. Tested via `test-composed` recipe but
   not yet exercised against a real multi-repo model build end-to-end on
   the cluster (the NF4 Lloyd-Experts recipe awaits a build).