|
|
@@ -0,0 +1,149 @@
|
|
|
+# Project Context — sm121-launcher
|
|
|
+
|
|
|
+> Derived by `map-codebase` from a cold scan of the codebase. This is the
|
|
|
+> project's long-term memory: stack, architecture, conventions, and the
|
|
|
+> signals that influence planning. Refresh after significant changes.
|
|
|
+
|
|
|
+## Stack
|
|
|
+
|
|
|
+- **Language:** Rust, edition 2024 (strict typing, no `unsafe`)
|
|
|
+- **Binary:** `sm121` — CLI for launching vLLM/SGLang inference containers
|
|
|
+ across a DGX Spark cluster via SSH
|
|
|
+- **Core dependencies:**
|
|
|
+ - `clap` 4.6 (derive) — CLI argument parsing
|
|
|
+ - `tokio` 1 (full) — async runtime; all SSH/Docker ops are async
|
|
|
+ - `russh` 0.61 + `russh-keys` 0.50.0-beta.7 — SSH client (cluster ops)
|
|
|
+ - `anyhow` 1.0 — error handling (`?` + `.context()` + `bail!`)
|
|
|
+ - `serde` + `serde_json` + `toml` + `toml_edit` — config (TOML) + recipe files
|
|
|
+ - `tracing` + `tracing-subscriber` (env-filter) — structured logging
|
|
|
+ - `sha2` 0.11 — Docker image checksum labels
|
|
|
+ - `base64` 0.22 — file-content encoding for SSH upload
|
|
|
+ - `walkdir` 2 — recursive docker-dir checksumming
|
|
|
+ - `reqwest` 0.13 (rustls, no default-features) — Home Assistant REST API
|
|
|
+ - `fluent-i18n` 0.1.0 — localization (6 locales: en/ru/es/it/fr/de)
|
|
|
+ - `dirs` 6 — config-dir discovery
|
|
|
+
|
|
|
+## Architecture
|
|
|
+
|
|
|
+### Entry points
|
|
|
+- `src/main.rs` — `main()`: locale init, tracing init, dispatch on `Commands`
|
|
|
+- `src/cli.rs` — `Commands` enum (clap derive): `serve`, `stop`, `logs`,
|
|
|
+ `health`, `doctor`, `exec`, `power`, `tmux`, `init-host-keys`, `build`
|
|
|
+
|
|
|
+### Primary data flow (serve)
|
|
|
+```
|
|
|
+main.rs → cli dispatch → orchestrate/mod.rs::serve()
|
|
|
+ ├─ cluster_state.rs::select_cluster() (auto: smallest with NCCL + required_nodes)
|
|
|
+ ├─ orchestrate/mod.rs (power-on via home_assistant.rs if --on)
|
|
|
+ ├─ before_serve.rs (cluster + recipe hooks, parallel SSH)
|
|
|
+ ├─ docker/build.rs::handle_build() (ONLY if --build-node; checksum-gated)
|
|
|
+ ├─ docker/mod.rs::serve()
|
|
|
+ │ ├─ doctor.rs::try_repaired_image()
|
|
|
+ │ ├─ docker/stale.rs::warn_if_stale() (checksum label vs dir, non-blocking)
|
|
|
+ │ ├─ docker/sync.rs (digest-based image sync to peers)
|
|
|
+ │ ├─ docker/command.rs (build `docker run` cmd per node)
|
|
|
+ │ │ ├─ docker/compose.rs (composed_model dir: symlinks + files)
|
|
|
+ │ │ ├─ docker/mods.rs (entrypoint wrapping for [mods])
|
|
|
+ │ │ └─ resolve extra_mounts + path_refs
|
|
|
+ │ └─ ssh.rs::run_ssh() per node (tokio::spawn, parallel)
|
|
|
+ └─ docker/logs.rs (follow logs from head node)
|
|
|
+```
|
|
|
+
|
|
|
+### Module layout
|
|
|
+- `config/` — `Config`, `Cluster`, `Node`, `Recipe`, `ComposedModel` types +
|
|
|
+ TOML loading (`config.toml` + `recipes/<name>/recipe.toml`)
|
|
|
+- `docker/` — container lifecycle: `build`, `sync`, `command`, `compose`,
|
|
|
+ `mods`, `health`, `logs`, `deploy`, `stale`, `tail`
|
|
|
+- `orchestrate/` — high-level serve/stop/power orchestration
|
|
|
+- `tmux/` — alternative launcher (tmux sessions instead of docker)
|
|
|
+- `ssh.rs` — russh wrapper: `run_ssh`, `run_ssh_with_retry`, `upload_script`,
|
|
|
+ `upload_dir`, `fetch_host_key`, `load_secret_key`
|
|
|
+- `power.rs`, `home_assistant.rs` — HA entity control (power on/off/switch)
|
|
|
+- `before_serve.rs`, `after_stop.rs` — hook execution (parallel across nodes)
|
|
|
+- `cluster_state.rs` — occupancy tracking + auto-cluster selection
|
|
|
+- `builtin_scripts.rs` — `include_str!` of `host-scripts/*.sh` (stop/restore
|
|
|
+ DGX services), referenced by alias from recipes
|
|
|
+- `host_keys.rs` — SSH host key pinning into `config.toml`
|
|
|
+- `doctor.rs` — `.repaired` image tag variant detection
|
|
|
+
|
|
|
+### Config discovery (main.rs:38-39)
|
|
|
+1. `~/.config/sm121/config.toml`
|
|
|
+2. `./config.toml` in cwd
|
|
|
+3. `<default_config_dir>/config.toml`
|
|
|
+
|
|
|
+### External artifacts (repo-rooted)
|
|
|
+- `recipes/<name>/recipe.toml` — 22 recipes
|
|
|
+- `docker-images/<dir>/` — 4 image build dirs (Dockerfile + build.sh +
|
|
|
+ versions.env)
|
|
|
+- `host-scripts/*.sh` — 2 built-in scripts (embedded at compile time)
|
|
|
+- `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent translations
|
|
|
+
|
|
|
+## Conventions (Observed)
|
|
|
+
|
|
|
+- **Error handling:** anyhow throughout. 47 `bail!` (validation), 21
|
|
|
+ `.context()` (error enrichment), 15 `anyhow!` (ad-hoc). Errors bubble with
|
|
|
+ context; no custom error types. SSH failures include command + output in
|
|
|
+ the error message.
|
|
|
+- **Observability:** `tracing` macros — 85 `info!`, 36 `warn!`, 1 `debug!`.
|
|
|
+ `tracing-subscriber` with env-filter (default `info`). No structured JSON
|
|
|
+ logging; human-readable with ANSI colors.
|
|
|
+- **Type safety:** strict Rust, no `unsafe`, no `#[allow(...)]` (clippy
|
|
|
+ warnings fixed by refactoring, never suppressed). `#![deny(clippy::wildcard_imports)]`.
|
|
|
+- **Testing:** 128 tests across 18 files. Inline `#[cfg(test)] mod tests` +
|
|
|
+ co-located `tests.rs` siblings (e.g. `config/tests.rs`, `docker/tests.rs`,
|
|
|
+ `orchestrate/tests.rs`). Async tests use `#[tokio::test]`.
|
|
|
+- **File-size discipline:** ≤500 lines per `.rs` file, enforced by
|
|
|
+ `make check-lines`. Submodules extracted when a file approaches the limit.
|
|
|
+- **Shell-quoting:** `shq()` wraps strings in single quotes for safe SSH
|
|
|
+ interpolation. A known footgun: `shq` output inside `"..."` makes the
|
|
|
+ single quotes literal (caused the composed-dir zero-symlinks bug, fixed
|
|
|
+ in `0208fc1`). Convention: keep `shq()` bare, quote only `$vars`.
|
|
|
+- **No TODO/FIXME markers** in the codebase (clean).
|
|
|
+- **Commit style:** Conventional Commits (`feat(...)`, `fix(...)`, no
|
|
|
+ `Co-authored-by` footers — P1 rule).
|
|
|
+
|
|
|
+## Signals / Active Considerations
|
|
|
+
|
|
|
+1. **Debt hotspots — 4 files at/near the 500-line limit:**
|
|
|
+ - `power.rs` (500), `docker/mod.rs` (499), `docker/command.rs` (499),
|
|
|
+ `docker/build.rs` (488)
|
|
|
+ - Any new feature touching these requires refactoring first (extract
|
|
|
+ submodules). This is the tightest constraint on where work can land.
|
|
|
+
|
|
|
+2. **i18n thread-local bug (systemic, pre-existing):** `fluent_i18n::set_locale`
|
|
|
+ is thread-local; `tokio::spawn` worker threads don't inherit the locale
|
|
|
+ set on main. 16 spawn sites affected. Non-en translations only render on
|
|
|
+ the main thread; en (fallback) renders everywhere. Fixing requires either
|
|
|
+ a process-global locale store or per-task `set_locale` calls.
|
|
|
+
|
|
|
+3. **i18n key drift:** en has 127 keys; other locales have 91 (36 keys
|
|
|
+ missing). Non-en translations are incomplete — newly added keys
|
|
|
+ (composed-dir-*, stale warnings) need translation.
|
|
|
+
|
|
|
+4. **Stop-flow cleanup gap:** `composed_model` dirs in `/tmp` aren't cleaned
|
|
|
+ if containers exited with `--rm` (stop_cluster only runs when running
|
|
|
+ containers exist). Harmless symlinks accumulate. Minor.
|
|
|
+
|
|
|
+5. **Docker image staleness:** `serve` without `--build-node` now WARNs on
|
|
|
+ checksum mismatch (`63b5518`), but proceeds. Stale images can still run
|
|
|
+ — the check is non-blocking by design (operator may want the current
|
|
|
+ image). Documented in AGENTS.md.
|
|
|
+
|
|
|
+6. **Cluster safety (operational, not code):** The cluster may be serving
|
|
|
+ live traffic at any time. `stop`/`power off`/`power switch` MUST be
|
|
|
+ preceded by `health` + user confirmation. A stray `stop` has killed
|
|
|
+ running agents mid-task before. Enforced by discipline, not code.
|
|
|
+
|
|
|
+7. **Two source-of-truth files (post-bigpowers adoption):** AGENTS.md
|
|
|
+ (cluster/recipe/DGX ops, the original SoT) now coexists with
|
|
|
+ CLAUDE.md + CONVENTIONS.md + specs/ (dev workflow, bigpowers). Split-
|
|
|
+ domains decision: AGENTS.md owns cluster/recipe technical reference;
|
|
|
+ CLAUDE.md/CONVENTIONS.md own dev workflow + hard gates. Both must stay
|
|
|
+ consistent on the gates that overlap (≤500 lines, make check, no
|
|
|
+ #[allow]).
|
|
|
+
|
|
|
+8. **Composed-model mechanism is new and feature-rich:** `[[composed_model]]`
|
|
|
+ with recursive symlink defaults, subdir globs, generated files, and
|
|
|
+ container-side absolute symlinks. Tested via `test-composed` recipe but
|
|
|
+ not yet exercised against a real multi-repo model build end-to-end on
|
|
|
+ the cluster (the NF4 Lloyd-Experts recipe awaits a build).
|
