launcher/specs/tech-architecture/tech-stack.md

Project Context — sm121-launcher

Derived by map-codebase from a cold scan of the codebase. This is the project's long-term memory: stack, architecture, conventions, and the signals that influence planning. Refresh after significant changes.

Stack

• Language: Rust, edition 2024 (strict typing, no unsafe)
• Binary: sm121 — CLI for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH
• Core dependencies:
  • clap 4.6 (derive) — CLI argument parsing
  • tokio 1 (full) — async runtime; all SSH/Docker ops are async
  • russh 0.61 + russh-keys 0.50.0-beta.7 — SSH client (cluster ops)
  • anyhow 1.0 — error handling (? + .context() + bail!)
  • serde + serde_json + toml + toml_edit — config (TOML) + recipe files
  • tracing + tracing-subscriber (env-filter) — structured logging
  • sha2 0.11 — Docker image checksum labels
  • base64 0.22 — file-content encoding for SSH upload
  • walkdir 2 — recursive docker-dir checksumming
  • reqwest 0.13 (rustls, no default-features) — Home Assistant REST API
  • fluent-i18n 0.1.0 — localization (6 locales: en/ru/es/it/fr/de)
  • dirs 6 — config-dir discovery

Architecture

Entry points

• src/main.rs — main(): locale init, tracing init, dispatch on Commands
• src/cli.rs — Commands enum (clap derive): serve, stop, logs, health, doctor, exec, power, tmux, init-host-keys, build

Primary data flow (serve)

main.rs → cli dispatch → orchestrate/mod.rs::serve()
  ├─ cluster_state.rs::select_cluster()   (auto: smallest with NCCL + required_nodes)
  ├─ orchestrate/mod.rs                  (power-on via home_assistant.rs if --on)
  ├─ before_serve.rs                     (cluster + recipe hooks, parallel SSH)
  ├─ docker/build.rs::handle_build()      (ONLY if --build-node; checksum-gated)
  ├─ docker/mod.rs::serve()
  │    ├─ doctor.rs::try_repaired_image()
  │    ├─ docker/stale.rs::warn_if_stale()  (checksum label vs dir, non-blocking)
  │    ├─ docker/sync.rs                  (digest-based image sync to peers)
  │    ├─ docker/command.rs               (build `docker run` cmd per node)
  │    │    ├─ docker/compose.rs          (composed_model dir: symlinks + files)
  │    │    ├─ docker/mods.rs             (entrypoint wrapping for [mods])
  │    │    └─ resolve extra_mounts + path_refs
  │    └─ ssh.rs::run_ssh() per node      (tokio::spawn, parallel)
  └─ docker/logs.rs                      (follow logs from head node)

Module layout

• config/ — Config, Cluster, Node, Recipe, ComposedModel types + TOML loading (config.toml + recipes/<name>/recipe.toml)
• docker/ — container lifecycle: build, sync, command, compose, mods, health, logs, deploy, stale, tail
• orchestrate/ — high-level serve/stop/power orchestration
• tmux/ — alternative launcher (tmux sessions instead of docker)
• ssh.rs — russh wrapper: run_ssh, run_ssh_with_retry, upload_script, upload_dir, fetch_host_key, load_secret_key
• power.rs, home_assistant.rs — HA entity control (power on/off/switch)
• before_serve.rs, after_stop.rs — hook execution (parallel across nodes)
• cluster_state.rs — occupancy tracking + auto-cluster selection
• builtin_scripts.rs — include_str! of host-scripts/*.sh (stop/restore DGX services), referenced by alias from recipes
• host_keys.rs — SSH host key pinning into config.toml
• doctor.rs — .repaired image tag variant detection

Config discovery (main.rs:38-39)

1. ~/.config/sm121/config.toml
2. ./config.toml in cwd
3. <default_config_dir>/config.toml

External artifacts (repo-rooted)

• recipes/<name>/recipe.toml — 22 recipes
• docker-images/<dir>/ — 4 image build dirs (Dockerfile + build.sh + versions.env)
• host-scripts/*.sh — 2 built-in scripts (embedded at compile time)
• locales/{en,ru,es,it,fr,de}/main.ftl — Fluent translations

Conventions (Observed)

• Error handling: anyhow throughout. 47 bail! (validation), 21 .context() (error enrichment), 15 anyhow! (ad-hoc). Errors bubble with context; no custom error types. SSH failures include command + output in the error message.
• Observability: tracing macros — 85 info!, 36 warn!, 1 debug!. tracing-subscriber with env-filter (default info). No structured JSON logging; human-readable with ANSI colors.
• Type safety: strict Rust, no unsafe, no #[allow(...)] (clippy warnings fixed by refactoring, never suppressed). #![deny(clippy::wildcard_imports)].
• Testing: 128 tests across 18 files. Inline #[cfg(test)] mod tests + co-located tests.rs siblings (e.g. config/tests.rs, docker/tests.rs, orchestrate/tests.rs). Async tests use #[tokio::test].
• File-size discipline: ≤500 lines per .rs file, enforced by make check-lines. Submodules extracted when a file approaches the limit.
• Shell-quoting: shq() wraps strings in single quotes for safe SSH interpolation. A known footgun: shq output inside "..." makes the single quotes literal (caused the composed-dir zero-symlinks bug, fixed in 0208fc1). Convention: keep shq() bare, quote only $vars.
• No TODO/FIXME markers in the codebase (clean).
• Commit style: Conventional Commits (feat(...), fix(...), no Co-authored-by footers — P1 rule).

Signals / Active Considerations

1. Debt hotspots — 4 files at/near the 500-line limit:

   • power.rs (500), docker/mod.rs (499), docker/command.rs (499), docker/build.rs (488)
   • Any new feature touching these requires refactoring first (extract submodules). This is the tightest constraint on where work can land.

2. i18n thread-local bug (systemic, pre-existing): fluent_i18n::set_locale is thread-local; tokio::spawn worker threads don't inherit the locale set on main. 16 spawn sites affected. Non-en translations only render on the main thread; en (fallback) renders everywhere. Fixing requires either a process-global locale store or per-task set_locale calls.

3. i18n key drift: en has 127 keys; other locales have 91 (36 keys missing). Non-en translations are incomplete — newly added keys (composed-dir-*, stale warnings) need translation.

4. Stop-flow cleanup gap: composed_model dirs in /tmp aren't cleaned if containers exited with --rm (stop_cluster only runs when running containers exist). Harmless symlinks accumulate. Minor.

5. Docker image staleness: serve without --build-node now WARNs on checksum mismatch (63b5518), but proceeds. Stale images can still run — the check is non-blocking by design (operator may want the current image). Documented in AGENTS.md.

6. Cluster safety (operational, not code): The cluster may be serving live traffic at any time. stop/power off/power switch MUST be preceded by health + user confirmation. A stray stop has killed running agents mid-task before. Enforced by discipline, not code.

7. Two source-of-truth files (post-bigpowers adoption): AGENTS.md (cluster/recipe/DGX ops, the original SoT) now coexists with CLAUDE.md + CONVENTIONS.md + specs/ (dev workflow, bigpowers). Split- domains decision: AGENTS.md owns cluster/recipe technical reference; CLAUDE.md/CONVENTIONS.md own dev workflow + hard gates. Both must stay consistent on the gates that overlap (≤500 lines, make check, no #[allow]).

8. Composed-model mechanism is new and feature-rich: [[composed_model]] with recursive symlink defaults, subdir globs, generated files, and container-side absolute symlinks. Tested via test-composed recipe but not yet exercised against a real multi-repo model build end-to-end on the cluster (the NF4 Lloyd-Experts recipe awaits a build).