docs: rewrite README to match current implementation (sm121 binary, all commands, serve flags, recipe fields, mikrotik/registry config, build flow)

README.md

@@ -1,241 +1,230 @@
-# sm121-launcher
+# sm121
 
-Launch vLLM / SGLang inference clusters across DGX Spark machines via SSH.
+Launch vLLM / SGLang inference clusters across a DGX Spark fleet over SSH.
+Docker is the default deployment target; tmux is a first-class alternative
+(`deploy = "tmux"` in the recipe).
 
-## Quick Install
+## Install
 
 ```bash
-# From source
 cargo install --path .
-
-# Or download a pre-built binary (requires Rust toolchain to build)
-cargo build --release
-# Binary at target/release/sm121-launcher
+# or: cargo build --release   → target/release/sm121
 ```
 
-## Quick Start
+## Quick start
 
 ```bash
-# Create config.toml (see Configuration below)
-# Launch a recipe
-sm121-launcher serve --recipe deepseek-v4-flash
-
-# Follow logs
-sm121-launcher logs --recipe deepseek-v4-flash --follow
-
-# Stop containers
-sm121-launcher stop --recipe deepseek-v4-flash
+sm121 config init-global            # create the global config if missing
+sm121 -d serve --recipe deepseek-v4-flash   # dry-run: print the commands
+sm121 serve --recipe deepseek-v4-flash      # launch (waits for the engine)
+sm121 logs --recipe deepseek-v4-flash --follow
+sm121 bench --recipe deepseek-v4-flash --label smoke   # tool-eval-bench
+sm121 stop --recipe deepseek-v4-flash  # stops ALL nodes, verifies each one
+sm121 health                          # cluster-level health (nodes + switches)
 ```
 
+`-d`/`--dry-run` and `-c/--config <path>` are top-level flags:
+`sm121 -d serve ...`, `sm121 -c /other/config.toml serve ...`.
+
 ## Commands
 
 | Command | Description |
 |---------|-------------|
-| `serve --recipe <name>` | Launch recipe on cluster |
-| `logs --recipe <name> [--follow]` | Show container logs |
-| `stop --recipe <name>` | Stop and remove containers |
-| `health [--cluster <name>]` | Check cluster health |
+| `serve --recipe <name>` | Launch; records the run in the ledger, waits for the engine (`/v1/models`, up to 30 min; `--no-wait` skips) |
+| `stop --recipe <name>` | Stop **all** nodes in parallel, verify each (60 s poll); `--no-wait` opts out |
+| `logs --recipe <name> [--follow]` | Container logs; follow = liveness watch (30 s probes, 2 misses = died) |
+| `bench --recipe <name>` | `tool-eval-bench` against a live serve; reports auto-land in `recipes/<lane>/reports/` (+ run package: recipe copy + manifest) |
+| `report [--serve]` | Table of tested recipes; `--serve` = interactive HTML (SSE, in-TOML navigation) |
+| `health [--cluster <name>]` | Cluster-level health: UP only when all nodes reachable **and** blocking extras (HA switches, MikroTik routers) are on |
 | `doctor --recipe <name> [--repair]` | Check/repair node prerequisites |
-| `exec --cluster <name> -- <command>` | Run command on cluster nodes |
-| `exec --cluster <name> --alias <name>` | Run predefined alias |
-| `power off --cluster <name> [--node <name>]` | Shutdown + HA turn_off |
-| `power on --cluster <name> [--node <name>]` | HA turn_on (wake nodes) |
-| `power switch --from <src> --to <tgt>` | Switch clusters |
-| `tmux serve --recipe <name>` | Launch via tmux (no Docker) |
-| `tmux status --recipe <name>` | Check tmux session health |
-| `tmux logs --recipe <name>` | Capture tmux pane output |
-| `init-host-keys` | Auto-write SSH host keys |
-| `-d` / `--dry-run` | Preview commands without executing |
-
-### Cluster selection
+| `exec --cluster <name> -- <cmd>` | Run a shell command on cluster nodes; `--alias <name>` runs a predefined one |
+| `build --recipe <name> --build-node <n>` | Build the recipe's image from `docker-images/<dir>/` on a node (clone-at-pin) |
+| `publish --recipe <name> [--push]` | Self-contained recipe+image bundle; `--push` also pushes the image under a versioned tag |
+| `registry` | Browse/install recipes from the public registry |
+| `runlock list / close <recipe>` | Offline run-ledger management (per-recipe `.lock`) |
+| `web` | Local dashboard (axum + SSE): recipes, topology map, LED board, jobs, serve/stop jobs |
+| `power off/on/switch` | Power management via Home Assistant + MikroTik RouterOS (see below) |
+| `tmux serve/status/logs` | tmux deployment target (no Docker) |
+| `init-host-keys` | Auto-discover SSH host keys into config |
+| `config path / init-global` | Config resolution helpers |
+| `-d` / `--dry-run` | Print instead of execute (all commands) |
+
+### `serve` flags (cluster selection & overrides)
 
 ```bash
-sm121-launcher serve --recipe deepseek-v4-flash              # auto (smallest match)
-sm121-launcher serve --recipe glm51 --cluster 4nodes-switch  # exact name
-sm121-launcher serve --recipe glm51 --cluster 4             # prefix match
-sm121-launcher serve --recipe glm51 --cluster 2             # picks 2nodes-dac
+sm121 serve --recipe R --cluster auto            # smallest cluster that fits
+sm121 serve --recipe R --cluster 4               # exact name, prefix, or auto
+sm121 serve --recipe R --solo                    # first free node, single-node run
+sm121 serve --recipe R --force                   # skip occupancy check (co-serve)
+sm121 serve --recipe R --fit=2                   # scale to N nodes, TP tracks
+sm121 serve --recipe R --mirror                  # per-node local HF mirror (no NAS at load)
+sm121 serve --recipe R --clear                   # rm -rf host extra_mounts dirs first
+sm121 serve --recipe R --build-node dgx1         # build the image first (if stale)
+sm121 serve --recipe R --sync-image              # push the image to launch nodes first
 ```
 
-Prefix must be unambiguous — error if multiple clusters match.
+Cluster selection: exact name, unambiguous prefix (`4` → `4nodes-switch`), or
+`auto` (smallest cluster with NCCL config + matching `required_nodes`).
 
 ## Recipes
 
-Recipes are TOML files in `recipes/<name>/recipe.toml`, resolved relative
-to the current directory (then `~/.config/sm121-launcher/recipes/`).
-This repo is the source of truth: a recipe and its image build context
-(`docker-images/<dir>/` — Dockerfile, patches, version pins) are
-versioned together, since the served image must match the recipe's
-revision exactly. `sm121 publish` flattens both into a self-contained
-bundle; the registry data repo <https://git.kilork.org/sm121/recipes>
-stores those published bundles (discovery/sharing artifact, never the
-source).
+Recipes are TOML: `recipes/<name>/recipe.toml`, resolved from `./recipes`
+first, then each `recipe_sources` entry in the config. This repo is the
+source of truth: a recipe and its image build context (`docker-images/<dir>/`)
+are versioned together — the served image must match the recipe revision.
+`sm121 publish` flattens both into a self-contained bundle; the public
+registry <https://git.kilork.org/sm121/recipes> stores those bundles only.
 
-### Creating a recipe
+### Core fields
 
 ```toml
-engine = "vllm"                        # or "sglang" (default: "vllm")
-name = "My Model"
-docker_image = "my-image:tag"
+engine = "vllm"                        # or "sglang"
+name = "My Model"                      # display name
+docker_image = "jovian-judgement:r35-glm53"
+docker_image_dir = "jovian-judgement"  # build dir under docker-images/ (default: recipe name)
 model = "org/model-name"
-required_nodes = 2                     # optional
-tensor_parallel_size = 2               # TP across nodes
-pipeline_parallel_size = 1             # vLLM only, 0 = auto (equals nnodes)
+parent_model = "org/base-model"        # optional; report grouping
+port = 30000
+required_nodes = 2                     # optional; also caps the launch set
+solo = false                           # optional; single-node on first free node
+deploy = "docker"                      # or "tmux"
+tensor_parallel_size = 2
+pipeline_parallel_size = 1             # vLLM only, 0 = auto
 distributed_backend = "mp"             # "mp" (default) or "ray"
-shm_size = "64gb"                      # optional, only added when set
+shm_size = "64gb"                      # optional
+entrypoint = ["vllm", "serve"]         # optional; used by docker and tmux
 
-entrypoint = ["vllm", "serve"]         # optional, appended after image
+args = ["--kv-cache-dtype", "fp8"]     # engine args, passed through verbatim
 
-args = [
-    "--kv-cache-dtype", "fp8",
-    "--max-model-len", "32768",
-]
+[env]
+KEY = "value"                          # wins over launcher defaults (last -e wins)
+PORT = { from = "port" }               # value resolved from a recipe field at serve time
 
-before_serve = [                       # recipe-level pre-serve hooks
-    { command = "echo ready", on_failure = "abort" },
+before_serve = [
+    "sudo nvidia-smi -lgc",            # bare string = raw command
+    { command = "echo hi", on_failure = "warn" },
+    { script = "stop_dgx_services" },  # built-in host script alias (embedded in binary)
 ]
-
-[env]
-VLLM_LOGGING_COLOR = "1"              # override any launcher default
+after_stop = [{ script = "restore_dgx_services" }]
 ```
 
-## Configuration
+Full spec — `extra_mounts` (aliases, HF `repo` path refs, `local_files`
+hardlinks), `composed_model`, `[mods]` (in-container patches before the
+engine), `venv`/`venv_setup` (tmux), `build_profile` + `[build.versions]`
+(image pin overrides) — is in [`AGENTS.md`](AGENTS.md#recipe-format-recipesnamerecipetoml).
 
-The launcher reads `config.toml` from the current directory,
-falling back to `~/.config/sm121-launcher/config.toml`.
+## Image builds
 
-### Minimal
+A recipe's `docker_image` is built **on demand** (`build` or
+`serve --build-node`): the image dir is uploaded to the node, built in a tmux
+session, and stamped with a `sm121.checksum` label (recursive SHA-256 of the
+dir); later serves skip the build when the label matches. Builds are
+clone-at-pin: every input is a pinned `KEY=VAL` in `versions.env`, staged by
+`fetch-deps.sh` and verified against the committed `locks/deps.lock`
+(commit SHAs / gitlink SHAs / blob fingerprint) — any drift aborts the build.
 
-```toml
-ssh_user = "kilork"
-ssh_key_path = "/path/to/ssh/key"
+Pins can be overridden without editing the image dir: a `build_profile`
+references `profiles/<name>.env` in the image dir; precedence:
+`versions.env` < profile < recipe `[build.versions]`.
 
-[[clusters]]
-name = "4nodes-switch"
-master_ib_ip = "192.168.200.15"
-nodes = [
-    { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0 },
-    { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1 },
-    { name = "dgx3", ssh_ip = "10.0.0.3", ib_ip = "192.168.200.17", rank = 2 },
-    { name = "dgx4", ssh_ip = "10.0.0.4", ib_ip = "192.168.200.19", rank = 3 },
-]
-```
+With `[registry]` in the config, a successful build pushes the image under a
+versioned tag (`{base}/{image}:<checksum>`, + `:latest`); a local publication
+ledger skips duplicate pushes unless `--force`. `publish --push` pushes
+without a build. See [`AGENTS.md`](AGENTS.md#image-registry-pushregistry).
 
-### With Home Assistant (power management)
+## Configuration
+
+Config resolution: `-c/--config <path>` > `./config.toml` > the global
+config (`~/.config/sm121/config.toml` on Linux,
+`~/Library/Application Support/sm121/config.toml` on macOS —
+`sm121 config path`). The global config exists so `sm121` works from any
+cwd; it may carry no clusters of its own — `cluster_sources` files provide
+them (merged by name, file wins).
 
 ```toml
+ssh_user = "kilork"
+ssh_key_path = "/path/to/key"
+
+recipe_sources = ["/path/to/more/recipes"]    # extra recipe dirs (after ./recipes)
+cluster_sources = ["/path/to/clusters.toml"]  # extra cluster defs
+
 [[clusters]]
 name = "4nodes-switch"
-master_ib_ip = "192.168.200.15"
-extra_entities = ["switch.network_switch"]   # cluster-level switches
+master_ib_ip = "192.168.200.15"               # optional; default: rank-0 node's ib_ip
 nodes = [
     { name = "dgx1", ssh_ip = "10.0.0.1", ib_ip = "192.168.200.15", rank = 0, entity = "switch.dgx1" },
     { name = "dgx2", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.13", rank = 1, entity = "switch.dgx2" },
-    { name = "dgx3", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.17", rank = 2, entity = "switch.dgx3" },
-    { name = "dgx4", ssh_ip = "10.0.0.2", ib_ip = "192.168.200.19", rank = 3, entity = "switch.dgx4" },
+]
+extra_entities = [
+    "switch.network",                          # bare string = blocking (off → cluster down)
+    { entity = "switch.fan", blocks = false }, # toggled + shown, never gates up/down
 ]
 
 [nccl."4nodes-switch"]
 ib_hca = "rocep1s0f0,roceP2p1s0f0"
 socket_ifname = "enp1s0f0np0"
-ib_gid_index = 3
+ib_gid_index = 3          # optional: omit → auto-detect RoCE-v2 IPv4 GID per node at serve
 ib_disable = 0
 
 [home_assistant]
-url = "http://homeassistant.local:8123"
+url = "http://ha.lan:8123"
 token = "long-lived-access-token"
 shutdown_delay_seconds = 30
 
-[exec_aliases]
-gpu-lock = "sudo nvidia-smi -lgc 300,2300"
-shutdown = "sudo shutdown now"
-
-[before_serve."4nodes-switch"]
-commands = [
-    "sudo nvidia-smi -lgc 300,2300",
-    { command = "sudo nvidia-smi -pm 1", on_failure = "warn" },
-]
-```
-
-### NCCL config
-
-Required for each cluster. Controls InfiniBand/RoCE communication:
-
-| Field | Example | Purpose |
-|-------|---------|---------|
-| `ib_hca` | `rocep1s0f0,roceP2p1s0f0` | RDMA devices |
-| `socket_ifname` | `enp1s0f0np0` | Network interface |
-| `ib_gid_index` | `3` | RoCE GID index |
-| `ib_disable` | `0` | Keep IB enabled |
-
-## Power Management
+[mikrotik]                 # RouterOS devices probed via API (health) + graceful shutdown
+address = "192.168.88.39"
+name = "SW2-MUC"           # must match a cluster entity name
+username = "sm121"
+password = "..."
+on_entity = "switch.sw6_mt"   # optional HA relay used for power on; absent → manual reminder
 
-Requires `[home_assistant]` in config.toml.
+[registry]                 # optional: image push target (see Image builds above)
+base = "ghcr.io/kilork"
 
-```bash
-# Graceful shutdown + wait + turn off smart plugs
-sm121-launcher power off --cluster 4nodes-switch
-
-# Wake nodes via smart plugs
-sm121-launcher power on --cluster 4nodes-switch
-
-# Switch clusters (disable 4 → enable 2)
-sm121-launcher power switch --from 4 --to 2
+[exec_aliases]
+gpu-lock = "sudo nvidia-smi -lgc 300,2300"
 
-# Single node
-sm121-launcher power off --cluster 4nodes-switch --node dgx1
+[mount_aliases]            # host paths by name for recipe [[extra_mounts]]
+jit_cache = "/mnt/nas/ai/jit"
 
-# Dry-run
-sm121-launcher -d power off --cluster 4nodes-switch
+[before_serve."*"]         # global: every cluster; per-cluster key supplements
+commands = ["sudo nvidia-smi -lgc 0,2000"]
 ```
 
-**Power-off flow**: SSH `sudo shutdown now` → wait `shutdown_delay_seconds` → HA `turn_off` entity → HA `turn_off` cluster extras.
+## Power management
 
-**Power-on flow**: HA `turn_on` (no SSH, assumes smart plug wakes the machine).
+`power off`: SSH `sudo shutdown now` → wait `shutdown_delay_seconds` →
+HA `turn_off` per-node entities + cluster extras; MikroTik devices get a
+graceful RouterOS `system shutdown` instead of a plain relay cut (and are
+probed by `health` via a RouterOS API login — a dead router takes the
+cluster down like any dead switch). `power on`: HA `turn_on` (no SSH;
+MikroTik powers up through its `on_entity` relay, or a manual reminder).
+`power switch --from A --to B`: disable only what `A` has, enable all of `B`.
 
-**Switch flow**: disable nodes/extras only in source → enable all in target.
+> ⚠️ A cluster may be serving live traffic (possibly the model serving this
+> very session). Always `health` and confirm before `stop` / `power off`.
 
-## Environment Variables
+## Web dashboard
 
-The launcher sets these automatically. Recipe `[env]` overrides any.
-
-### Auto-set (always)
-
-| Env var | Source | Description |
-|---------|--------|-------------|
-| `NCCL_IB_DISABLE` | `nccl.<name>.ib_disable` | InfiniBand on/off |
-| `NCCL_IB_HCA` | `nccl.<name>.ib_hca` | RDMA devices |
-| `NCCL_SOCKET_IFNAME` | `nccl.<name>.socket_ifname` | NCCL control interface |
-| `GLOO_SOCKET_IFNAME` | same | Gloo (PyTorch) interface |
-| `TP_SOCKET_IFNAME` | same | vLLM TP coordinator |
-| `MN_IF_NAME` | same | Multi-node interface |
-| `UCX_NET_DEVICES` | same | UCX devices |
-| `OMPI_MCA_btl_tcp_if_include` | same | OpenMPI TCP interface |
-| `NCCL_IB_GID_INDEX` | `nccl.<name>.ib_gid_index` | RoCE GID |
-| `VLLM_HOST_IP` / `SGLANG_WORKER_IP` | `node.ib_ip` | Engine host IP |
-| `HF_HOME` | `/root/.cache/huggingface` | HF cache (volume-mounted) |
-| `OMP_NUM_THREADS` | `4` | OpenMP threads |
-| `NCCL_IGNORE_CPU_AFFINITY` | `1` | Don't pin NCCL threads |
-| `NCCL_NET` | `IB` | NCCL IB plugin |
-| `NCCL_DEBUG` | `WARN` | NCCL log level |
-| `VLLM_LOGGING_COLOR` | `1` | Colored logs (head only) |
-| `TERM` | `xterm-256color` | Terminal type (head only) |
-
-### Ray backend (when `distributed_backend = "ray"`)
-
-`RAY_ADDRESS`, `RAY_PORT`, `RAY_NODE_IP_ADDRESS`, `RAY_OVERRIDE_NODE_IP_ADDRESS`
+`sm121 web` — local axum + SSE dashboard: the recipes panel (the `report
+--serve` table, with run-ledger state and in-TOML navigation), a physical
+topology map (grid, 200G/10G fabric, HA glyphs, cluster frames), a
+cluster-level LED board, and serve/stop/power jobs with live status.
 
 ## Internationalization
 
-Messages are localized. Auto-detects system locale.
+Messages are localized (Fluent FTL in `locales/{en,ru,es,it,fr,de}/`),
+auto-detected from the system locale:
 
 ```bash
-# Override locale (works on both macOS and Linux)
-LANG=ru_RU.UTF-8 sm121-launcher serve --recipe deepseek-v4-flash
+LANG=ru_RU.UTF-8 sm121 serve --recipe deepseek-v4-flash
 ```
 
-Available: English (fallback), Russian, Spanish, Italian, French, German.
-
-## Project Links
+## Further reading
 
-- **Development docs**: [`DEVELOPMENT.md`](DEVELOPMENT.md)
-- **Agent reference**: [`AGENTS.md`](AGENTS.md)
+- **[`AGENTS.md`](AGENTS.md)** — full operational reference: recipe format,
+  image builds/pins, env vars, mods, composed models, registry, NCCL
+  auto-detect, power, gotchas.
+- **[`DEVELOPMENT.md`](DEVELOPMENT.md)** — architecture, module layout, dev workflow.
+- **`specs/tech-architecture/`** — measured lane states and findings.
+- **`history/night-shifts/`** — shift logs.