Plan: sm121 web dashboard (design doc)

specs/tech-architecture/web-dashboard.md: axum + SSE dashboard design —
thin RPC layer over the existing serve/stop/power/health code paths,
one multiplexed SSE bus, supervisor polling via cluster_state::probe,
tracked jobs (single-flight), log attaches with ring buffers, single
embedded vanilla HTML/CSS/JS UI (no build step), light/dark themes,
floating windows, raw-content views, and the safety rails (type-the-name
power-off confirm, localhost-only default + bearer token when remote).

Decisions locked: embedded single-file UI, default port 3100.
AGENTS.md: pointer under Commands (marked planned).

AGENTS.md

@@ -20,6 +20,9 @@ See [`DEVELOPMENT.md`](DEVELOPMENT.md) for architecture, module layout, and dev
 serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys/config
 ```
 
+`sm121 web` (planned) — local axum + SSE dashboard (LEDs, toggles, live log
+windows, light/dark themes). Design: `specs/tech-architecture/web-dashboard.md`.
+
 Cluster: exact name, prefix (`4`→`4nodes-switch`), or `auto` (smallest with NCCL + matching `required_nodes`).
 
 `serve --solo` overrides cluster selection: finds the first free node across

specs/tech-architecture/web-dashboard.md

@@ -0,0 +1,230 @@
+# `sm121 web` — dashboard design
+
+> Plan for the `web` subcommand: an axum + SSE dashboard that manages the
+> same things the CLI manages (serve/stop/power/logs/health/raw config),
+> with LED status, toggles, live log windows, light/dark themes, and a
+> floating-window UI. Read this before implementing; update the phase list
+> as it ships.
+
+## Purpose
+
+One command that replaces the *monitoring + action* loop of
+`health`/`logs -f`/`serve`/`stop`/`power` with a single browser surface.
+The CLI stays the source of truth and the fallback; the web layer is a thin
+RPC/event layer over the **existing** command functions — no re-implementation
+of serve/stop/power logic.
+
+**Non-goals (v1):** multi-user auth beyond an optional bearer token,
+metrics/history (no time-series storage), recipe *editing* (raw view is
+read-only; edits happen in the files), mobile layout (desktop-first).
+
+## Command
+
+```
+sm121 web [--port 3100] [--bind 127.0.0.1] [--token <secret>] [--open]
+```
+
+- `--port` / `--bind`: default `127.0.0.1:3100`. Binding non-localhost
+  **requires** `--token` (checked as `Authorization: Bearer`).
+- `--open`: open the browser (`open` on macOS, `xdg-open` otherwise).
+- Config resolution identical to the CLI (`--config` > cwd > global).
+- Runs until Ctrl-C. Exits non-zero if the port is taken.
+
+## Architecture
+
+```
+sm121 web
+└─ axum Router ── AppState { config, jobs, log_hub, bus }
+    ├─ /            embedded UI (include_str! of src/web/ui.html)
+    ├─ /api/*       REST (actions + snapshots + raw content)
+    ├─ /events      single multiplexed SSE stream
+    ├─ supervisor   tokio task: polls cluster_state::probe every 4 s →
+    │               broadcasts state snapshot on the bus
+    └─ jobs         serve/stop/power run as tracked tokio tasks that call
+                    the existing docker::serve / stop_cluster / power fns
+```
+
+Key properties:
+
+- **In-process reuse.** `POST /api/serve` calls `docker::serve(config, …)`
+  with `ServeOpts` built from the request body — the same code path the CLI
+  uses (before_serve hooks, mount clearing, NCCL, image sync all apply).
+- **One event bus.** `tokio::sync::broadcast<Event>` (cap 1024). Event =
+  `State | Job | Log`. The single `/events` SSE endpoint forwards every
+  event; the UI routes by `type`. No per-topic server state.
+- **Supervisor is read-only.** `cluster_state::probe` (what `health` does:
+  `docker ps` per node over SSH) every 4 s. Same SSH load as a manual
+  `health` loop; fine for ≤ a few nodes.
+- **Logs.** A log *attach* = one `docker logs -f` SSH session (reusing
+  `ssh_session_follow`) whose output is newline-split and pushed to the bus
+  as `Log` events, plus a per-attach ring buffer (2000 lines) so new
+  windows can fetch history via `GET /api/logs?lines=N` before attaching.
+  Cap 8 concurrent attaches (ponytail: per-attach SSH sessions; queue if
+  this ever matters). `run_ssh_follow` gets a callback-based sibling
+  (`run_ssh_stream(config, node, cmd, on_chunk)`) — the existing fn becomes
+  a wrapper, CLI behavior unchanged.
+
+## API
+
+### REST
+
+| Method | Path | Purpose |
+|---|---|---|
+| GET | `/api/state` | full snapshot (clusters, nodes, containers, jobs, log-attaches) |
+| POST | `/api/serve` | `{cluster, recipe, solo?, force?, fit?, clear?, mirror?, dry_run?}` → `{job}` |
+| POST | `/api/stop` | `{cluster, recipe, node?}` → `{job}` |
+| POST | `/api/power` | `{action: off\|on\|switch, cluster, from?, to?}` → `{job}` — **confirm required** (see safety) |
+| POST | `/api/logs` | attach `{node, recipe, rank?}` → `{attach_id}` (max 8) |
+| DELETE | `/api/logs` | detach `{attach_id}` |
+| GET | `/api/logs?node=&recipe=&lines=200` | ring-buffer history for a fresh window |
+| GET | `/api/recipes` | recipe names (from all `recipe_sources`) + deploy target |
+| GET | `/api/raw/config` | active config file, verbatim |
+| GET | `/api/raw/recipe/<name>` | recipe.toml, verbatim |
+| GET | `/api/raw/effective` | merged `Config` as JSON (post cluster-source merge) |
+
+Errors: `{ "error": "…" }` with 4xx/5xx; a running job for the same
+(cluster, recipe) returns 409 on a second serve/stop (single-flight).
+
+### SSE (`/events`)
+
+Newline-delimited JSON, one event per `data:`:
+
+```json
+{"type":"state","ts":1690000000,"snapshot":{…}}
+{"type":"job","id":"a1","event":"started|progress|ok|failed","cluster":"qwen38","recipe":"…","message":"…"}
+{"type":"log","attach":"l2","node":"dgx1.lan","recipe":"qwen38-27b-nvfp4-unsloth","rank":0,"line":"…"}
+```
+
+- State events: every supervisor tick (4 s) and immediately after any job
+  state change.
+- Reconnect: browser `EventSource` auto-reconnects; on open the client
+  re-fetches `GET /api/state` (SSE is fire-and-forget, not a log).
+- Log lines are split server-side on `\n`; a chunk without a trailing
+  newline is buffered (vLLM banner lines are long but single-line).
+
+## Jobs
+
+```rust
+struct Job { id, kind: Serve|Stop|Power, cluster, recipe: Option<String>,
+             status: Running|Ok|Failed, started: Instant, message: String }
+```
+
+- Registry: `Arc<Mutex<HashMap<String, Job>>>` in `AppState` (lazy: no
+  external map crate).
+- Each action spawns a tokio task that (a) inserts `Running`, (b) calls the
+  existing fn, (c) sets `Ok`/`Failed` + emits a `Job` event.
+- **Process-death semantics:** if `sm121 web` dies mid-job, the job state is
+  lost but the containers/actions are unaffected (docker + SSH are the real
+  side effects). `sm121 health` / `/api/state` is the source of truth.
+- `dry_run: true` on serve is exposed (a "dry-run preview" toggle) — it
+  returns the rendered `docker run` instead of launching; the raw view shows
+  it.
+
+## UI (single embedded file, no build step)
+
+`src/web/ui.html` — one HTML file (HTML + CSS + vanilla JS), embedded via
+`include_str!` and served at `/`. No npm, no bundler, no static assets:
+CSS transitions/keyframes cover the animation budget; `EventSource` is
+native. A build-toolchain frontend is deliberately **not** chosen — the UI
+is a status board, not an app with a component graph. (Revisit only if the
+UI outgrows ~2–3k lines.)
+
+### Layout & windows
+
+- **Window manager (vanilla JS):** absolutely-positioned window divs with a
+  titlebar (drag), corner resize handle, close, minimize-to-dock. Z-order by
+  click-to-front; new windows scale-in. A dock bar (bottom) holds minimize
+  buttons + a "new window" menu (State / Logs / Jobs / Raw Config / Raw
+  Recipe).
+- **Window kinds** (parameterized views):
+  - `state` — the LED grid (default, opens at start).
+  - `logs {node, recipe, rank}` — live stream + history + pause + raw-copy.
+  - `jobs` — in-flight jobs with progress lines.
+  - `raw {kind: config|recipe:name|effective}` — verbatim content, monospace,
+    copy button, JSON pretty for `effective`.
+- **LED grid:** rows = clusters → nodes; each node row: power LED,
+  occupancy LED, per-container LEDs (Up / Created / Exited), and action
+  toggles (serve/stop). Click a node → spawn its logs window. Colors:
+  green/red/amber (running/failed/starting), gray (off).
+- **Toggles:** serve/stop (with a two-state switch showing the job
+  transition), power (guarded — see safety), log follow per window, theme.
+
+### Theming & animation
+
+- `light` / `dark` via a `data-theme` attribute; default `prefers-color-scheme`,
+  choice persisted in `localStorage`. One CSS custom-property set per theme
+  (`--bg`, `--fg`, `--led-*`, `--accent`, …) — no second stylesheet.
+- Animation budget (CSS-only): LED glow pulse while a job runs; switch
+  spring (transform transition); window open/close scale+fade; log-line
+  fade-in; amber "attention" pulse on failed container; skeleton shimmer
+  while `/api/state` is pending.
+
+## Safety rails (non-negotiable)
+
+The AGENTS.md live-traffic warning applies to the UI with extra force — a
+toggle is one click away:
+
+- **`power off` / `power switch`:** two-step confirm — the window shows the
+  current health of the target cluster first, and requires the user to type
+  the cluster name to arm. No silent power-off.
+- **`stop`:** confirm dialog naming the container(s) that will be killed
+  (from the current snapshot).
+- **`serve`:** occupancy warnings surface in the job window (the CLI
+  already bails on occupied nodes unless `force`; the `force` checkbox is
+  labeled as such and default-off).
+- Bind-to-localhost default + optional bearer token (above) — the board can
+  act on the cluster, so it must not be an open LAN endpoint.
+
+## Module layout (≤500 lines each)
+
+| File | Contents |
+|---|---|
+| `src/web/mod.rs` | `run(config, opts)` — router, AppState, supervisor spawn |
+| `src/web/events.rs` | `Event` enum, bus, `/events` SSE handler, state snapshot builder |
+| `src/web/jobs.rs` | Job registry, single-flight, action spawning (serve/stop/power) |
+| `src/web/logs.rs` | attach/detach, ring buffer, `run_ssh_stream` wiring |
+| `src/web/api.rs` | REST handlers + raw-content endpoints |
+| `src/web/ui.html` | the UI (embedded; exempt from the 500-line rule — not Rust) |
+| `src/ssh.rs` | + `run_ssh_stream(config, node, cmd, on_chunk)` (~15 lines) |
+| `src/cli.rs` / `src/main.rs` | `Web` subcommand + dispatch arm |
+
+## New dependencies
+
+- `axum = { version = "0.8", features = ["sse"] }` — the only addition
+  (`futures`, `serde_json`, `tokio` already present; same-origin → no
+  `tower-http`).
+
+## Phases
+
+1. **P1 — read-only board.** axum server, `/api/state`, supervisor poll,
+   `/events` SSE (state only), UI shell: theme toggle + LED grid in a
+   window. *Ship test: dashboard reflects a live serve/stop done from the
+   CLI within ~4 s.*
+2. **P2 — actions.** jobs registry, serve/stop/power POSTs, confirm flows,
+   jobs window, job LEDs, dry-run preview.
+3. **P3 — logs & windows.** log attaches + ring buffers + SSE log events,
+   window manager polish (resize/minimize/dock), "new window" flows from
+   the LED grid.
+4. **P4 — raw & polish.** raw content windows (config/recipe/effective),
+   copy buttons, animation pass, `--open`.
+
+## Testing
+
+- Unit: event bus broadcast/late-joiner semantics; job single-flight
+  (409); ring-buffer wrap + `?lines=` slice; `run_ssh_stream` newline
+  chunking (fake stream).
+- Integration (temp config, fake cluster, no SSH): boot server on an
+  ephemeral port → `GET /api/state` 200 + JSON shape; `POST /api/serve
+  {dry_run: true}` → job `ok` with rendered command; `GET
+  /api/raw/effective` reflects a `cluster_sources` merge; token enforced
+  when `--bind 0.0.0.0`.
+- UI: no automation in v1 — the phase-ship tests above + manual checklist
+  (theme persists, window drag/resize, SSE reconnect after network blip,
+  power confirm can't be armed without typing the name).
+
+## Decisions (locked 2026-08-22)
+
+1. **UI stack:** single embedded vanilla file (`src/web/ui.html`) — confirmed.
+   A build-toolchain frontend is only worth it if the UI grows a real
+   component graph (revisit at ~3k lines).
+2. **Default port:** `3100` (next to the 30000 serving ports).