|
|
@@ -38,7 +38,7 @@ sm121 web
|
|
|
├─ / embedded UI (include_str! of src/web/ui.html)
|
|
|
├─ /api/* REST (actions + snapshots + raw content)
|
|
|
├─ /events single multiplexed SSE stream
|
|
|
- ├─ supervisor tokio task: polls cluster_state::probe every 4 s →
|
|
|
+ ├─ supervisor tokio task: polls cluster_state::probe every 30 s →
|
|
|
│ broadcasts state snapshot on the bus
|
|
|
└─ jobs serve/stop/power run as tracked tokio tasks that call
|
|
|
the existing docker::serve / stop_cluster / power fns
|
|
|
@@ -53,8 +53,9 @@ Key properties:
|
|
|
`State | Job | Log`. The single `/events` SSE endpoint forwards every
|
|
|
event; the UI routes by `type`. No per-topic server state.
|
|
|
- **Supervisor is read-only.** `cluster_state::probe` (what `health` does:
|
|
|
- `docker ps` per node over SSH) every 4 s. Same SSH load as a manual
|
|
|
- `health` loop; fine for ≤ a few nodes.
|
|
|
+ `docker ps` per node over SSH) every 30 s. Same SSH load as a manual
|
|
|
+ `health` loop (less); fine for ≤ a few nodes. Job-driven state changes
|
|
|
+ (P2) push immediately, so the 30 s only bounds *external* changes.
|
|
|
- **Logs.** A log *attach* = one `docker logs -f` SSH session (reusing
|
|
|
`ssh_session_follow`) whose output is newline-split and pushed to the bus
|
|
|
as `Log` events, plus a per-attach ring buffer (2000 lines) so new
|
|
|
@@ -95,7 +96,7 @@ Newline-delimited JSON, one event per `data:`:
|
|
|
{"type":"log","attach":"l2","node":"dgx1.lan","recipe":"qwen38-27b-nvfp4-unsloth","rank":0,"line":"…"}
|
|
|
```
|
|
|
|
|
|
-- State events: every supervisor tick (4 s) and immediately after any job
|
|
|
+- State events: every supervisor tick (30 s) and immediately after any job
|
|
|
state change.
|
|
|
- Reconnect: browser `EventSource` auto-reconnects; on open the client
|
|
|
re-fetches `GET /api/state` (SSE is fire-and-forget, not a log).
|