|
|
@@ -6,18 +6,24 @@ Rust CLI (`edition 2024`) for launching vLLM/SGLang inference containers across
|
|
|
|
|
|
The add-on image builds the launcher **with `--features mimalloc`** (the CI
|
|
|
Dockerfile does this; a plain `cargo build` stays on the default allocator).
|
|
|
-On musl the default malloc never returns memory to the OS, so the web
|
|
|
-process's transient per-tick spikes (report collects, SSE serialization, log
|
|
|
-strings) ratcheted RSS to ~2.7 GiB and the Pi OOM-killed it daily
|
|
|
-(2026-10-13, dmesg `oom-kill … task=sm121`). mimalloc trims; the web
|
|
|
-process now holds ~40 MiB. Keep the add-on build on mimalloc.
|
|
|
-
|
|
|
-Related levers already in place: the report refresh loop is lazy (full
|
|
|
-re-collect only on debounced file events; the 30 s tick only re-runs the
|
|
|
-stale cross-check, riding the dashboard's fleet bus — no extra SSH), and the
|
|
|
-SSH private key is parsed once and cached (`ssh.rs`). Heap hunting:
|
|
|
-`cargo build --release --features dhat` + graceful SIGTERM exit dumps a
|
|
|
-per-callsite retained-heap `dhat.json` (summarize with `/tmp/dhat-sum.py`).
|
|
|
+The 2026-10-13/14 daily Pi OOMs (`dmesg oom-kill … task=sm121`, anon-rss
|
|
|
+~2.7 GiB) were rooted in the report panel's file watcher: `recipe_dirs()`
|
|
|
+always includes the cwd (`.`), the container cwd is `/`, and notify's
|
|
|
+recursive walk followed the `/sys` `subsystem` symlink loops — an endless
|
|
|
+walk (~4.6 GB/h allocation churn) the allocator ratcheted into RSS until
|
|
|
+OOM. Fixes: `watchable_recipe_dirs()` (watch only real recipe workspaces —
|
|
|
+never `/`), one shared HA REST client (a reqwest+rustls build per probe
|
|
|
+tick was another ~2 GB/h), the lazy report refresh (full re-collect only on
|
|
|
+debounced file events; the 30 s tick only re-runs the stale cross-check,
|
|
|
+riding the dashboard's fleet bus — no extra SSH), the SSH key parsed once
|
|
|
+and cached (`ssh.rs`), and mimalloc as the backstop. Post-fix RSS is
|
|
|
+~40 MiB flat (was 270 MiB/hour growth). Heap hunting: `cargo build --release
|
|
|
+--features dhat` + clean SIGTERM exit dumps a per-callsite retained-heap
|
|
|
+`dhat-heap.json` (summarize with `/tmp/dhat-sum.py`); the profiler must be
|
|
|
+created in `main` before anything allocates. Note: axum graceful shutdown
|
|
|
+hangs on live SSE — web closes connections on the signal instead.
|
|
|
+
|
|
|
+## Dev quick-ref
|
|
|
|
|
|
## Dev quick-ref
|
|
|
|
