Browse Source

report panel: bell popup anchored, sync results as notifications, click-to-highlight, real uptime for fleet rows

- The notification popup lived in the toolbar row with right:0, so it opened at
  the far edge instead of under the bell. It is now placed off the bell's own
  box (fixed, clamped to the viewport) and follows a resize.
- Repo/sync messages no longer sit in the toolbar forever: they go to the bell
  as notifications (failures stay, with the git error). Startup state only
  speaks up when something is wrong.
- Notifications say what changed ('new recipe · <model>', 'started',
  'stopped', bench/config/removal), and run-state changes are no longer also
  reported as a config change.
- Clicking a notification now finds the recipe: filters/page reset if needed,
  row expanded, scrolled into view and flashed (3.2 s).
- Uptime was empty on any machine without a run ledger (the whole add-on): the
  fleet probe now carries each container's CreatedAt (parsed with its own
  offset), the row gets fleet_since, and the column shows a real uptime with an
  'up since' tooltip.
Alexander Korolev 1 hour ago
parent
commit
e450441d44

+ 46 - 0
specs/tech-architecture/dsv41-1017-tuning.md

@@ -245,3 +245,49 @@ write the serving layer), not an evening. What *is* worth stealing:
   if we ever need reproducible evals.
 
 Re-check when the repo grows a serving path or an sm120/sm121 target.
+
+## Внешние контракты (2026-10-23): FLARE `gb10`-пакет и рецепт eugr
+
+### local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE
+
+У репозитория есть **ветка `gb10`** (README: `/raw/gb10/README.md`) — подписанный
+FLARE-пакет ровно под наши четыре Spark:
+
+- `deepseek-v41-flash-1017-tp4-c16-dspark5.safetensors`: metadata-only первичный
+  файл + **17 шардов ≈ 511.3 GB** (12 весовых + 5 графовых/вспомогательных,
+  каждый < 64 GB), lossless CSF-чекпойнт; `gb202` — отдельный пакет под 4
+  дискретные GPU на одной машине;
+- заявлено: TP4, **16 одновременных запросов**, DSpark до **5 драфтов с
+  adaptive verification**, prefill чанками **4096**, контекст **1,048,576** на
+  запрос (по KV), >= 515 GB на первом Spark;
+- рекомендованный набор флагов (это то, что надо воспроизвести в наших лейнах):
+
+| флаг FLARE | наш аналог в vLLM | у нас сейчас |
+|---|---|---|
+| `--embedding-table-storage disk` | `--engram-config '{"table_memory":"disk"}'` | ✅ есть |
+| `--embedding-table-resident-scales` | `disk_resident_scales: true` в том же JSON (≈ 5.72 GiB RAM на V4.1) | ❌ **не включено — A/B** |
+| `--embedding-table-cache-size 1GiB` | строкового кэша строк в `EngramConfig` нет (только `table_memory`/`disk_resident_scales`/`projection_tp`) | — |
+| `--max-concurrency 16` | `--max-num-seqs 16` (у нас 4) | ❌ 4 |
+| `--mtp-draft-tokens 5 --mtp-adaptive-verification` | `--speculative-config dspark k=5` (у нас без `enable_adaptive_verification`) | частично |
+
+Вывод: `disk_resident_scales` — наш аналог «resident scales», он прямо обещан
+в конфиге стокового vLLM 0.32 (`vllm/config/engram.py`), и это следующий
+дешёвый шаг тюнинга энгрэма (скейлы E8M0 не читаются с диска на каждом
+промахе; ~5.7 GiB RAM на ноде).
+
+### Рецепт eugr (`eugr/spark-vllm-docker`, `recipes/deepseek-v4-flash.yaml`)
+
+1017-рецепта у него нет (только не-1017 `deepseek-v4-flash` на TP2), но форма
+полезна как чек-лист того, чего в нашем лейне нет:
+
+- `--tokenizer-mode deepseek_v4` — ❗ у нас не задан;
+- `--reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}'`
+  плюс `--default-chat-template-kwargs.thinking=true` /
+  `.reasoning_effort=high` — ❗ у нас только `--reasoning-parser`;
+- env `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1` (у него `max_model_len 500000` при
+  GMU 0.8) и `VLLM_USE_BREAKABLE_CUDAGRAPH=1`, `DG_JIT_USE_NVRTC=0`;
+- спека `{"method":"mtp","num_speculative_tokens":2}` — у нас dspark k=5;
+- бэкенд `ray`, у нас `mp`.
+
+Проверять по одному, с замером: `--tokenizer-mode`, `--default-chat-template-kwargs`,
+`disk_resident_scales`, затем `--max-num-seqs 16`.

+ 5 - 0
src/report/mod.rs

@@ -119,6 +119,10 @@ pub(crate) struct ReportRow {
     /// (fleet), so a remote serve is visible without a local .lock.
     #[serde(default, skip_serializing_if = "is_false")]
     fleet_up: bool,
+    /// Container creation time (unix secs) for a `fleet_up` row — the uptime
+    /// column's only source on a machine without a run ledger.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    fleet_since: Option<u64>,
     /// Family / model / per-TP medal ranks (see [`quality::Medals`]).
     #[serde(default)]
     medals: quality::Medals,
@@ -330,6 +334,7 @@ pub(crate) fn collect(config: &Config, show_archived: bool) -> Vec<ReportRow> {
                 stale: false,
                 runs: lock.total,
                 fleet_up: false,
+                fleet_since: None,
                 medals: quality::Medals::default(),
                 reports,
             });

+ 113 - 36
src/report/page.html

@@ -51,10 +51,25 @@
   #bell svg { vertical-align: -3px; }
   #bellcount { position: absolute; top: -7px; right: -7px; min-width: 17px; height: 17px; line-height: 17px; text-align: center;
                font-size: 10px; font-weight: 700; color: #fff; background: #c0392b; border-radius: 9px; padding: 0 3px; }
-  #notifbox { position: absolute; z-index: 6; right: 0; top: 100%; width: min(460px, 90vw); max-height: 60vh; overflow: auto;
-              background: canvas; border: 1px solid color-mix(in srgb, currentColor 25%, transparent); border-radius: 5px; padding: 6px; }
+  /* Anchored to the bell by JS (#bar is a full-width flex row, so a plain
+     right:0 put the popup at the far edge of the toolbar, not under the
+     bell). position:fixed keeps it out of any parent overflow/transform. */
+  #notifbox { position: fixed; z-index: 30; width: min(460px, 92vw); max-height: 60vh; overflow: auto;
+              background: canvas; color: canvastext; border: 1px solid color-mix(in srgb, currentColor 25%, transparent);
+              border-radius: 6px; padding: 6px; box-shadow: 0 8px 26px rgba(0, 0, 0, .28); }
   #bar { position: relative; }
   .noti { display: flex; gap: 8px; padding: 5px 6px; border-bottom: 1px solid color-mix(in srgb, currentColor 12%, transparent); cursor: pointer; }
+  .noti.warn .nic { color: #d2822a; }
+  .noti .sub { font-size: 11px; }
+  /* A recipe picked from the notification list: flashed so the eye finds it
+     in a long table (the row may be far down the page). */
+  @keyframes hitflash {
+    0% { background: color-mix(in srgb, #f0a030 42%, transparent); }
+    60% { background: color-mix(in srgb, #f0a030 26%, transparent); }
+    100% { background: transparent; }
+  }
+  tr.row.hit > td { animation: hitflash 3.2s ease-out; }
+  tr.row.hit > td:first-child { box-shadow: inset 3px 0 0 #f0a030; }
   .noti:hover { background: color-mix(in srgb, currentColor 6%, canvas); }
   .noti:last-child { border-bottom: none; }
   .nic { font-weight: 700; }
@@ -122,7 +137,6 @@
   </label>
   <button id="prev">‹</button><span id="count"></span><button id="next">›</button>
   <button id="syncbtn" title="sync the recipe repos now (dashboard job; new recipes appear via the live stream)" style="display:none">⟳ sync</button>
-  <span id="repostat" class="dim" style="display:none"></span>
   <span id="live" class="dim" title="live updates">● off</span>
   <button id="bell" title="updates" aria-label="updates">
     <svg width="17" height="17" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M18 8a6 6 0 0 0-12 0c0 7-3 9-3 9h18s-3-2-3-9"/><path d="M13.7 21a2 2 0 0 1-3.4 0"/></svg>
@@ -232,7 +246,10 @@ function cell(r, k) {
   if (k === "pct") { const e = latest(r, e => e.pct != null); return e ? e.pct : ""; }
   if (k === "tps1" || k === "tps2") { const e = latest(r, e => e.perf && e.perf.length); return tg(e, 0, k === "tps1" ? 1 : 2) ?? ""; }
   if (k === "stars") { const si = starInfo(r); return si ? si.pct : -1; } // sort by the absolute score
-  if (k === "state" || k === "uptime") return r.up != null ? nowSecs() - r.up : r.fleet_up ? 0 : -1; // live uptime secs (fleet-up: unknown start)
+  if (k === "state" || k === "uptime") {
+    const since = r.up != null ? r.up : r.fleet_since; // ledger, else the container's own start
+    return since != null ? nowSecs() - since : -1;
+  }
   if (k === "runs") return r.runs ?? 0;
   return r[k];
 }
@@ -246,17 +263,30 @@ function cell(r, k) {
 function notifDiff(prev, next) {
   const out = [];
   const byName = new Map(prev.map(r => [r.name, r]));
-  const meta = r => JSON.stringify({ ...r, reports: [] });
+  // Run state (up / fleet_up / fleet_since / stale) has its own notification
+  // kinds — leaving it in the meta blob would double-report a serve as a
+  // "config changed" as well.
+  const meta = r => JSON.stringify({
+    ...r, reports: [], up: undefined, up_cluster: undefined,
+    fleet_up: undefined, fleet_since: undefined, stale: undefined, runs: undefined,
+  });
   const item = (t, r, e) => ({
     t, name: r.name,
-    label: e ? e.label : (t === "recipe" ? (r.model || r.title) : "recipe config"),
+    label: e ? e.label
+      : t === "recipe" ? `new recipe · ${r.model || r.title}`
+      : t === "up" ? "started"
+      : t === "down" ? "stopped"
+      : "recipe config changed",
     date: e ? e.date : "", time: e ? e.time : null,
     score: e ? e.pct : null, points: e ? e.points : null,
     files: e ? (e.files || [e.file]) : [],
   });
+  const running = r => r.up != null || r.fleet_up;
   for (const r of next) {
     const old = byName.get(r.name);
     if (!old) { out.push(item("recipe", r, null)); continue; }
+    if (!running(old) && running(r)) out.push(item("up", r, null));
+    else if (running(old) && !running(r)) out.push(item("down", r, null));
     if (meta(old) !== meta(r)) out.push(item("config", r, null));
     const oldR = new Map(old.reports.map(e => [e.date + "\u0000" + e.label, e]));
     for (const e of r.reports) {
@@ -294,12 +324,21 @@ const bellEl = document.getElementById("bell");
 const notifBox = document.getElementById("notifbox");
 let UNREAD = 0;
 let NOTIFS = [];
-const NICON = { bench: "+", change: "↻", recipe: "★", config: "✎", removed: "−" };
+const NICON = { bench: "+", change: "↻", recipe: "★", config: "✎", removed: "−",
+                up: "▶", down: "⏹", sync: "⟳" };
 function renderBell() {
   const c = document.getElementById("bellcount");
   c.textContent = String(UNREAD);
   c.style.display = UNREAD ? "" : "none";
 }
+// Push a non-dataset notification (repo sync result, config warning …).
+function notify(t, name, label, warn) {
+  NOTIFS.unshift({ t, name: name || "", label, warn: !!warn, files: [] });
+  if (NOTIFS.length > 100) NOTIFS.length = 100;
+  UNREAD += 1;
+  renderBell(); renderNotifs();
+  bellEl.classList.remove("ring"); void bellEl.offsetWidth; bellEl.classList.add("ring");
+}
 function renderNotifs() {
   if (!NOTIFS.length) {
     notifBox.innerHTML = '<div class="dim" style="padding:8px">no updates yet</div>';
@@ -310,26 +349,64 @@ function renderNotifs() {
     const score = n.score != null ? `${n.score} /100` : n.points || "";
     const links = n.files.map(f =>
       `<a class="open" target="_blank" href="${auth("/r/" + encodeURIComponent(n.name) + "/" + encodeURIComponent(f))}" title="open ${esc(f)}">${esc(f.split(".").pop())}</a>`).join(" ");
-    return `<div class="noti" data-name="${esc(n.name)}" title="show in table">
+    const cls = "noti" + (n.warn ? " warn" : "");
+    return `<div class="${cls}" data-name="${esc(n.name)}" title="${n.name ? "show in table" : ""}">
       <span class="nic">${NICON[n.t] || "•"}</span>
       <div><b>${esc(n.name)}</b> <span class="dim">${esc(n.label)}</span><br>
       <span class="dim">${esc(when)}</span> ${score ? esc(score) : ""} ${links}</div>
     </div>`;
   }).join("");
 }
+// The popup hangs off the bell's own box, clamped to the viewport.
+function placeNotifs() {
+  const r = bellEl.getBoundingClientRect();
+  const w = Math.min(460, window.innerWidth * 0.92);
+  notifBox.style.width = w + "px";
+  const left = Math.max(8, Math.min(r.right - w, window.innerWidth - w - 8));
+  const below = window.innerHeight - r.bottom - 16;
+  const top = below < 220 ? Math.max(8, r.top - Math.min(360, window.innerHeight * 0.6)) : r.bottom + 6;
+  notifBox.style.left = left + "px";
+  notifBox.style.top = top + "px";
+}
 bellEl.addEventListener("click", e => {
   e.stopPropagation();
   const open = notifBox.style.display === "block";
-  notifBox.style.display = open ? "none" : "block";
-  if (!open) { UNREAD = 0; renderBell(); }
+  if (open) { notifBox.style.display = "none"; return; }
+  placeNotifs();
+  notifBox.style.display = "block";
+  UNREAD = 0; renderBell();
+});
+window.addEventListener("resize", () => {
+  if (notifBox.style.display === "block") placeNotifs();
 });
+
+// Pick a recipe out of the notification list: make sure the row is on
+// screen (drop a filter/page that hides it), expand it, and flash it.
+function focusRecipe(name) {
+  if (!DATA.some(r => r.name === name)) return;
+  S.openRow = name;
+  render();
+  let row = document.querySelector(`tr.row[data-n="${cssEsc(name)}"]`);
+  if (!row) {
+    S.q = ""; S.uponly = ""; S.tp = ""; S.medal = ""; S.page = 0;
+    const q = document.getElementById("q"); if (q) q.value = "";
+    render();
+    row = document.querySelector(`tr.row[data-n="${cssEsc(name)}"]`);
+  }
+  if (!row) return; // still hidden by a collapsed group — the row is expanded
+  row.classList.remove("hit"); void row.offsetWidth; row.classList.add("hit");
+  row.scrollIntoView({ block: "center", behavior: "smooth" });
+  setTimeout(() => row.classList.remove("hit"), 3400);
+}
+const cssEsc = s => (window.CSS && CSS.escape ? CSS.escape(s) : s.replace(/["\\]/g, "\\$&"));
 document.addEventListener("click", () => { notifBox.style.display = "none"; });
 notifBox.addEventListener("click", e => {
+  if (e.target.closest("a")) return; // report links open on their own
   const it = e.target.closest(".noti");
   if (!it) return;
-  S.openRow = it.dataset.name;
+  const name = it.dataset.name;
   notifBox.style.display = "none";
-  render();
+  if (name) focusRecipe(name);
 });
 
 let es = null;
@@ -521,10 +598,14 @@ function render() {
               ? `<span title="ledger says running but no live container found (node down? killed serve?) — verify and close: sm121 runlock close ${esc(r.name)}"><span style="color:#d2822a">●</span><span class="dim"> stale</span></span>`
               : `<span title="up since ${fmtSince(r.up)} UTC"><span style="color:green">●</span><span class="dim"> up</span></span>`)
           : (r.fleet_up
-              ? `<span title="running in the fleet — a container was found on a cluster node, but this machine has no local run ledger (served by another launcher)"><span style="color:green">●</span><span class="dim"> up</span><span class="dim" style="font-size:.8em">(fleet)</span></span>`
+              ? `<span title="running in the fleet — a container was found on a cluster node, but this machine has no local run ledger (served by another launcher)${r.fleet_since ? `; started ${fmtSince(r.fleet_since)} UTC` : ""}"><span style="color:green">●</span><span class="dim"> up</span><span class="dim" style="font-size:.8em">(fleet)</span></span>`
               : '<span class="dim" title="not running">○</span>');
       } else if (c[0] === "uptime") {
-        body = r.up != null ? fmtUp(v) : '<span class="dim">—</span>';
+        // `up` (ledger) or the container's creation time (fleet): a serve
+        // with no local .lock still has a real uptime.
+        body = v >= 0
+          ? `<span title="up since ${fmtSince(r.up != null ? r.up : r.fleet_since)} UTC">${fmtUp(v)}</span>`
+          : '<span class="dim" title="not running">—</span>';
       } else if (c[0] === "runs") {
         body = v > 0 ? esc(v)
           : '<span class="dim" title="no launch ledger on this machine (or never launched)">—</span>';
@@ -732,33 +813,27 @@ document.getElementById("mscopy").addEventListener("click", async (e) => {
         setTimeout(() => { e.target.textContent = "copy"; }, 1200); } catch { /* clipboard blocked */ }
 });
 const syncBtn = document.getElementById("syncbtn");
-const repoStat = document.getElementById("repostat");
-// An empty recipe table must say WHY: no repo configured, an archived repo
-// and a repo whose clone keeps failing all render the same otherwise.
-function showRepoStat(text, bad) {
-  if (!repoStat) return;
-  repoStat.style.display = text ? "" : "none";
-  repoStat.textContent = text;
-  repoStat.style.color = bad ? "#e8833a" : "";
-  repoStat.title = text;
-}
-async function loadRepoStatus() {
+// Repo/sync state arrives as notifications, not as a permanent toolbar text:
+// the panel stays quiet when everything is fine, and a failure stays in the
+// list (with the git error) instead of blinking away.
+async function reportRepoState(quietWhenFine) {
   try {
     const j = await (await fetch("/api/repos")).json();
     if (!j.configured) {
-      // Cached clones still serve recipes even with an empty [[recipe_repos]].
-      if (j.cached_roots) showRepoStat(`repos: ${j.cached_roots} cached (none configured)`, false);
-      else showRepoStat("⚠ no recipe repos configured", true);
+      notify("sync", "", "no recipe repos configured (set sm121_recipe_repos)", true);
       return;
     }
     const arch = (j.repos || []).filter(r => r.archived).length;
     const bad = (j.repos || []).filter(r => !r.cached && !r.archived).length;
-    if (bad) { showRepoStat(`⚠ ${bad}/${j.configured} repo(s) not synced`, true); return; }
-    if (arch) { showRepoStat(`repos: ${j.configured - arch}/${j.configured} synced (${arch} archived)`, false); return; }
-    showRepoStat(`repos: ${j.configured} synced`, false);
-  } catch { /* hosted mode has no /api/repos (standalone report) */ }
+    if (bad) notify("sync", "", `${bad} of ${j.configured} recipe repo(s) not synced yet`, true);
+    else if (!quietWhenFine)
+      notify("sync", "", `${j.configured - arch} repo(s) synced${arch ? ` (${arch} archived)` : ""}`);
+  } catch { /* standalone report: no /api/repos */ }
+}
+if (EMBED) {
+  syncBtn.style.display = "";
+  reportRepoState(false);
 }
-if (EMBED) { syncBtn.style.display = ""; loadRepoStatus(); }
 syncBtn.addEventListener("click", async () => {
   syncBtn.disabled = true; syncBtn.textContent = "⟳ …";
   try {
@@ -767,9 +842,11 @@ syncBtn.addEventListener("click", async () => {
     const bad = !j.ok;
     syncBtn.textContent = bad ? "⟳ ✗" : `⟳ ${j.repos} ok`;
     syncBtn.title = j.detail || (bad ? String(res.status) : "recipe repos synced");
-    showRepoStat(j.detail || "", bad);
-    if (!bad) loadRepoStatus();
-  } catch (err) { syncBtn.textContent = "⟳ ✗"; syncBtn.title = String(err); showRepoStat(String(err), true); }
+    notify("sync", "", j.detail || (bad ? String(res.status) : "recipe repos synced"), bad);
+  } catch (err) {
+    syncBtn.textContent = "⟳ ✗"; syncBtn.title = String(err);
+    notify("sync", "", String(err), true);
+  }
   setTimeout(() => { syncBtn.disabled = false; syncBtn.textContent = "⟳ sync"; }, 6000);
 });
 // column checkboxes

+ 22 - 0
src/report/report.playwright.mjs

@@ -51,6 +51,17 @@ ok("sse connected (● live)", true);
 await page.click("#bell");
 await page.waitForSelector("#notifbox", { state: "visible" });
 ok("bell opens the panel", await page.isVisible("#notifbox"));
+// the popup must hang off the bell itself, not the far edge of the toolbar
+const pos = await page.evaluate(() => {
+  const b = document.getElementById("bell").getBoundingClientRect();
+  const n = document.getElementById("notifbox").getBoundingClientRect();
+  return { dLeft: Math.abs(n.left - b.left), dRight: Math.abs(n.right - b.right), offscreen: n.right > innerWidth + 1 || n.left < -1 };
+});
+ok(
+  "bell popup anchored to the bell",
+  !pos.offscreen && Math.min(pos.dLeft, pos.dRight) < 220,
+  JSON.stringify(pos),
+);
 ok("empty panel says so", (await page.textContent("#notifbox")).includes("no updates yet"));
 await page.click("#bell"); // close again
 
@@ -80,6 +91,17 @@ ok(
   "panel lists the new bench",
   (await page.textContent("#notifbox")).includes(label),
 );
+// clicking an item focuses its recipe in the table (and flashes it)
+const hit = await page.evaluate(() => !!document.querySelector("#notifbox .noti[data-name]"));
+if (hit) {
+  await page.click("#notifbox .noti[data-name]");
+  const flashed = await page.waitForFunction(
+    () => !!document.querySelector("tr.row.hit"),
+    null,
+    { timeout: 4000 },
+  ).then(() => true).catch(() => false);
+  ok("notification click highlights the recipe row", flashed);
+}
 ok(
   "badge reset after opening",
   await page.evaluate(() => document.getElementById("bellcount").style.display === "none"),

+ 1 - 0
src/report/serve_tests.rs

@@ -22,6 +22,7 @@ fn row(name: &str, up: Option<i64>) -> ReportRow {
         up_cluster: None,
         runs: 0,
         fleet_up: false,
+        fleet_since: None,
         medals: Medals::default(),
         stale: false,
         reports: Vec::new(),

+ 33 - 11
src/report/stale.rs

@@ -62,10 +62,17 @@ pub(crate) fn mark_stale(
 /// start time); `stale` and `fleet_up` are mutually exclusive by
 /// construction (stale = ledger open + no container; fleet_up = no ledger +
 /// container present).
-pub(crate) fn mark_fleet_up(rows: &mut [ReportRow], live_recipes: &HashSet<String>) {
+pub(crate) fn mark_fleet_up(rows: &mut [ReportRow], live_recipes: &HashMap<String, Option<u64>>) {
     for row in rows {
         // Recomputed every pass, so a stopped container clears it again.
-        row.fleet_up = row.up.is_none() && !row.stale && live_recipes.contains(&row.name);
+        row.fleet_up = row.up.is_none() && !row.stale && live_recipes.contains_key(&row.name);
+        // The container's own creation time is the start time when there is
+        // no ledger to ask (the add-on has none) — the uptime column needs it.
+        row.fleet_since = if row.fleet_up {
+            live_recipes.get(&row.name).copied().flatten()
+        } else {
+            None
+        };
     }
 }
 
@@ -101,12 +108,19 @@ pub(crate) async fn cross_check(
     let (reachable, live) = match fleet {
         Some(snap) => {
             let mut reachable: HashMap<String, bool> = HashMap::new();
-            let mut live: HashSet<String> = HashSet::new();
+            let mut live: HashMap<String, Option<u64>> = HashMap::new();
             for c in &snap.clusters {
                 for n in &c.nodes {
                     reachable.insert(n.name.clone(), n.reachable);
                     for ct in &n.containers {
-                        live.insert(ct.recipe.clone());
+                        // earliest container wins: every rank of a serve
+                        // starts within seconds of the others
+                        let e = live.entry(ct.recipe.clone()).or_insert(ct.created);
+                        if let (Some(a), Some(b)) = (*e, ct.created)
+                            && b < a
+                        {
+                            *e = Some(b);
+                        }
                     }
                 }
             }
@@ -115,17 +129,18 @@ pub(crate) async fn cross_check(
         None => {
             let f = crate::web::probe::probe(cfg).await;
             let mut reachable: HashMap<String, bool> = HashMap::new();
-            let mut live: HashSet<String> = HashSet::new();
+            let mut live: HashMap<String, Option<u64>> = HashMap::new();
             for (node, st) in &f.nodes {
                 reachable.insert(node.clone(), st.reachable);
                 for c in &st.containers {
-                    live.insert(c.recipe.clone());
+                    live.insert(c.recipe.clone(), c.created);
                 }
             }
             (reachable, live)
         }
     };
-    mark_stale(rows, cluster_nodes, &reachable, &live);
+    let live_names: HashSet<String> = live.keys().cloned().collect();
+    mark_stale(rows, cluster_nodes, &reachable, &live_names);
     // The inverse direction: a container in the fleet with NO local ledger
     // run (the add-on machine) still renders as running.
     mark_fleet_up(rows, &live);
@@ -154,6 +169,7 @@ mod tests {
             up_cluster: up_cluster.map(String::from),
             runs: 1,
             fleet_up: false,
+            fleet_since: None,
             medals: Medals::default(),
             stale: false,
             reports: vec![],
@@ -204,7 +220,10 @@ mod tests {
     #[test]
     fn fleet_container_without_ledger_marks_fleet_up() {
         let mut rows = vec![row("r-a", None, None), row("r-b", None, None)];
-        mark_fleet_up(&mut rows, &HashSet::from(["r-a".to_string()]));
+        mark_fleet_up(
+            &mut rows,
+            &HashMap::from([("r-a".to_string(), Some(1_700_000_000))]),
+        );
         assert!(rows[0].fleet_up, "live container, no ledger -> fleet_up");
         assert!(!rows[1].fleet_up);
     }
@@ -214,7 +233,10 @@ mod tests {
         // a ledger run (with or without a live container) keeps its own
         // state: green "up" or stale — fleet_up only fills the ledger gap
         let mut rows = vec![row("r-a", Some(1000), Some("c1"))];
-        mark_fleet_up(&mut rows, &HashSet::from(["r-a".to_string()]));
+        mark_fleet_up(
+            &mut rows,
+            &HashMap::from([("r-a".to_string(), Some(1_700_000_000))]),
+        );
         assert!(!rows[0].fleet_up);
     }
 
@@ -233,9 +255,9 @@ mod tests {
         assert!(!rows[0].stale, "stale clears once the lane is live again");
 
         let mut rows = vec![row("r-b", None, None)];
-        mark_fleet_up(&mut rows, &HashSet::from(["r-b".to_string()]));
+        mark_fleet_up(&mut rows, &HashMap::from([("r-b".to_string(), None)]));
         assert!(rows[0].fleet_up);
-        mark_fleet_up(&mut rows, &HashSet::new());
+        mark_fleet_up(&mut rows, &HashMap::new());
         assert!(
             !rows[0].fleet_up,
             "fleet_up clears when the container is gone"

+ 12 - 1
src/report/ui_tests.rs

@@ -114,7 +114,18 @@ fn page_notif_diff_js() {
          if (bi.files[0] !== \"2026-01-02_new.md\") throw new Error(\"bench files: \" + JSON.stringify(bi));
          if (d.find(x => x.t === \"config\").name !== \"r-a\") throw new Error(\"config item\");
          const ri = d.find(x => x.t === \"recipe\");
-         if (ri.name !== \"r-b\" || ri.label !== \"m/B\") throw new Error(\"recipe item: \" + JSON.stringify(ri));
+         // the label says WHAT happened, not just which model it is
+         if (ri.name !== \"r-b\" || !/^new recipe .*m\\/B$/.test(ri.label))
+           throw new Error(\"recipe item: \" + JSON.stringify(ri));
+         // 2b: run state changes get their own items (started / stopped)
+         const up0 = row(\"r-c\", \"m/C\", \"img:4\", []); up0.up = null; up0.fleet_up = false;
+         const up1 = row(\"r-c\", \"m/C\", \"img:4\", []); up1.up = 1700000000;
+         const du = notifDiff([up0], [up1]);
+         if (du.length !== 1 || du[0].t !== \"up\" || du[0].label !== \"started\")
+           throw new Error(\"up item: \" + JSON.stringify(du));
+         const dd = notifDiff([up1], [up0]);
+         if (dd.length !== 1 || dd[0].t !== \"down\" || dd[0].label !== \"stopped\")
+           throw new Error(\"down item: \" + JSON.stringify(dd));
          // 3: entry value change → change
          const a2 = row(\"r-a\", \"m/A\", \"img:1\", [rep(\"r-a\", \"old\", \"2026-01-01\", 95.0)]);
          const d2 = notifDiff([a0], [a2]);

+ 2 - 0
src/web/events.rs

@@ -257,6 +257,7 @@ mod tests {
                 reachable: true,
                 occupied: true,
                 containers: vec![ContainerSnap {
+                    created: None,
                     name: "sm121_r_0".into(),
                     status: "Up 3 hours".into(),
                     cluster: "a".into(),
@@ -309,6 +310,7 @@ mod tests {
                 reachable: true,
                 occupied: true,
                 containers: vec![ContainerSnap {
+                    created: None,
                     name: "c2".into(),
                     status: "Up".into(),
                     cluster: "a".into(),

+ 78 - 1
src/web/probe.rs

@@ -17,6 +17,58 @@ pub struct ContainerSnap {
     pub status: String,
     pub cluster: String,
     pub recipe: String,
+    /// Container creation time (unix seconds) from `docker ps` `CreatedAt`.
+    /// A serve's container is created at launch, so this is the row's start
+    /// time on a machine with no run ledger (the add-on) — the uptime column
+    /// reads it. `None` when the daemon printed something unexpected.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub created: Option<u64>,
+}
+
+/// `docker ps --format {{.CreatedAt}}` → unix seconds.
+///
+/// The daemon prints `YYYY-MM-DD HH:MM:SS ±HHMM ZONE` in its own timezone
+/// (`2026-10-23 19:34:47 +0200 CEST`), so the offset is subtracted rather
+/// than assumed to be UTC. Anything else returns `None` (the column then
+/// falls back to "—" instead of inventing a number).
+pub fn created_epoch(s: &str) -> Option<u64> {
+    // `2026-10-23 19:34:47 +0200 CEST` → date, time, offset, zone
+    let parts: Vec<&str> = s.split_whitespace().collect();
+    let (stamp, clock) = (parts.first()?, parts.get(1)?);
+    let mut d = stamp.split('-');
+    let (y, mo, da): (i64, i64, i64) = (
+        d.next()?.parse().ok()?,
+        d.next()?.parse().ok()?,
+        d.next()?.parse().ok()?,
+    );
+    let mut t = clock.split(':');
+    let (h, mi, se): (i64, i64, i64) = (
+        t.next()?.parse().ok()?,
+        t.next()?.parse().ok()?,
+        t.next()?.parse().ok()?,
+    );
+    let offset = parts
+        .get(2)
+        .and_then(|o| {
+            let neg = o.starts_with('-');
+            let v: i64 = o.get(1..5)?.parse().ok()?;
+            Some(if neg {
+                -((v / 100) * 3600 + (v % 100) * 60)
+            } else {
+                (v / 100) * 3600 + (v % 100) * 60
+            })
+        })
+        .unwrap_or(0);
+    // days since the epoch (civil-from-days, Howard Hinnant's algorithm)
+    let y2 = y - if mo <= 2 { 1 } else { 0 };
+    let era = if y2 >= 0 { y2 } else { y2 - 399 } / 400;
+    let yoe = y2 - era * 400;
+    let mp = (mo + 9) % 12;
+    let doy = (153 * mp + 2) / 5 + da - 1;
+    let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+    let days = era * 146_097 + doe - 719_468;
+    let secs = days * 86_400 + h * 3600 + mi * 60 + se - offset;
+    u64::try_from(secs).ok()
 }
 
 /// Per-node state with containers.
@@ -85,7 +137,7 @@ async fn ha_states(config: &Config) -> HashMap<String, Option<bool>> {
 /// cluster with `extra_entities`. Unreachable nodes come back as the
 /// default state (reachable: false) — probe never fails.
 pub async fn probe(config: &Config) -> Fleet {
-    const CMD: &str = r#"docker ps --filter label=sm121.cluster --format '{{.Names}}\t{{.Status}}\t{{.Label "sm121.cluster"}}\t{{.Label "sm121.recipe"}}' 2>/dev/null"#;
+    const CMD: &str = r#"docker ps --filter label=sm121.cluster --format '{{.Names}}\t{{.Status}}\t{{.Label "sm121.cluster"}}\t{{.Label "sm121.recipe"}}\t{{.CreatedAt}}' 2>/dev/null"#;
     let mut seen = std::collections::HashSet::new();
     let mut handles = Vec::new();
     for cluster in &config.clusters {
@@ -132,6 +184,7 @@ pub async fn probe(config: &Config) -> Fleet {
                             status: s.to_string(),
                             cluster: c.to_string(),
                             recipe: r.to_string(),
+                            created: f.next().and_then(created_epoch),
                         });
                     }
                 }
@@ -177,3 +230,27 @@ pub async fn probe(config: &Config) -> Fleet {
         ha: ha_states(config).await,
     }
 }
+
+#[cfg(test)]
+mod created_tests {
+    use super::created_epoch;
+
+    #[test]
+    fn created_parses_docker_stamp() {
+        // 2026-10-23T19:34:47Z, printed by a daemon in +0200
+        assert_eq!(
+            created_epoch("2026-10-23 21:34:47 +0200 CEST"),
+            Some(1_792_784_087)
+        );
+        assert_eq!(
+            created_epoch("2026-10-23 19:34:47 +0000 UTC"),
+            Some(1_792_784_087)
+        );
+    }
+
+    #[test]
+    fn created_rejects_garbage() {
+        assert_eq!(created_epoch(""), None);
+        assert_eq!(created_epoch("yesterday"), None);
+    }
+}