launcher/CHANGELOG.md

Changelog

Semantic versioning for the launcher CLI (sm121). The add-on image has its own version and its own sm121/CHANGELOG.md; do not mix them.

0.2.6 — 2026-11-02

• FLARE package mirroring survives the new package format. Current (format 2) packages are blob stores: blobs/<sha> holds the data once and models/<version>/{model.safetensors,weights/,deployments/} are hardlinks into it. The mirror copied dirname(package) with rsync -a, so it expanded every link — a 488 GB package produced a second full copy instead of a mirror. It now mirrors the store root with rsync -aH and only prunes a flat package's dedicated directory (--delete on a store root would drop every other prepared model). A $HOME-relative local_dir also stays expandable (the shell quoting used to leave a literal $HOME).
• An already-downloaded package is adopted, not re-verified. A store filled by a plain flare download carries no mirror marker, so every serve repeated the 500 GB verification pass; the package is now resolved from flare models list --json, and only when exactly one prepared version matches <revision>-tp<N> (two candidates mean download — never guess, a wrong pick serves the wrong model silently).
• Companion probing is done inside the package's own directory in both layouts (a blob store has no weights/ at the mirror root).
• serve/stop honour the pinned FLARE controller. [flare.clusters.<c>] .controller_node (or the flat [flare].controller_node) now decides which node runs flare download / orbit run, instead of always our cluster rank 0. The two diverge as soon as the orbit is re-paired or reordered — dgx4 became the orbit's rank 0 for prod4 because its 2 TB disk is the only one that holds the 488 GB package store plus the 488 GB agent cache, and the old behaviour reached the wrong machine, failing with "… is not an authorized controller of this node".
• Orbit node names are matched by name, not by our rank. orbit_nodes lists the orbit's ranks in the orbit's order (gx10-dgx4 first when dgx4 is the orbit's rank 0), so indexing it with our rank named the wrong node (gx10-dgx3 for a dgx4 controller). A list that does not name our nodes keeps the positional fallback.
• Detail + measurements: specs/tech-architecture/flare-bringup.md ("Package distribution: CLI store vs agent cache").

0.2.5 — 2026-10-29

• Image sync now waits for the unpack. Every DGX Spark node uses the containerd snapshotter (io.containerd.snapshotter.v1), whose docker load returns before the rootfs snapshot is materialized — a docker run issued seconds after the sync can die with "unable to start container process … stat …: no such file" (dgx3 lost exactly that race on the first Mistral Large 4 serve: rank 2 died silently, --rm removed the evidence, and the rendezvous hung until its 30-min timeout). The sync now probes exec readiness on each peer (a trivial /bin/sh container start, retried up to 2 min) before declaring the sync done; a serve refuses to launch against a half-materialized image.
• Also on dgx3-class races: the container's docker run no longer races the image sync's last hop — sync-image OK now means exec-ready, not just digest-present.

0.2.4 — 2026-10-29

• Liveness checks are cached and less aggressive. Entity states are kept process-wide (HA 120 s, RouterOS 300 s); the dashboard's 30 s poll no longer re-authenticates every switch or re-fetches every HA entity on each tick. A power action invalidates the cache so the next snapshot is fresh.
• Switch liveness is a plain TCP connect, not a RouterOS login. health and the dashboard only need to know whether the switch is reachable; the RouterOS login was unnecessary API load. power off still logs in and sends /system/shutdown.
• Bench log belongs to the current job. GET /api/bench/log follows the recipe's newest bench job and only accepts a log written at/after its start; a previous campaign is labelled previous, and a fresh launch shows waiting instead of a stale log.
• Job ids survive a web restart. The id counter resumes past persisted j<hex> ids, and finish_job updates the still-running job with that id, not an older finished one.
• Bench append semantics. run_tool always appends; bench::run truncates the log once and writes launcher progress notes, so the dashboard window sees readiness/notes and the depth suite shares one report log.
• Remote bench forces a pty. ssh -tt avoids the "Pseudo-terminal will not be allocated" path when the launcher has no terminal.
• Upstream watch uses 100 commits and follows redirects.

0.2.3 — 2026-10-28

• FLARE topology/LED board: every rank node is busy. flare orbit status rows carry the orbit node and rank; the probe/health fold one pseudo-container onto each rank node, not just the controller. A 4-rank FLARE serve now shows 4 busy nodes and free 0, not one busy node and three idle.
• Shared-node dedup. A group is visible in all8/pair12 too because those paired clusters share nodes; the lane is owned by the cluster whose node count and the recipe's required_nodes match the group's rank rows. Duplicate pseudo-containers on a shared node are collapsed by (node, group).
• FLARE logs. Web log windows detect engine = "flare" and stream flare orbit attach <group> --cluster <orbit> on the controller (not Docker logs). The CLI does the same and prints both SSH stdout/stderr; run_ssh's 5 s inactivity timeout had killed the attach stream.
• GET /api/logs history and attach keys work for FLARE; the recipe-row ▤ logs opens the controller's orbit session.

0.2.2 — 2026-10-28

• Dashboard bench window. The recipes panel's ⚡ bench button now opens a bench window instead of firing the standard run blindly: choose recipe, cluster, target node (node), run host (ssh_host, e.g. pluto.lan), label, seed, concurrency, parallel, timeout, tokenizer, perf-only, depth and no-think. The choices persist in localStorage (sm121-bench-params).
• Live harness log. GET /api/bench/log?recipe=… tail-reads the newest bench log for the recipe; the window polls it while the run starts so early failures (readiness, tokenizer, harness) are visible before a long wait.
• Cancel. POST /api/bench/cancel sets the job's cancellation flag; run_tool polls it and kills the tool-eval-bench child (also the SSH child for remote runs), so the job finishes failed and the log stops.
• Reopen. The bench window is a named window: closing it leaves a dock chip, the dock menu has bench…, and the last recipe/job is restored on reload.
• Bench API now accepts node, seed, concurrency, parallel, timeout, tokenizer, no_think; the report row's bench button posts to the parent dashboard and opens the window.

0.2.1 — 2026-10-28

• FLARE cluster identification. A cluster is only paired with an orbit if the private config names it ([flare.clusters.<our-cluster>]). The flat orbit_cluster is a fallback for serve, not for the dashboard/health: our clusters share nodes (dgx1 is rank 0 of prod4, 6nodes-switch, 2nodes-dac, qwen38 and 1node-1), so probing an unpaired cluster with sparks made the sparks lane look like that cluster's lane. serve now refuses an unpaired FLARE cluster instead of silently launching onto the flat default.
• Dashboard shows the orbit. Each cluster head carries a ◆ prod4 badge (orbit + controller + live groups; dimmed when nothing runs), and the topology frame/chip labels include the orbit. health folds only the selected cluster's lane, not every lane that shares its controller.
• Prepared-package identification. After a successful mirror the controller writes .sm121-flare-package.json (repo, revision, TP, package version, local model.safetensors). A serve with a matching marker — checked against the CURRENT flare models list --json version, so a lab re-cut under the same revision is detected — skips both flare download verification and the 500 GB copy.
• Incremental mirror with rsync. The local mirror uses rsync -a --delete (marker excluded), so a changed/stale package copies only the changed files; a partial mirror resumes instead of being wiped and re-copied. Download/mirror/orbit launch all use long-running SSH (the 5 s inactivity timeout used to kill a 500 GB copy or a detached launch).
• Settings editor. POST /api/settings/extra/check validates without writing; parse errors report line/column for the textarea; Ctrl+Enter saves. The private block stays in /data/config-extra.toml, edited in the dashboard's settings window.

0.2.0 — 2026-10-28

• FLARE: cached flare download now resolves the prepared package through flare models list --json instead of failing on the metadata-only "Resolved package" line.
• FLARE: flare download gets --tensor-parallel from the recipe's launch size; a TP4 package is no longer rejected as "TP size 1".
• FLARE: local mirror copies the package directory recursively, including weights/ and deployments/ companions; it clears the destination first and runs over a long-lived SSH session (read-only package files and a 500 GB copy must not be killed by the 5 s inactivity timeout).
• FLARE: flare download and flare orbit run also use long-running SSH, so a cached-package verification or a detached launch is not cut short.
• FLARE ledger: serve records the recipe directory name, not the display name field — the old recipe.name lookup failed with recipe dir not found for 'DeepSeek-V4.1-Flash-1017 (…) '.
• FLARE gb10: production recipe updated to the 2026-10-27 package (c32, 1 GiB row cache, resident scales, --served-model-name).
• Tooling: sm121 installed with cargo install --path . --force; version now follows Cargo.toml, not a hard-coded 0.1.0.