|
|
@@ -63,9 +63,21 @@ Each row was re-tested on 2026-11-16; "before" is what our docs/config said.
|
|
|
- **Installing a core is a copy, not a hardlink**:
|
|
|
`cores/<id>-<bytes>` held its own inodes (`hardlinks=1`) next to the download
|
|
|
store, so a controller that keeps both needs the payload twice.
|
|
|
-- **`orbit run --persistent`** saves the launch for restarts; a non-persistent group
|
|
|
- (both lanes today, `deepseek1017` and its predecessor) must be relaunched by hand
|
|
|
- with the full flag set after any agent restart.
|
|
|
+- **Both lanes are `--persistent` (since 2026-11-16).** The definition lives in the
|
|
|
+ controller's `orbit.toml` (`[[group]]`: core, serving flags, rank-zero flags,
|
|
|
+ rank-ordered nodes, transfer streams, publisher keys) and survives stop, history
|
|
|
+ expiry and reboot. Restart is then one short command — no flags:
|
|
|
+
|
|
|
+ ```bash
|
|
|
+ ssh dgx1.lan 'flare orbit run --name deepseek1017 --detach' # prod, dgx1-4
|
|
|
+ ssh dgx5.lan 'flare orbit run --name ds41-tp4 --detach' # sparks, dgx5-8
|
|
|
+ ```
|
|
|
+
|
|
|
+ Persistence does **not** start anything after a reboot; it only removes the need to
|
|
|
+ remember the flag set. `--dangerously-approve-key` / `--dangerously-accept-publisher`
|
|
|
+ are not stored in the definition (trust lives in `trust.toml`), which is why the
|
|
|
+ saved group starts without them. An override on a later run applies to that launch
|
|
|
+ only unless `--persistent` is passed again.
|
|
|
|
|
|
## 4. The 2026-11-16 prod window (record)
|
|
|
|
|
|
@@ -84,6 +96,15 @@ Total outage ≈ **78 s**. Verification after: `/v1/models` (`deepseek-v4.1-flas
|
|
|
8 concurrent requests all 200 (1.0–2.5 s). Rollback binaries kept as
|
|
|
`flare.orig-20261116` at both paths on each node.
|
|
|
|
|
|
+### Follow-up: both lanes made persistent (same day)
|
|
|
+
|
|
|
+`deepseek1017` was relaunched with `--persistent` (10:49:54→10:50:28Z, ~34 s), then
|
|
|
+stopped and restored with the short form `orbit run --name deepseek1017 --detach`
|
|
|
+to prove the restore path (10:50:33→10:51:27Z, ~54 s). The restored rank command
|
|
|
+line is flag-for-flag identical to the pre-window one. `ds41-tp4` was launched
|
|
|
+persistent from the start.
|
|
|
+
|
|
|
+
|
|
|
## 5. Not fixed by this upgrade
|
|
|
|
|
|
- `dgx4` still has the KingSpec XG7000-2TB 2242 whose stop-and-go prefill poisoned
|