# FLARE: fleet upgrade to main@6b68b9b4 and the corrections it forces (2026-11-16) Written after rebuilding both lanes and brain-checking the old claims against the running system. **This document supersedes specific statements in `flare-bringup.md`, `flare-storage-nas.md` and `flare-roce-poison.md`**; those files stay as history (they describe the October stand) but where they disagree with this page, this page was measured on 2026-11-16. ## 1. What is deployed | | prod lane | sparks lane | |---|---|---| | nodes | dgx1→dgx4, cluster `sparks`, group `deepseek1017` | dgx5→dgx8, cluster `dgx58`, group `ds41-tp4` | | controller | `gx10-dgx1` | `gx10-dgx5` | | API | `http://dgx1.lan:30000/v1`, alias `deepseek-v4.1-flash` | `http://dgx5.lan:30000/v1`, same alias | | core | `db20209d361f473b8452f24ad517fa0d-33966105` (ABI 10) | `edb5231cb9b847b7914a6475a5c02b3d-24324121` (ABI 14) | | model | `checkpoint-expanded-scales` (local-inference-lab) | `JMNI-Labs/DeepSeek-V4.1-Flash-1017-FLARE@gb10-tp4` | Both lanes run **one identical binary**, sha256 `3cd3f442ccb9b499f8469be1f878ee49541f6edfab41db5c140495aa257aa178` (26,120,544 bytes) at `/usr/bin/flare` and `~/.local/lib/flare-agent/flare`. Source: `NVIDIA/hyperflare-inference` **main `6b68b9b4`** (2026-11-15T18:14:40Z, the v0.2.1 line). Built on **dgx5**: ```bash git clone --depth 1 https://github.com/NVIDIA/hyperflare-inference ~/hyperflare-src cd ~/hyperflare-src cargo build --release --locked --manifest-path flare-cli/Cargo.toml --features roce # -> flare-cli/target/release/flare ``` ## 2. Corrections to our earlier claims Each row was re-tested on 2026-11-16; "before" is what our docs/config said. | # | earlier claim | measured now | |---|---|---| | 1 | build with `--manifest-path flare/Cargo.toml`, binary `flare/target/release/flare`, "~6 min, 20 MB" | the serving binary is the **`flare-cli`** crate: `--manifest-path flare-cli/Cargo.toml`, output `flare-cli/target/release/flare`. Pass `--features roce` (harmless on aarch64 — it is on by default there — required on x86_64). ~21 min cold with 344 crates | | 2 | `flare --version` prints `0.1.0` for every build | a source build prints **`flare 0.0.0`** (the crate carries no version; tags stamp release builds only). Never use `--version` as the identity test — use `sha256sum` | | 3 | PR #184 detection: `strings \| grep -c "embedding read worker panicked"` → 1 | **obsolete**: current main contains no such string (0 in both PPA and source builds), because the disk-Engram reader was rewritten. The marker cannot distinguish builds any more | | 4 | `--no-embedding-table-prefetch` is part of the publisher's TP4 recipe | **removed from the CLI.** Disk prefetch is now native on Linux and there is no switch; `--embedding-table-workers` exists but is Windows-only. Launching with the publisher's `launch.sh` verbatim fails on this flag | | 5 | cache layout is `packages/` and `paths.package_cache` | installed cores live in **`cores/-/`**, prepared models and peer shares in `models///`, hub transport in `transport/hub/v1/`. `package_cache` is still accepted as an **alias** for `core_cache`, so an old `config.toml` silently redirects the store | | 6 | FLARE rejects `reasoning_effort` / `chat_template_kwargs`; thinking is fixed by the package | **false from this build on.** `reasoning_effort: high` returns `reasoning_content`; `chat_template_kwargs.enable_thinking=false` returns content with no reasoning. Verified on both lanes | | 7 | a forced named `tool_choice` with thinking off returns HTTP 500 (TC-45) | **fixed**: returns 200 with a correct `tool_calls[0] = get_weather {"city":"Berlin"}` | | 8 | packages record "proxy ABI 9/10/11" and the runtime must match | runtime `PROXY_ABI = 14` and it **loads cores recording 9, 10, 11 or 14**. Proven live: the prod core (ABI 10, prepared by the October ignition) serves on the ABI-14 runtime | | 9 | rank processes survive an agent restart | ranks are **children of the local agent**: restarting the unit kills them. What survives is controller loss (control plane only). Both remain true simultaneously — do not conflate them | | 10 | version directories are stable identities | every acquisition **re-stamps** the version directory (`gb10-tp4-202611152102` → `…2357` on re-acquire). Never hard-code a version path in a group definition; re-point it after a re-acquire | ## 3. Operational facts worth keeping - **Mixed builds cannot pair.** A new CLI against old agents fails immediately with `handshake read failed: failed to fill whole buffer`. The swap must therefore be one window: stop the group → swap both paths on every node → restart the agents → relaunch with the new CLI. - **`--gpus 0` on `orbit run` is not "the node's GPU 0"**; it is read as a GPU count and fails with `core has no tensor-parallel size 1 deployment (available: [4])`. Omit the flag and let the cluster's node count set tensor parallelism. - **Hub transport keeps the payload one time.** `models/` entries are hardlinks into `transport/hub/v1/blobs` (inode-shared: 196 of 197 files, 480.1 GiB of 480.1 GiB shared). A re-acquire after a broken tree re-hashes the local blobs (~950 MB/s, NIC ~0) instead of re-downloading. Copy such trees with `-aH`. - **Installing a core is a copy, not a hardlink**: `cores/-` held its own inodes (`hardlinks=1`) next to the download store, so a controller that keeps both needs the payload twice. - **Both lanes are `--persistent` (since 2026-11-16).** The definition lives in the controller's `orbit.toml` (`[[group]]`: core, serving flags, rank-zero flags, rank-ordered nodes, transfer streams, publisher keys) and survives stop, history expiry and reboot. Restart is then one short command — no flags: ```bash ssh dgx1.lan 'flare orbit run --name deepseek1017 --detach' # prod, dgx1-4 ssh dgx5.lan 'flare orbit run --name ds41-tp4 --detach' # sparks, dgx5-8 ``` Persistence does **not** start anything after a reboot; it only removes the need to remember the flag set. `--dangerously-approve-key` / `--dangerously-accept-publisher` are not stored in the definition (trust lives in `trust.toml`), which is why the saved group starts without them. An override on a later run applies to that launch only unless `--persistent` is passed again. ## 4. The 2026-11-16 prod window (record) dgx1–dgx4 were moved from `e3fe697eab1f…` (build of 2026-11-04) to `3cd3f442…` without changing flags, core or publisher trust: | event | UTC | |---|---| | `orbit stop deepseek1017` (drained, not killed) | 10:44:05 | | binaries swapped on all four (`host-scripts/flare-swap.sh`, FORCE=1) | ~10:44:2x | | group relaunched with the identical flag set | 10:44:50 | | `/health` 200 | 10:45:23 | Total outage ≈ **78 s**. Verification after: `/v1/models` (`deepseek-v4.1-flash`, 1,048,576 ctx), `pi` completion via `dgx1/deepseek-v4.1-flash`, forced tool call 200, 8 concurrent requests all 200 (1.0–2.5 s). Rollback binaries kept as `flare.orig-20261116` at both paths on each node. ### Follow-up: both lanes made persistent (same day) `deepseek1017` was relaunched with `--persistent` (10:49:54→10:50:28Z, ~34 s), then stopped and restored with the short form `orbit run --name deepseek1017 --detach` to prove the restore path (10:50:33→10:51:27Z, ~54 s). The restored rank command line is flag-for-flag identical to the pre-window one. `ds41-tp4` was launched persistent from the start. ## 5. Apt decoupling (2026-11-16) The upstream PPA (`https://ppa.local-inference-lab.ai noble/main`, keyring `/etc/apt/keyrings/lil-flare-archive.asc`, package `flare-inference`) shipped exactly one payload: **`/usr/bin/flare`**. We overwrite that file with our source build, so any `apt upgrade` would have replaced the CLI while the agent kept the old inode — the mixed-build handshake failure, delivered by apt. There is also no way for `apt-get install -y --only-upgrade flare-inference` to be "harmless": the package owns the path. All eight nodes were decoupled the same way: ```bash cp -a /usr/bin/flare /tmp/flare-keep # our build sudo apt-get purge -y flare-inference # removes the file it owns sudo install -m 0755 /tmp/flare-keep /usr/bin/flare # ours, now unowned sudo rm -f /etc/apt/sources.list.d/flare.sources \ /etc/apt/keyrings/lil-flare-archive.asc ``` Result on every node: `dpkg -S /usr/bin/flare` → *no path found*, no `flare` apt source, no LIL key, `apt-get update` clean, `/usr/bin/flare` and `~/.local/lib/flare-agent/flare` both `3cd3f442…`. Unattended upgrades are disabled anyway (`APT::Periodic::Unattended-Upgrade "0"`), so the fleet now has no path by which an upstream build arrives uninvited: upgrades are manual, `/usr/local/share/hyperflare/v0.2.1` (staged on all eight nodes, with the bundled LICENSE/NOTICE/THIRD_PARTY_NOTICES replacing the purged package docs), and rollback is the single `.orig-20261116` generation. **Rebuilds are now entirely ours.** Bumping upstream means: watch the repo, rebuild, stage a new `vX.Y.Z`, swap with `host-scripts/flare-swap.sh`, restart the agents together, relaunch both persistent groups. ## 6. Not fixed by this upgrade - `dgx4` still has the KingSpec XG7000-2TB 2242 whose stop-and-go prefill poisoned RoCE on the October package — unchanged by a runtime upgrade; see `flare-roce-poison.md`. The disk-bound Engram behaviour it causes is a hardware property, not a build property. - TP4 disk Engram still reads ~189 GiB of tables through a 1 GiB row cache; the sparks-lane measurement (`--embedding-table-resident-scales`, disk) fits ~10.5 GiB of GPU headroom at c32, i.e. no change in the KV budget.