6b68b9b4 and the corrections it forces (2026-11-16)Written after rebuilding both lanes and brain-checking the old claims against the
running system. This document supersedes specific statements in
flare-bringup.md, flare-storage-nas.md and flare-roce-poison.md; those
files stay as history (they describe the October stand) but where they disagree
with this page, this page was measured on 2026-11-16.
| prod lane | sparks lane | |
|---|---|---|
| nodes | dgx1→dgx4, cluster sparks, group deepseek1017 |
dgx5→dgx8, cluster dgx58, group ds41-tp4 |
| controller | gx10-dgx1 |
gx10-dgx5 |
| API | http://dgx1.lan:30000/v1, alias deepseek-v4.1-flash |
http://dgx5.lan:30000/v1, same alias |
| core | db20209d361f473b8452f24ad517fa0d-33966105 (ABI 10) |
edb5231cb9b847b7914a6475a5c02b3d-24324121 (ABI 14) |
| model | checkpoint-expanded-scales (local-inference-lab) |
JMNI-Labs/DeepSeek-V4.1-Flash-1017-FLARE@gb10-tp4 |
Both lanes run one identical binary, sha256
3cd3f442ccb9b499f8469be1f878ee49541f6edfab41db5c140495aa257aa178
(26,120,544 bytes) at /usr/bin/flare and ~/.local/lib/flare-agent/flare.
Source: NVIDIA/hyperflare-inference main 6b68b9b4 (2026-11-15T18:14:40Z,
the v0.2.1 line). Built on dgx5:
git clone --depth 1 https://github.com/NVIDIA/hyperflare-inference ~/hyperflare-src
cd ~/hyperflare-src
cargo build --release --locked --manifest-path flare-cli/Cargo.toml --features roce
# -> flare-cli/target/release/flare
Each row was re-tested on 2026-11-16; "before" is what our docs/config said.
| # | earlier claim | measured now |
|---|---|---|
| 1 | build with --manifest-path flare/Cargo.toml, binary flare/target/release/flare, "~6 min, 20 MB" |
the serving binary is the flare-cli crate: --manifest-path flare-cli/Cargo.toml, output flare-cli/target/release/flare. Pass --features roce (harmless on aarch64 — it is on by default there — required on x86_64). ~21 min cold with 344 crates |
| 2 | flare --version prints 0.1.0 for every build |
a source build prints flare 0.0.0 (the crate carries no version; tags stamp release builds only). Never use --version as the identity test — use sha256sum |
| 3 | PR #184 detection: strings <bin> \| grep -c "embedding read worker panicked" → 1 |
obsolete: current main contains no such string (0 in both PPA and source builds), because the disk-Engram reader was rewritten. The marker cannot distinguish builds any more |
| 4 | --no-embedding-table-prefetch is part of the publisher's TP4 recipe |
removed from the CLI. Disk prefetch is now native on Linux and there is no switch; --embedding-table-workers exists but is Windows-only. Launching with the publisher's launch.sh verbatim fails on this flag |
| 5 | cache layout is packages/ and paths.package_cache |
installed cores live in cores/<core_id>-<bytes>/, prepared models and peer shares in models/<model>/<version>/, hub transport in transport/hub/v1/. package_cache is still accepted as an alias for core_cache, so an old config.toml silently redirects the store |
| 6 | FLARE rejects reasoning_effort / chat_template_kwargs; thinking is fixed by the package |
false from this build on. reasoning_effort: high returns reasoning_content; chat_template_kwargs.enable_thinking=false returns content with no reasoning. Verified on both lanes |
| 7 | a forced named tool_choice with thinking off returns HTTP 500 (TC-45) |
fixed: returns 200 with a correct tool_calls[0] = get_weather {"city":"Berlin"} |
| 8 | packages record "proxy ABI 9/10/11" and the runtime must match | runtime PROXY_ABI = 14 and it loads cores recording 9, 10, 11 or 14. Proven live: the prod core (ABI 10, prepared by the October ignition) serves on the ABI-14 runtime |
| 9 | rank processes survive an agent restart | ranks are children of the local agent: restarting the unit kills them. What survives is controller loss (control plane only). Both remain true simultaneously — do not conflate them |
| 10 | version directories are stable identities | every acquisition re-stamps the version directory (gb10-tp4-202611152102 → …2357 on re-acquire). Never hard-code a version path in a group definition; re-point it after a re-acquire |
handshake read failed: failed to fill whole buffer. The swap must therefore be
one window: stop the group → swap both paths on every node → restart the agents →
relaunch with the new CLI.--gpus 0 on orbit run is not "the node's GPU 0"; it is read as a GPU
count and fails with core has no tensor-parallel size 1 deployment (available: [4]).
Omit the flag and let the cluster's node count set tensor parallelism.models/ entries are hardlinks into
transport/hub/v1/blobs (inode-shared: 196 of 197 files, 480.1 GiB of 480.1 GiB
shared). A re-acquire after a broken tree re-hashes the local blobs
(~950 MB/s, NIC ~0) instead of re-downloading. Copy such trees with -aH.cores/<id>-<bytes> held its own inodes (hardlinks=1) next to the download
store, so a controller that keeps both needs the payload twice.Both lanes are --persistent (since 2026-11-16). The definition lives in the
controller's orbit.toml ([[group]]: core, serving flags, rank-zero flags,
rank-ordered nodes, transfer streams, publisher keys) and survives stop, history
expiry and reboot. Restart is then one short command — no flags:
ssh dgx1.lan 'flare orbit run --name deepseek1017 --detach' # prod, dgx1-4
ssh dgx5.lan 'flare orbit run --name ds41-tp4 --detach' # sparks, dgx5-8
Persistence does not start anything after a reboot; it only removes the need to
remember the flag set. --dangerously-approve-key / --dangerously-accept-publisher
are not stored in the definition (trust lives in trust.toml), which is why the
saved group starts without them. An override on a later run applies to that launch
only unless --persistent is passed again.
dgx1–dgx4 were moved from e3fe697eab1f… (build of 2026-11-04) to 3cd3f442…
without changing flags, core or publisher trust:
| event | UTC |
|---|---|
orbit stop deepseek1017 (drained, not killed) |
10:44:05 |
binaries swapped on all four (host-scripts/flare-swap.sh, FORCE=1) |
~10:44:2x |
| group relaunched with the identical flag set | 10:44:50 |
/health 200 |
10:45:23 |
Total outage ≈ 78 s. Verification after: /v1/models (deepseek-v4.1-flash,
1,048,576 ctx), pi completion via dgx1/deepseek-v4.1-flash, forced tool call 200,
8 concurrent requests all 200 (1.0–2.5 s). Rollback binaries kept as
flare.orig-20261116 at both paths on each node.
deepseek1017 was relaunched with --persistent (10:49:54→10:50:28Z, ~34 s), then
stopped and restored with the short form orbit run --name deepseek1017 --detach
to prove the restore path (10:50:33→10:51:27Z, ~54 s). The restored rank command
line is flag-for-flag identical to the pre-window one. ds41-tp4 was launched
persistent from the start.
The upstream PPA (https://ppa.local-inference-lab.ai noble/main, keyring
/etc/apt/keyrings/lil-flare-archive.asc, package flare-inference) shipped
exactly one payload: /usr/bin/flare. We overwrite that file with our source
build, so any apt upgrade would have replaced the CLI while the agent kept the
old inode — the mixed-build handshake failure, delivered by apt. There is also no
way for apt-get install -y --only-upgrade flare-inference to be "harmless": the
package owns the path.
All eight nodes were decoupled the same way:
cp -a /usr/bin/flare /tmp/flare-keep # our build
sudo apt-get purge -y flare-inference # removes the file it owns
sudo install -m 0755 /tmp/flare-keep /usr/bin/flare # ours, now unowned
sudo rm -f /etc/apt/sources.list.d/flare.sources \
/etc/apt/keyrings/lil-flare-archive.asc
Result on every node: dpkg -S /usr/bin/flare → no path found, no flare
apt source, no LIL key, apt-get update clean, /usr/bin/flare and
~/.local/lib/flare-agent/flare both 3cd3f442…. Unattended upgrades are
disabled anyway (APT::Periodic::Unattended-Upgrade "0"), so the fleet now has
no path by which an upstream build arrives uninvited: upgrades are manual,
/usr/local/share/hyperflare/v0.2.1 (staged on all eight nodes, with the bundled
LICENSE/NOTICE/THIRD_PARTY_NOTICES replacing the purged package docs), and rollback
is the single .orig-20261116 generation.
Rebuilds are now entirely ours. Bumping upstream means: watch the repo,
rebuild, stage a new vX.Y.Z, swap with host-scripts/flare-swap.sh, restart the
agents together, relaunch both persistent groups.
host-scripts/flare-fleet.shThe window steps are scripted so the next upgrade is a command, not a checklist:
./host-scripts/flare-fleet.sh status # read-only fleet snapshot
./host-scripts/flare-fleet.sh check # same, exit 1 on any drift
./host-scripts/flare-fleet.sh build --ref main --out /tmp/flare
./host-scripts/flare-fleet.sh upgrade --binary /tmp/flare --release v0.3.0 --dry-run
./host-scripts/flare-fleet.sh upgrade --binary /tmp/flare --release v0.3.0 --yes
./host-scripts/flare-fleet.sh rollback --yes
upgrade refuses to act without --yes, aborts if any node is unreachable
or if the staged sha does not match the local binary on every node, and stops
with a rollback hint if the fleet ends up mixed. It prunes .orig-* generations
to the newest after success. build is reproducible: rebuilding
main@6b68b9b4 on 2026-11-16 produced 3cd3f442… again, byte for byte.
Two things the tooling taught us, worth keeping in mind:
GROUPS is a bash special variable. Assigning it is silently ignored
(bash re-populates it with the user's group IDs), so an env-driven group list
must use another name — the script uses FLARE_LANES..orig-* differs per node (dgx1-4 20261116, dgx5-8 older or
absent, because they were installed by hand rather than through a fleet
window). rollback therefore refuses rather than restoring different builds
and creating the very state it should fix — the first fleet-wide upgrade run
fixes this, since flare-swap.sh then writes the same .orig-<date> on every
node. Until then, recovery is to re-deploy the current build from
/usr/local/share/hyperflare/v0.2.1.dgx4 still has the KingSpec XG7000-2TB 2242 whose stop-and-go prefill poisoned
RoCE on the October package — unchanged by a runtime upgrade; see
flare-roce-poison.md. The disk-bound Engram behaviour it causes is a hardware
property, not a build property.--embedding-table-resident-scales, disk) fits ~10.5 GiB
of GPU headroom at c32, i.e. no change in the KV budget.