Jelajahi Sumber

docs(flare): record main@6b68b9b4 upgrade and correct superseded claims

Both lanes now run one binary (sha256 3cd3f442..., main@6b68b9b4, built from
flare-cli + --features roce on dgx5). Verified against the running system:

- serving crate is flare-cli, not flare; source builds print 0.0.0, so the
  PR-184 strings marker and flare --version are no longer identity tests
- --no-embedding-table-prefetch is gone (disk prefetch is native on Linux)
- store is cores/<id>-<bytes> + models/<version>; package_cache is an alias
- runtime PROXY_ABI 14 loads cores recording 9/10/11/14 (prod core is ABI 10)
- reasoning_effort and chat_template_kwargs.enable_thinking are honoured now
- forced named tool_choice returns 200 (was 500, TC-45)
- ranks are children of the local agent: an agent restart kills them

Bumps flare-upstream.pin to 6b68b9b4 and adds
specs/tech-architecture/flare-2026-11-16-upgrade.md with the prod window record
(stop 10:44:05Z, health 10:45:23Z, ~78 s outage, flags unchanged).
Alexander Korolev 2 hari lalu
induk
melakukan
dc65bc13d7

+ 95 - 0
specs/tech-architecture/flare-2026-11-16-upgrade.md

@@ -0,0 +1,95 @@
+# FLARE: fleet upgrade to main@6b68b9b4 and the corrections it forces (2026-11-16)
+
+Written after rebuilding both lanes and brain-checking the old claims against the
+running system. **This document supersedes specific statements in
+`flare-bringup.md`, `flare-storage-nas.md` and `flare-roce-poison.md`**; those
+files stay as history (they describe the October stand) but where they disagree
+with this page, this page was measured on 2026-11-16.
+
+## 1. What is deployed
+
+| | prod lane | sparks lane |
+|---|---|---|
+| nodes | dgx1→dgx4, cluster `sparks`, group `deepseek1017` | dgx5→dgx8, cluster `dgx58`, group `ds41-tp4` |
+| controller | `gx10-dgx1` | `gx10-dgx5` |
+| API | `http://dgx1.lan:30000/v1`, alias `deepseek-v4.1-flash` | `http://dgx5.lan:30000/v1`, same alias |
+| core | `db20209d361f473b8452f24ad517fa0d-33966105` (ABI 10) | `edb5231cb9b847b7914a6475a5c02b3d-24324121` (ABI 14) |
+| model | `checkpoint-expanded-scales` (local-inference-lab) | `JMNI-Labs/DeepSeek-V4.1-Flash-1017-FLARE@gb10-tp4` |
+
+Both lanes run **one identical binary**, sha256
+`3cd3f442ccb9b499f8469be1f878ee49541f6edfab41db5c140495aa257aa178`
+(26,120,544 bytes) at `/usr/bin/flare` and `~/.local/lib/flare-agent/flare`.
+
+Source: `NVIDIA/hyperflare-inference` **main `6b68b9b4`** (2026-11-15T18:14:40Z,
+the v0.2.1 line). Built on **dgx5**:
+
+```bash
+git clone --depth 1 https://github.com/NVIDIA/hyperflare-inference ~/hyperflare-src
+cd ~/hyperflare-src
+cargo build --release --locked --manifest-path flare-cli/Cargo.toml --features roce
+# -> flare-cli/target/release/flare
+```
+
+## 2. Corrections to our earlier claims
+
+Each row was re-tested on 2026-11-16; "before" is what our docs/config said.
+
+| # | earlier claim | measured now |
+|---|---|---|
+| 1 | build with `--manifest-path flare/Cargo.toml`, binary `flare/target/release/flare`, "~6 min, 20 MB" | the serving binary is the **`flare-cli`** crate: `--manifest-path flare-cli/Cargo.toml`, output `flare-cli/target/release/flare`. Pass `--features roce` (harmless on aarch64 — it is on by default there — required on x86_64). ~21 min cold with 344 crates |
+| 2 | `flare --version` prints `0.1.0` for every build | a source build prints **`flare 0.0.0`** (the crate carries no version; tags stamp release builds only). Never use `--version` as the identity test — use `sha256sum` |
+| 3 | PR #184 detection: `strings <bin> \| grep -c "embedding read worker panicked"` → 1 | **obsolete**: current main contains no such string (0 in both PPA and source builds), because the disk-Engram reader was rewritten. The marker cannot distinguish builds any more |
+| 4 | `--no-embedding-table-prefetch` is part of the publisher's TP4 recipe | **removed from the CLI.** Disk prefetch is now native on Linux and there is no switch; `--embedding-table-workers` exists but is Windows-only. Launching with the publisher's `launch.sh` verbatim fails on this flag |
+| 5 | cache layout is `packages/` and `paths.package_cache` | installed cores live in **`cores/<core_id>-<bytes>/`**, prepared models and peer shares in `models/<model>/<version>/`, hub transport in `transport/hub/v1/`. `package_cache` is still accepted as an **alias** for `core_cache`, so an old `config.toml` silently redirects the store |
+| 6 | FLARE rejects `reasoning_effort` / `chat_template_kwargs`; thinking is fixed by the package | **false from this build on.** `reasoning_effort: high` returns `reasoning_content`; `chat_template_kwargs.enable_thinking=false` returns content with no reasoning. Verified on both lanes |
+| 7 | a forced named `tool_choice` with thinking off returns HTTP 500 (TC-45) | **fixed**: returns 200 with a correct `tool_calls[0] = get_weather {"city":"Berlin"}` |
+| 8 | packages record "proxy ABI 9/10/11" and the runtime must match | runtime `PROXY_ABI = 14` and it **loads cores recording 9, 10, 11 or 14**. Proven live: the prod core (ABI 10, prepared by the October ignition) serves on the ABI-14 runtime |
+| 9 | rank processes survive an agent restart | ranks are **children of the local agent**: restarting the unit kills them. What survives is controller loss (control plane only). Both remain true simultaneously — do not conflate them |
+| 10 | version directories are stable identities | every acquisition **re-stamps** the version directory (`gb10-tp4-202611152102` → `…2357` on re-acquire). Never hard-code a version path in a group definition; re-point it after a re-acquire |
+
+## 3. Operational facts worth keeping
+
+- **Mixed builds cannot pair.** A new CLI against old agents fails immediately with
+  `handshake read failed: failed to fill whole buffer`. The swap must therefore be
+  one window: stop the group → swap both paths on every node → restart the agents →
+  relaunch with the new CLI.
+- **`--gpus 0` on `orbit run` is not "the node's GPU 0"**; it is read as a GPU
+  count and fails with `core has no tensor-parallel size 1 deployment (available: [4])`.
+  Omit the flag and let the cluster's node count set tensor parallelism.
+- **Hub transport keeps the payload one time.** `models/` entries are hardlinks into
+  `transport/hub/v1/blobs` (inode-shared: 196 of 197 files, 480.1 GiB of 480.1 GiB
+  shared). A re-acquire after a broken tree re-hashes the local blobs
+  (~950 MB/s, NIC ~0) instead of re-downloading. Copy such trees with `-aH`.
+- **Installing a core is a copy, not a hardlink**:
+  `cores/<id>-<bytes>` held its own inodes (`hardlinks=1`) next to the download
+  store, so a controller that keeps both needs the payload twice.
+- **`orbit run --persistent`** saves the launch for restarts; a non-persistent group
+  (both lanes today, `deepseek1017` and its predecessor) must be relaunched by hand
+  with the full flag set after any agent restart.
+
+## 4. The 2026-11-16 prod window (record)
+
+dgx1–dgx4 were moved from `e3fe697eab1f…` (build of 2026-11-04) to `3cd3f442…`
+without changing flags, core or publisher trust:
+
+| event | UTC |
+|---|---|
+| `orbit stop deepseek1017` (drained, not killed) | 10:44:05 |
+| binaries swapped on all four (`host-scripts/flare-swap.sh`, FORCE=1) | ~10:44:2x |
+| group relaunched with the identical flag set | 10:44:50 |
+| `/health` 200 | 10:45:23 |
+
+Total outage ≈ **78 s**. Verification after: `/v1/models` (`deepseek-v4.1-flash`,
+1,048,576 ctx), `pi` completion via `dgx1/deepseek-v4.1-flash`, forced tool call 200,
+8 concurrent requests all 200 (1.0–2.5 s). Rollback binaries kept as
+`flare.orig-20261116` at both paths on each node.
+
+## 5. Not fixed by this upgrade
+
+- `dgx4` still has the KingSpec XG7000-2TB 2242 whose stop-and-go prefill poisoned
+  RoCE on the October package — unchanged by a runtime upgrade; see
+  `flare-roce-poison.md`. The disk-bound Engram behaviour it causes is a hardware
+  property, not a build property.
+- TP4 disk Engram still reads ~189 GiB of tables through a 1 GiB row cache; the
+  sparks-lane measurement (`--embedding-table-resident-scales`, disk) fits ~10.5 GiB
+  of GPU headroom at c32, i.e. no change in the KV budget.

+ 8 - 0
specs/tech-architecture/flare-bringup.md

@@ -1,5 +1,10 @@
 # FLARE bring-up on the Spark stand (2026-10-22)
 
+> **Partially superseded 2026-11-16.** The build command, the `packages/` cache
+> layout and the `flare --version` identity test below describe the October stand.
+> See [`flare-2026-11-16-upgrade.md`](flare-2026-11-16-upgrade.md) for what is
+> deployed now and which of these statements no longer hold.
+
 NVIDIA published [`NVIDIA/flare-local-inference`](https://github.com/NVIDIA/flare-local-inference)
 on 2026-10-20: a local inference server for **SM120/SM121** GPU. Python
 (`ignition`) prepares a model with PyTorch + FlashInfer/b12x, tunes kernels,
@@ -60,6 +65,9 @@ reserve = "4G"
 git clone --depth 1 https://github.com/NVIDIA/flare-local-inference ~/flare-src
 cd ~/flare-src && uv sync --locked --extra prepare && \
   cargo build --release --manifest-path flare/Cargo.toml      # ~6 min, 20 MB binary
+# NOTE 2026-11-16: the serving binary is the `flare-cli` crate now —
+#   cargo build --release --locked --manifest-path flare-cli/Cargo.toml --features roce
+# see specs/tech-architecture/flare-2026-11-16-upgrade.md
 
 # the agent binary MUST be this same build (see the ABI note below)
 scp flare/target/release/flare <peer>:/tmp/flare-new && \

+ 4 - 0
specs/tech-architecture/flare-storage-nas.md

@@ -1,5 +1,9 @@
 # FLARE storage: expanded checkpoint on node NVMe, everything else on NAS (2026-11-10)
 
+> **Partly superseded 2026-11-16.** Store layout (`packages/` → `cores/` +
+> `models/`), the `flare --version` test and the PR-184 marker test below no longer
+> apply. See [`flare-2026-11-16-upgrade.md`](flare-2026-11-16-upgrade.md).
+
 Goal, in the operator's words: **only the expanded checkpoint stays on the sparks
 NVMe** (260 GiB for `DeepSeek-V4.1-Flash` tp4) and **all other model artefacts live
 on the NAS**. This document is written to be executed by someone with no context

+ 1 - 1
specs/tech-architecture/flare-upstream.pin

@@ -1 +1 @@
-585159c95ef6f5d126dd070a79f5f48e16d2b1c1 2026-11-04T02:14
+6b68b9b4aae1b30ce352d201fc6aa501a3cfa525 2026-11-15T18:14