Startseite Erkunden Hilfe
Anmelden
sm121
/
launcher
2
0
Fork 0
Dateien Issues 0 Pull-Requests 0 Wiki
Struktur: a1d58a7374
Branches Tags
launcher
/
specs
/
tech-architecture
/
flare-bringup.md

flare-bringup.md 13 KB
Verlauf Originalformat

FLARE bring-up on the Spark stand (2026-10-22)

NVIDIA published NVIDIA/flare-local-inference on 2026-10-20: a local inference server for SM120/SM121 GPU. Python (ignition) prepares a model with PyTorch + FlashInfer/b12x, tunes kernels, compiles forwards and captures CUDA graphs into a runtime package; a Rust runtime (helios, driven by the flare CLI) serves that package without Python. flare orbit drives multi-host ranks through a per-host flare agent.

Our earlier FLARE lane (dsv41-flash-1017-*, 0/100, kernel timeouts) ran a closed build; the source explains what we were missing.

What now works

deepseek-ai/DeepSeek-V4.1-Flash-1017 (the production model, pinned revision 1bae0e1088993b9b82de08f59de5be5a34e42170) prepared at TP4 across dgx5–dgx8 and served by FLARE:

  • model id DeepSeek-V4.1-Flash, max_model_len 1,048,576
  • KV cache 24.0 GiB/rank → 28,923,136 tokens, 32 shared snapshot slots
  • TP collectives over RoCE (2 rails per peer over roceP2p1s0f0,rocep1s0f0)
  • package: /mnt/nas/ai/flare/dsv41-flash-tp4.safetensors (7 MB index) + 477 GiB of content-addressed blobs beside it

First-boot numbers (single request unless noted, streaming, thinking on):

measure FLARE production SGLang EP2 4x (2026-10-18 run)
prefill ~1,430 t/s (2K prompt, TTFT 1.43 s) · ~1,700 t/s (8K, 4.67 s) 2,800–3,127 t/s
decode c1 19.2 t/s 47.8–50.7 t/s
decode c2 29.6 t/s aggregate 64.2–68.3 t/s
cached 8K prefix TTFT 4.67 → 2.01 s —

So the first boot is ~2.4x slower on decode than our production lane. Nothing is tuned yet: --max-concurrency stayed at the package's 4, the scheduler policy is balanced, and the graph/pool geometry is whatever the preparation picked. That is the next thing to move, not a verdict.

Procedure (dgx5–8, cluster sparks)

# one-time per host
sudo apt-get install -y build-essential cmake ninja-build pkg-config \
    protobuf-compiler libssl-dev python3-dev liburing-dev
curl -LsSf https://astral.sh/uv/install.sh | sh

# ~/.config/flare/config.toml on every host
[paths]
hub_cache = "/mnt/nas/ai/huggingface/hub"
kernel_cache = "/home/kilork/.cache/flare/kernels"
package_cache = "/mnt/nas/ai/flare/packages"
export_staging = "/mnt/nas/ai/flare/staging"
cuda = "/usr/local/cuda"

[memory]
reserve = "4G"

git clone --depth 1 https://github.com/NVIDIA/flare-local-inference ~/flare-src
cd ~/flare-src && uv sync --locked --extra prepare && \
  cargo build --release --manifest-path flare/Cargo.toml      # ~6 min, 20 MB binary

# the agent binary MUST be this same build (see the ABI note below)
scp flare/target/release/flare <peer>:/tmp/flare-new && \
  ssh <peer> 'systemctl --user stop flare-agent; mv /tmp/flare-new ~/.local/lib/flare-agent/flare; systemctl --user start flare-agent'

# preparation token: identical >=32 random bytes, mode 600, on every host
head -c 48 /dev/urandom | base64 > ~/.config/flare/agent.token && chmod 600 …

# rank zero
uv run ignition export deepseek-v4.1-flash --gpus 0 --ranks 4 \
  --rendezvous <rank0-ip>:29600 --max-concurrency 4 \
  --output /mnt/nas/ai/flare/dsv41-flash-tp4.safetensors --verbose
# each peer, while rank zero waits
uv run ignition join <rank0-ip>:29600 --rank N --ranks 4

# serve (rank zero runs the HTTP API)
flare orbit run gx10-dgx5:/mnt/nas/ai/flare/dsv41-flash-tp4.safetensors \
  --cluster sparks --host 0.0.0.0 --port 30000 --detach \
  --dangerously-approve-key <fp> --dangerously-accept-publisher local-development
flare orbit status | attach sparks | stop sparks | ls

Failure modes hit on the way (all fixed)

symptom cause fix
orbit status → unreachable: handshake read failed agents were the 2026-10-20 release, the controller a fresh build install the freshly built flare as ~/.local/lib/flare-agent/flare on every host, restart the unit
FileNotFoundError: ~/.config/flare/agent.token the preparation token is our own shared secret, not an Orbit identity write identical ≥32 random bytes, mode 600, on every host
b12x.loader build dependency is missing: /usr/include/python3.12/Python.h on ranks 1–3 python3-dev missing (the b12x loader compiles a small extension at load) apt install python3-dev
OSError: [Errno 28] No space left on device during export the package blob store lives under [paths].package_cache — default ~/.cache/flare/packages on rank zero, 37 GB before the disk filled point package_cache (and export_staging) at the NAS; keep kernel_cache local so the tuning cache survives
launch refused: N GiB free under the agent cache; this launch stores 121.8 GiB there every rank copies the package to its own agent cache free ≥130 GiB per host
package RoCE kernels use proxy ABI 10, this runtime provides 9 upstream skew: the pinned flashinfer (local-inference-lab 03de0b66">@03de0b66) has ROCE_ABI_VERSION 10, the vendored helios/native/roce_proxy.c in our 2026-10-21 clone still had 9 git pull — commit e350914a3b "Match native RoCE transport to the pinned FlashInfer proxy ABI" (2026-10-21 16:31) — rebuild the binary, reinstall on every agent. Do not patch the constant: the two C files differ by a whole ring transport (317 diff lines)

Disk after the bring-up: dgx5 383 GB, dgx6 342 GB, dgx7 220 GB, dgx8 120 GB free (the 122 GiB package copy per host is the large item).

Useful knobs for the next pass

  • flare orbit run --scheduler-policy throughput | balanced, --max-concurrency N, --kv-cache-limit {memory,context,concurrency}, --memory-limit.
  • --mtp-draft-tokens N (package default 5), --no-mtp, --speculative-token-capacity — the DSpark path.
  • Newest upstream commit adds adaptive verification scoring, and docs/glm-batched-speculation-defect.md documents a Spark-only batched speculation corruption fixed at 22887c1 (rare TP4 device error at 16 concurrent remains open) — read it before trusting multi-request batches.
  • ignition export --help lists per-recipe bloat/knob controls (--expert-scale-compression csf, --online-mxfp8, --packed-verification, --prompt-logprobs, --tuning-gpus).

Upstream watch (FLARE moves several times a day)

The repo shipped 12+ commits in the twelve hours after our first bring-up, and one of them was the RoCE ABI fix that unblocked us at all. Treat the upstream as a moving target and check it before every FLARE experiment:

scripts/flare-upstream-watch.sh          # HEAD vs the pin, flagged commits
scripts/flare-upstream-watch.sh --bump   # move the pin after reviewing

The pin lives in flare-upstream.pin. Commits touching RoCE/ABI, FlashInfer pins, speculation (dspark/dflash/mtp), recipes, package prepare/export and the agents are marked ⚠ — those are the ones that invalidate our agent binaries or prepared packages. Pull and rebuild both sides together; a package prepared by one revision is refused by another (proxy ABI).

What the watch turned up on 2026-10-23 (pin 3b9e86a → HEAD 1fd1f91, 17 commits in 14 hours):

  • Support MiniMax M3.1 Flash with packed DSpark speculation — the model we had just downloaded (252 GB, MiniMaxM3SparseForConditionalGeneration, mxfp8 block [1,32]) is now a first-class FLARE recipe.
  • Add Gemma 4 text recipe with Assistant MTP, Serve and prepare over hosts cabled as a chain, Repair prefix cache retention, Enable ARM SHA-256 acceleration for package hashing (the Pi-class agent path), Set recipe defaults to 32 requests and bounded speculative catalogs, Make multi-host prefix records safe when a peer fails.
  • The DeepSeek side keeps moving too: Preset DeepSeek V4.1 packed GPU speculation, Add a serving option for adaptive verification scoring alpha, Resume preparation checkpoints and share compiled forwards across ranks.

Package distribution: CLI store vs agent cache (measured 2026-11-02)

The gb10 package on the local-inference-lab HF repo was re-cut (format 2, blob store) and is now prepared and distributed differently from the 2026-10-22 build. What follows is measured, not inferred — the numbers come from two live 4-node lanes (prod4 = dgx1–4 with dgx4 as controller+rank 0, sparks = dgx5–8 with dgx4 as controller only).

Two stores, and only one of them is configurable

HF repo ──► `flare download` ──► CLI store   (paths.package_cache)
                                     │  source for staging
                                     ▼
                              `orbit run` ──► agent cache, per node
                                (~/.cache/flare/packages/<sha256>-<size>/)
  • paths.package_cache (config) moves the CLI store. The agent store is a separate location (~/.cache/flare/packages) that this key does not move; the launched rank receives it as --package-cache. Budget both.
  • A format 2 package is a blob store: blobs/<sha>.safetensors holds the data once and models/<version>/{model.safetensors,weights/,deployments/} are hardlinks into it (link count 2). Per-version data is not self-contained — copying models/<version> alone with a hardlink-blind tool expands every link and duplicates the whole package.
  • orbit run stages data through the ranks: the complete package goes to rank 0 first, then rank 0 relays to its peers over the fabric (fabric FETCH, "upstream hop"). The gx10-dgx5: sending … log lines name the hop's source, not the receiver — a rank-0 line is not a node copying to itself.

Measured footprint per node (gb10, 488 GB package, TP4)

role agent cache contents
rank 0 (seed) 488,241 MB the complete package
rank 1–3 267,111 MB each 183 GB shared Engram tables + ~84 GB own rank slice

The Engram tables are duplicated on every rank by design (rows are read per request at token time), so ~183 GB/node is not reclaimable.

The duplicate on a controller that is also rank 0

prod4 runs on dgx4 (2 TB): CLI store 488 GB plus agent cache 488 GB = 976 GB, because the same machine must both stage (CLI store) and serve (agent cache). sparks (dgx5–8, 916 GB disks) has no CLI store and therefore no duplication:

node role used free
dgx5 rank 0 615 GB 255 GB
dgx6/7/8 rank 1–3 ~396 GB ~473 GB

So a 916 GB Spark is a perfectly good FLARE rank — including rank 0 — as long as it is not also the controller's staging store.

Removing the duplicate: source the package from a node

orbit run accepts NODE:/absolute/path as the package argument ("A Hugging Face repository, local package path, or NODE:/absolute/path"), and the agent cache directory (<cache>/packages/<sha256>-<size>/) has exactly the flat package layout, so rank 0's own copy can seed the launch:

flare orbit run gx10-dgx5:/home/kilork/.cache/flare/packages/<id>/model.safetensors \
  --cluster sparks --name dsv41-flare-sparks --detach …

Verified: a real path passes package verification and fails only on GPU occupancy, while a bogus path fails earlier with cannot stat package file. With that form the controller needs no CLI store at all (dgx4's 488 GB becomes reclaimable once a node holds the package).

Launcher fixes this brought (0.2.6)

  • mirror_cmd is layout-aware: a blob store mirrors its store root with rsync -aH (hardlinks preserved) and without --delete (that root holds every other prepared package); a flat package keeps a dedicated destination and prunes. Before this, mirroring expanded the links — a 488 GB package produced a second full copy rather than a mirror.
  • local_package no longer requires the mirror marker: a store populated by a plain flare download is adopted via flare models list --json, but only when exactly one prepared version matches <revision>-tp<N> (two candidates mean "download", never guess).
  • companion probing happens inside the package's own directory in both layouts (it used to probe the mirror root, which a blob store does not have).

Re-pairing an orbit without the old controller (2026-11-02)

An orbit node only accepts a controller listed in its ~/.config/flare/agent/controllers.toml, and that record is signed by the pairing controller — a node that trusts gx10-dgx5 refuses gx10-dgx4 with <key> is not an authorized controller of this node, and orbit controllers add cannot be run by the un-authorized controller (it fails with "this controller is not in the controller set"). With dgx5 powered off (NAS failure) the only path was a clean re-pair, which is safe as long as no group is running on that orbit:

# on each node (keeps the key and the package cache)
flare agent reset
flare agent install --accept-first

# on the new controller — the HOSTS are resolved by name, so pass real hostnames
flare orbit pair dgx1.lan dgx2.lan dgx3.lan dgx4.lan

Two practical notes:

  • orbit pair resolves its HOST arguments through DNS and rewrites the node's address in orbit.toml; a short name that only resolves on some nodes (gx10-dgx1 instead of dgx1.lan) fails with "Temporary failure in name resolution". Use the FQDN form.
  • Rank order comes from [cluster.<name>].nodes in orbit.toml, and it decides who is rank 0 — the node that receives the complete package and serves the API. On a disk-constrained set put the roomiest node first.