# FLARE bring-up on the Spark stand (2026-10-22) NVIDIA published [`NVIDIA/flare-local-inference`](https://github.com/NVIDIA/flare-local-inference) on 2026-10-20: a local inference server for **SM120/SM121** GPU. Python (`ignition`) prepares a model with PyTorch + FlashInfer/b12x, tunes kernels, compiles forwards and captures CUDA graphs into a **runtime package**; a Rust runtime (`helios`, driven by the `flare` CLI) serves that package without Python. `flare orbit` drives multi-host ranks through a per-host `flare agent`. Our earlier FLARE lane (`dsv41-flash-1017-*`, 0/100, kernel timeouts) ran a closed build; the source explains what we were missing. ## What now works `deepseek-ai/DeepSeek-V4.1-Flash-1017` (the production model, pinned revision `1bae0e1088993b9b82de08f59de5be5a34e42170`) prepared at **TP4 across dgx5–dgx8** and served by FLARE: - model id `DeepSeek-V4.1-Flash`, `max_model_len` **1,048,576** - KV cache 24.0 GiB/rank → **28,923,136 tokens**, 32 shared snapshot slots - TP collectives over RoCE (`2 rails per peer over roceP2p1s0f0,rocep1s0f0`) - package: `/mnt/nas/ai/flare/dsv41-flash-tp4.safetensors` (7 MB index) + 477 GiB of content-addressed blobs beside it First-boot numbers (single request unless noted, streaming, thinking on): | measure | FLARE | production SGLang EP2 4x (2026-10-18 run) | |---|---|---| | prefill | ~1,430 t/s (2K prompt, TTFT 1.43 s) · ~1,700 t/s (8K, 4.67 s) | 2,800–3,127 t/s | | decode c1 | 19.2 t/s | 47.8–50.7 t/s | | decode c2 | 29.6 t/s aggregate | 64.2–68.3 t/s | | cached 8K prefix | TTFT 4.67 → 2.01 s | — | So the first boot is ~2.4x slower on decode than our production lane. Nothing is tuned yet: `--max-concurrency` stayed at the package's 4, the scheduler policy is `balanced`, and the graph/pool geometry is whatever the preparation picked. That is the next thing to move, not a verdict. ## Procedure (dgx5–8, cluster `sparks`) ```bash # one-time per host sudo apt-get install -y build-essential cmake ninja-build pkg-config \ protobuf-compiler libssl-dev python3-dev liburing-dev curl -LsSf https://astral.sh/uv/install.sh | sh # ~/.config/flare/config.toml on every host [paths] hub_cache = "/mnt/nas/ai/huggingface/hub" kernel_cache = "/home/kilork/.cache/flare/kernels" package_cache = "/mnt/nas/ai/flare/packages" export_staging = "/mnt/nas/ai/flare/staging" cuda = "/usr/local/cuda" [memory] reserve = "4G" ``` ```bash git clone --depth 1 https://github.com/NVIDIA/flare-local-inference ~/flare-src cd ~/flare-src && uv sync --locked --extra prepare && \ cargo build --release --manifest-path flare/Cargo.toml # ~6 min, 20 MB binary # the agent binary MUST be this same build (see the ABI note below) scp flare/target/release/flare :/tmp/flare-new && \ ssh 'systemctl --user stop flare-agent; mv /tmp/flare-new ~/.local/lib/flare-agent/flare; systemctl --user start flare-agent' # preparation token: identical >=32 random bytes, mode 600, on every host head -c 48 /dev/urandom | base64 > ~/.config/flare/agent.token && chmod 600 … # rank zero uv run ignition export deepseek-v4.1-flash --gpus 0 --ranks 4 \ --rendezvous :29600 --max-concurrency 4 \ --output /mnt/nas/ai/flare/dsv41-flash-tp4.safetensors --verbose # each peer, while rank zero waits uv run ignition join :29600 --rank N --ranks 4 ``` ```bash # serve (rank zero runs the HTTP API) flare orbit run gx10-dgx5:/mnt/nas/ai/flare/dsv41-flash-tp4.safetensors \ --cluster sparks --host 0.0.0.0 --port 30000 --detach \ --dangerously-approve-key --dangerously-accept-publisher local-development flare orbit status | attach sparks | stop sparks | ls ``` ## Failure modes hit on the way (all fixed) | symptom | cause | fix | |---|---|---| | `orbit status` → `unreachable: handshake read failed` | agents were the 2026-10-20 release, the controller a fresh build | install the freshly built `flare` as `~/.local/lib/flare-agent/flare` on every host, restart the unit | | `FileNotFoundError: ~/.config/flare/agent.token` | the **preparation token** is our own shared secret, not an Orbit identity | write identical ≥32 random bytes, mode 600, on every host | | `b12x.loader build dependency is missing: /usr/include/python3.12/Python.h` on ranks 1–3 | `python3-dev` missing (the b12x loader compiles a small extension at load) | apt install `python3-dev` | | `OSError: [Errno 28] No space left on device` during export | the package blob store lives under `[paths].package_cache` — default `~/.cache/flare/packages` **on rank zero**, 37 GB before the disk filled | point `package_cache` (and `export_staging`) at the NAS; keep `kernel_cache` local so the tuning cache survives | | launch refused: `N GiB free under the agent cache; this launch stores 121.8 GiB there` | every rank copies the package to its own agent cache | free ≥130 GiB per host | | `package RoCE kernels use proxy ABI 10, this runtime provides 9` | **upstream skew**: the pinned `flashinfer` (`local-inference-lab` @03de0b66) has `ROCE_ABI_VERSION 10`, the vendored `helios/native/roce_proxy.c` in our 2026-10-21 clone still had 9 | `git pull` — commit `e350914a3b` "Match native RoCE transport to the pinned FlashInfer proxy ABI" (2026-10-21 16:31) — rebuild the binary, reinstall on every agent. Do **not** patch the constant: the two C files differ by a whole ring transport (317 diff lines) | Disk after the bring-up: dgx5 383 GB, dgx6 342 GB, dgx7 220 GB, dgx8 120 GB free (the 122 GiB package copy per host is the large item). ## Useful knobs for the next pass - `flare orbit run --scheduler-policy throughput | balanced`, `--max-concurrency N`, `--kv-cache-limit {memory,context,concurrency}`, `--memory-limit`. - `--mtp-draft-tokens N` (package default 5), `--no-mtp`, `--speculative-token-capacity` — the DSpark path. - Newest upstream commit adds **adaptive verification scoring**, and `docs/glm-batched-speculation-defect.md` documents a Spark-only batched speculation corruption fixed at `22887c1` (rare TP4 device error at 16 concurrent remains open) — read it before trusting multi-request batches. - `ignition export --help` lists per-recipe bloat/knob controls (`--expert-scale-compression csf`, `--online-mxfp8`, `--packed-verification`, `--prompt-logprobs`, `--tuning-gpus`). ## Upstream watch (FLARE moves several times a day) The repo shipped 12+ commits in the twelve hours after our first bring-up, and one of them was the RoCE ABI fix that unblocked us at all. Treat the upstream as a moving target and check it **before** every FLARE experiment: ```bash scripts/flare-upstream-watch.sh # HEAD vs the pin, flagged commits scripts/flare-upstream-watch.sh --bump # move the pin after reviewing ``` The pin lives in [`flare-upstream.pin`](flare-upstream.pin). Commits touching RoCE/ABI, FlashInfer pins, speculation (dspark/dflash/mtp), recipes, package prepare/export and the agents are marked ⚠ — those are the ones that invalidate our agent binaries or prepared packages. **Pull and rebuild both sides together**; a package prepared by one revision is refused by another (`proxy ABI`). What the watch turned up on 2026-10-23 (pin `3b9e86a` → HEAD `1fd1f91`, 17 commits in 14 hours): - **`Support MiniMax M3.1 Flash with packed DSpark speculation`** — the model we had just downloaded (252 GB, `MiniMaxM3SparseForConditionalGeneration`, mxfp8 block [1,32]) is now a first-class FLARE recipe. - `Add Gemma 4 text recipe with Assistant MTP`, `Serve and prepare over hosts cabled as a chain`, `Repair prefix cache retention`, `Enable ARM SHA-256 acceleration for package hashing` (the Pi-class agent path), `Set recipe defaults to 32 requests and bounded speculative catalogs`, `Make multi-host prefix records safe when a peer fails`. - The DeepSeek side keeps moving too: `Preset DeepSeek V4.1 packed GPU speculation`, `Add a serving option for adaptive verification scoring alpha`, `Resume preparation checkpoints and share compiled forwards across ranks`.