spec: FLARE bring-up — DeepSeek-V4.1-Flash-1017 served by NVIDIA's flare on 4 Sparks

NVIDIA/flare-local-inference (published 2026-10-20) prepares a model into a
runtime package (PyTorch+FlashInfer/b12x, prepared CUDA graphs) and serves it
from a Rust runtime; orbit drives multi-host ranks through per-host agents.

The prod checkpoint now boots at TP4 on dgx5-8: 1M context, 24 GiB KV per rank
(28.9M tokens), 2 RoCE rails per peer, API on :30000. First-boot numbers are
~2.4x slower on decode than our SGLang EP2 production lane (19.2 vs 47.8 t/s
c1) with nothing tuned yet.

The spec records the full procedure and the six failure modes, including the
upstream RoCE proxy ABI skew between the pinned flashinfer (ABI 10) and the
vendored helios proxy (ABI 9), fixed upstream in e350914a3b one day after our
clone.

specs/tech-architecture/flare-bringup.md

@@ -0,0 +1,113 @@
+# FLARE bring-up on the Spark stand (2026-10-22)
+
+NVIDIA published [`NVIDIA/flare-local-inference`](https://github.com/NVIDIA/flare-local-inference)
+on 2026-10-20: a local inference server for **SM120/SM121** GPU. Python
+(`ignition`) prepares a model with PyTorch + FlashInfer/b12x, tunes kernels,
+compiles forwards and captures CUDA graphs into a **runtime package**; a Rust
+runtime (`helios`, driven by the `flare` CLI) serves that package without Python.
+`flare orbit` drives multi-host ranks through a per-host `flare agent`.
+
+Our earlier FLARE lane (`dsv41-flash-1017-*`, 0/100, kernel timeouts) ran a
+closed build; the source explains what we were missing.
+
+## What now works
+
+`deepseek-ai/DeepSeek-V4.1-Flash-1017` (the production model, pinned revision
+`1bae0e1088993b9b82de08f59de5be5a34e42170`) prepared at **TP4 across dgx5–dgx8**
+and served by FLARE:
+
+- model id `DeepSeek-V4.1-Flash`, `max_model_len` **1,048,576**
+- KV cache 24.0 GiB/rank → **28,923,136 tokens**, 32 shared snapshot slots
+- TP collectives over RoCE (`2 rails per peer over roceP2p1s0f0,rocep1s0f0`)
+- package: `/mnt/nas/ai/flare/dsv41-flash-tp4.safetensors` (7 MB index) +
+  477 GiB of content-addressed blobs beside it
+
+First-boot numbers (single request unless noted, streaming, thinking on):
+
+| measure | FLARE | production SGLang EP2 4x (2026-10-18 run) |
+|---|---|---|
+| prefill | ~1,430 t/s (2K prompt, TTFT 1.43 s) · ~1,700 t/s (8K, 4.67 s) | 2,800–3,127 t/s |
+| decode c1 | 19.2 t/s | 47.8–50.7 t/s |
+| decode c2 | 29.6 t/s aggregate | 64.2–68.3 t/s |
+| cached 8K prefix | TTFT 4.67 → 2.01 s | — |
+
+So the first boot is ~2.4x slower on decode than our production lane. Nothing
+is tuned yet: `--max-concurrency` stayed at the package's 4, the scheduler
+policy is `balanced`, and the graph/pool geometry is whatever the preparation
+picked. That is the next thing to move, not a verdict.
+
+## Procedure (dgx5–8, cluster `sparks`)
+
+```bash
+# one-time per host
+sudo apt-get install -y build-essential cmake ninja-build pkg-config \
+    protobuf-compiler libssl-dev python3-dev liburing-dev
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# ~/.config/flare/config.toml on every host
+[paths]
+hub_cache = "/mnt/nas/ai/huggingface/hub"
+kernel_cache = "/home/kilork/.cache/flare/kernels"
+package_cache = "/mnt/nas/ai/flare/packages"
+export_staging = "/mnt/nas/ai/flare/staging"
+cuda = "/usr/local/cuda"
+
+[memory]
+reserve = "4G"
+```
+
+```bash
+git clone --depth 1 https://github.com/NVIDIA/flare-local-inference ~/flare-src
+cd ~/flare-src && uv sync --locked --extra prepare && \
+  cargo build --release --manifest-path flare/Cargo.toml      # ~6 min, 20 MB binary
+
+# the agent binary MUST be this same build (see the ABI note below)
+scp flare/target/release/flare <peer>:/tmp/flare-new && \
+  ssh <peer> 'systemctl --user stop flare-agent; mv /tmp/flare-new ~/.local/lib/flare-agent/flare; systemctl --user start flare-agent'
+
+# preparation token: identical >=32 random bytes, mode 600, on every host
+head -c 48 /dev/urandom | base64 > ~/.config/flare/agent.token && chmod 600 …
+
+# rank zero
+uv run ignition export deepseek-v4.1-flash --gpus 0 --ranks 4 \
+  --rendezvous <rank0-ip>:29600 --max-concurrency 4 \
+  --output /mnt/nas/ai/flare/dsv41-flash-tp4.safetensors --verbose
+# each peer, while rank zero waits
+uv run ignition join <rank0-ip>:29600 --rank N --ranks 4
+```
+
+```bash
+# serve (rank zero runs the HTTP API)
+flare orbit run gx10-dgx5:/mnt/nas/ai/flare/dsv41-flash-tp4.safetensors \
+  --cluster sparks --host 0.0.0.0 --port 30000 --detach \
+  --dangerously-approve-key <fp> --dangerously-accept-publisher local-development
+flare orbit status | attach sparks | stop sparks | ls
+```
+
+## Failure modes hit on the way (all fixed)
+
+| symptom | cause | fix |
+|---|---|---|
+| `orbit status` → `unreachable: handshake read failed` | agents were the 2026-10-20 release, the controller a fresh build | install the freshly built `flare` as `~/.local/lib/flare-agent/flare` on every host, restart the unit |
+| `FileNotFoundError: ~/.config/flare/agent.token` | the **preparation token** is our own shared secret, not an Orbit identity | write identical ≥32 random bytes, mode 600, on every host |
+| `b12x.loader build dependency is missing: /usr/include/python3.12/Python.h` on ranks 1–3 | `python3-dev` missing (the b12x loader compiles a small extension at load) | apt install `python3-dev` |
+| `OSError: [Errno 28] No space left on device` during export | the package blob store lives under `[paths].package_cache` — default `~/.cache/flare/packages` **on rank zero**, 37 GB before the disk filled | point `package_cache` (and `export_staging`) at the NAS; keep `kernel_cache` local so the tuning cache survives |
+| launch refused: `N GiB free under the agent cache; this launch stores 121.8 GiB there` | every rank copies the package to its own agent cache | free ≥130 GiB per host |
+| `package RoCE kernels use proxy ABI 10, this runtime provides 9` | **upstream skew**: the pinned `flashinfer` (`local-inference-lab` @03de0b66) has `ROCE_ABI_VERSION 10`, the vendored `helios/native/roce_proxy.c` in our 2026-10-21 clone still had 9 | `git pull` — commit `e350914a3b` "Match native RoCE transport to the pinned FlashInfer proxy ABI" (2026-10-21 16:31) — rebuild the binary, reinstall on every agent. Do **not** patch the constant: the two C files differ by a whole ring transport (317 diff lines) |
+
+Disk after the bring-up: dgx5 383 GB, dgx6 342 GB, dgx7 220 GB, dgx8 120 GB
+free (the 122 GiB package copy per host is the large item).
+
+## Useful knobs for the next pass
+
+- `flare orbit run --scheduler-policy throughput | balanced`, `--max-concurrency N`,
+  `--kv-cache-limit {memory,context,concurrency}`, `--memory-limit`.
+- `--mtp-draft-tokens N` (package default 5), `--no-mtp`,
+  `--speculative-token-capacity` — the DSpark path.
+- Newest upstream commit adds **adaptive verification scoring**, and
+  `docs/glm-batched-speculation-defect.md` documents a Spark-only batched
+  speculation corruption fixed at `22887c1` (rare TP4 device error at 16
+  concurrent remains open) — read it before trusting multi-request batches.
+- `ignition export --help` lists per-recipe bloat/knob controls
+  (`--expert-scale-compression csf`, `--online-mxfp8`, `--packed-verification`,
+  `--prompt-logprobs`, `--tuning-gpus`).