launcher/specs/tech-architecture/flare-bringup.md

FLARE bring-up on the Spark stand (2026-10-22)

NVIDIA published NVIDIA/flare-local-inference on 2026-10-20: a local inference server for SM120/SM121 GPU. Python (ignition) prepares a model with PyTorch + FlashInfer/b12x, tunes kernels, compiles forwards and captures CUDA graphs into a runtime package; a Rust runtime (helios, driven by the flare CLI) serves that package without Python. flare orbit drives multi-host ranks through a per-host flare agent.

Our earlier FLARE lane (dsv41-flash-1017-*, 0/100, kernel timeouts) ran a closed build; the source explains what we were missing.

What now works

deepseek-ai/DeepSeek-V4.1-Flash-1017 (the production model, pinned revision 1bae0e1088993b9b82de08f59de5be5a34e42170) prepared at TP4 across dgx5–dgx8 and served by FLARE:

• model id DeepSeek-V4.1-Flash, max_model_len 1,048,576
• KV cache 24.0 GiB/rank → 28,923,136 tokens, 32 shared snapshot slots
• TP collectives over RoCE (2 rails per peer over roceP2p1s0f0,rocep1s0f0)
• package: /mnt/nas/ai/flare/dsv41-flash-tp4.safetensors (7 MB index) + 477 GiB of content-addressed blobs beside it

First-boot numbers (single request unless noted, streaming, thinking on):

measure	FLARE	production SGLang EP2 4x (2026-10-18 run)
prefill	~1,430 t/s (2K prompt, TTFT 1.43 s) · ~1,700 t/s (8K, 4.67 s)	2,800–3,127 t/s
decode c1	19.2 t/s	47.8–50.7 t/s
decode c2	29.6 t/s aggregate	64.2–68.3 t/s
cached 8K prefix	TTFT 4.67 → 2.01 s	—

So the first boot is ~2.4x slower on decode than our production lane. Nothing is tuned yet: --max-concurrency stayed at the package's 4, the scheduler policy is balanced, and the graph/pool geometry is whatever the preparation picked. That is the next thing to move, not a verdict.

Procedure (dgx5–8, cluster sparks)

# one-time per host
sudo apt-get install -y build-essential cmake ninja-build pkg-config \
    protobuf-compiler libssl-dev python3-dev liburing-dev
curl -LsSf https://astral.sh/uv/install.sh | sh

# ~/.config/flare/config.toml on every host
[paths]
hub_cache = "/mnt/nas/ai/huggingface/hub"
kernel_cache = "/home/kilork/.cache/flare/kernels"
package_cache = "/mnt/nas/ai/flare/packages"
export_staging = "/mnt/nas/ai/flare/staging"
cuda = "/usr/local/cuda"

[memory]
reserve = "4G"

git clone --depth 1 https://github.com/NVIDIA/flare-local-inference ~/flare-src
cd ~/flare-src && uv sync --locked --extra prepare && \
  cargo build --release --manifest-path flare/Cargo.toml      # ~6 min, 20 MB binary

# the agent binary MUST be this same build (see the ABI note below)
scp flare/target/release/flare <peer>:/tmp/flare-new && \
  ssh <peer> 'systemctl --user stop flare-agent; mv /tmp/flare-new ~/.local/lib/flare-agent/flare; systemctl --user start flare-agent'

# preparation token: identical >=32 random bytes, mode 600, on every host
head -c 48 /dev/urandom | base64 > ~/.config/flare/agent.token && chmod 600 …

# rank zero
uv run ignition export deepseek-v4.1-flash --gpus 0 --ranks 4 \
  --rendezvous <rank0-ip>:29600 --max-concurrency 4 \
  --output /mnt/nas/ai/flare/dsv41-flash-tp4.safetensors --verbose
# each peer, while rank zero waits
uv run ignition join <rank0-ip>:29600 --rank N --ranks 4

# serve (rank zero runs the HTTP API)
flare orbit run gx10-dgx5:/mnt/nas/ai/flare/dsv41-flash-tp4.safetensors \
  --cluster sparks --host 0.0.0.0 --port 30000 --detach \
  --dangerously-approve-key <fp> --dangerously-accept-publisher local-development
flare orbit status | attach sparks | stop sparks | ls

Failure modes hit on the way (all fixed)

symptom	cause	fix
orbit status → unreachable: handshake read failed	agents were the 2026-10-20 release, the controller a fresh build	install the freshly built flare as ~/.local/lib/flare-agent/flare on every host, restart the unit
FileNotFoundError: ~/.config/flare/agent.token	the preparation token is our own shared secret, not an Orbit identity	write identical ≥32 random bytes, mode 600, on every host
b12x.loader build dependency is missing: /usr/include/python3.12/Python.h on ranks 1–3	python3-dev missing (the b12x loader compiles a small extension at load)	apt install python3-dev
OSError: [Errno 28] No space left on device during export	the package blob store lives under [paths].package_cache — default ~/.cache/flare/packages on rank zero, 37 GB before the disk filled	point package_cache (and export_staging) at the NAS; keep kernel_cache local so the tuning cache survives
launch refused: N GiB free under the agent cache; this launch stores 121.8 GiB there	every rank copies the package to its own agent cache	free ≥130 GiB per host
package RoCE kernels use proxy ABI 10, this runtime provides 9	upstream skew: the pinned flashinfer (local-inference-lab 03de0b66">@03de0b66) has ROCE_ABI_VERSION 10, the vendored helios/native/roce_proxy.c in our 2026-10-21 clone still had 9	git pull — commit e350914a3b "Match native RoCE transport to the pinned FlashInfer proxy ABI" (2026-10-21 16:31) — rebuild the binary, reinstall on every agent. Do not patch the constant: the two C files differ by a whole ring transport (317 diff lines)

Disk after the bring-up: dgx5 383 GB, dgx6 342 GB, dgx7 220 GB, dgx8 120 GB free (the 122 GiB package copy per host is the large item).

Useful knobs for the next pass

• flare orbit run --scheduler-policy throughput | balanced, --max-concurrency N, --kv-cache-limit {memory,context,concurrency}, --memory-limit.
• --mtp-draft-tokens N (package default 5), --no-mtp, --speculative-token-capacity — the DSpark path.
• Newest upstream commit adds adaptive verification scoring, and docs/glm-batched-speculation-defect.md documents a Spark-only batched speculation corruption fixed at 22887c1 (rare TP4 device error at 16 concurrent remains open) — read it before trusting multi-request batches.
• ignition export --help lists per-recipe bloat/knob controls (--expert-scale-compression csf, --online-mxfp8, --packed-verification, --prompt-logprobs, --tuning-gpus).