scripts: flare-upstream-watch — FLARE moves daily, check it before every experiment

Records the pin (3b9e86a -> HEAD 1fd1f91 as of 2026-10-23), prints the commits
since it with a flag on the ones that invalidate our agent binaries or prepared
packages (RoCE/ABI, FlashInfer pins, speculation, recipes, prepare/export,
agents), and documents the lesson from the bring-up: the ABI fix that unblocked
us landed eight hours after our clone.

The first run already shows MiniMax-M3.1-Flash becoming a first-class FLARE
recipe with packed DSpark speculation — the model that was just downloaded.

scripts/flare-upstream-watch.sh

@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# Watch NVIDIA/flare-local-inference upstream.
+#
+# FLARE moves several times a day (12+ commits in one night), and our clones,
+# the per-host agent binaries and the prepared packages all have to agree with
+# it: the RoCE proxy ABI is pinned in the repo, the model recipes change, and a
+# package prepared with one revision can be refused by another.
+#
+#   scripts/flare-upstream-watch.sh           # what changed since the pin
+#   scripts/flare-upstream-watch.sh --bump    # move the pin to current HEAD
+#
+# The pin lives next to this file's spec (specs/tech-architecture/
+# flare-upstream.pin) as a single line: <sha> <committer-date>.
+set -euo pipefail
+
+REPO="NVIDIA/flare-local-inference"
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+PIN_FILE="$ROOT/specs/tech-architecture/flare-upstream.pin"
+API="https://api.github.com/repos/$REPO"
+
+pin_sha=""
+pin_date=""
+if [[ -f "$PIN_FILE" ]]; then
+  read -r pin_sha pin_date <"$PIN_FILE" || true
+fi
+
+json="$(curl -fsS -m25 "$API/commits?per_page=40")"
+head_sha="$(printf '%s' "$json" | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["sha"])')"
+head_date="$(printf '%s' "$json" | python3 -c 'import json,sys; d=json.load(sys.stdin)[0]["commit"]["committer"]; print(d["date"][:16])')"
+
+echo "upstream: $REPO"
+echo "  HEAD   ${head_sha:0:10}  $head_date"
+echo "  pinned ${pin_sha:0:10}  ${pin_date:-never}"
+
+if [[ "${1:-}" == "--bump" ]]; then
+  printf '%s %s\n' "$head_sha" "$head_date" >"$PIN_FILE"
+  echo "  pin moved to ${head_sha:0:10}"
+  exit 0
+fi
+
+if [[ -z "$pin_sha" ]]; then
+  echo "  no pin recorded — run with --bump after reviewing"
+  exit 0
+fi
+
+if [[ "$pin_sha" == "$head_sha" ]]; then
+  echo "  up to date"
+  exit 0
+fi
+
+echo
+echo "new commits since the pin:"
+TMP_JSON="$(mktemp)"
+trap 'rm -f "$TMP_JSON"' EXIT
+printf '%s' "$json" >"$TMP_JSON"
+python3 - "$pin_sha" "$TMP_JSON" <<'PY'
+import json, re, sys
+
+pin = sys.argv[1]
+commits = json.load(open(sys.argv[2]))
+SIGNIFICANT = re.compile(
+    r"roce|abi|flashinfer pin|speculat|dspark|dflash|mtp|recipe|deepseek|glm"
+    r"|mimo|qwen|kimi|package|prepare|export|agent|orbit",
+    re.I,
+)
+
+stopped = False
+for c in commits:
+    sha = c["sha"]
+    if sha == pin:
+        stopped = True
+        break
+    msg = c["commit"]["message"].splitlines()[0]
+    date = c["commit"]["committer"]["date"][:16]
+    flag = "  ⚠" if SIGNIFICANT.search(msg) else "   "
+    print(f"{flag} {sha[:10]} {date} {msg[:110]}")
+if not stopped:
+    print("(pin is older than the 40 commits listed — fetch more)")
+PY
+
+cat <<'EOF'
+
+Before re-running a FLARE experiment, pull and rebuild BOTH sides — the
+per-host `flare` agent binaries and the prepared packages must come from the
+same revision, or the runtime refuses (see the "proxy ABI" row in
+specs/tech-architecture/flare-bringup.md).
+EOF

specs/tech-architecture/flare-bringup.md

@@ -110,4 +110,36 @@ free (the 122 GiB package copy per host is the large item).
   concurrent remains open) — read it before trusting multi-request batches.
 - `ignition export --help` lists per-recipe bloat/knob controls
   (`--expert-scale-compression csf`, `--online-mxfp8`, `--packed-verification`,
-  `--prompt-logprobs`, `--tuning-gpus`).
+  `--prompt-logprobs`, `--tuning-gpus`).
+
+## Upstream watch (FLARE moves several times a day)
+
+The repo shipped 12+ commits in the twelve hours after our first bring-up, and
+one of them was the RoCE ABI fix that unblocked us at all. Treat the upstream as
+a moving target and check it **before** every FLARE experiment:
+
+```bash
+scripts/flare-upstream-watch.sh          # HEAD vs the pin, flagged commits
+scripts/flare-upstream-watch.sh --bump   # move the pin after reviewing
+```
+
+The pin lives in [`flare-upstream.pin`](flare-upstream.pin). Commits touching
+RoCE/ABI, FlashInfer pins, speculation (dspark/dflash/mtp), recipes, package
+prepare/export and the agents are marked ⚠ — those are the ones that invalidate
+our agent binaries or prepared packages. **Pull and rebuild both sides together**;
+a package prepared by one revision is refused by another (`proxy ABI`).
+
+What the watch turned up on 2026-10-23 (pin `3b9e86a` → HEAD `1fd1f91`, 17
+commits in 14 hours):
+
+- **`Support MiniMax M3.1 Flash with packed DSpark speculation`** — the model we
+  had just downloaded (252 GB, `MiniMaxM3SparseForConditionalGeneration`,
+  mxfp8 block [1,32]) is now a first-class FLARE recipe.
+- `Add Gemma 4 text recipe with Assistant MTP`, `Serve and prepare over hosts
+  cabled as a chain`, `Repair prefix cache retention`, `Enable ARM SHA-256
+  acceleration for package hashing` (the Pi-class agent path),
+  `Set recipe defaults to 32 requests and bounded speculative catalogs`,
+  `Make multi-host prefix records safe when a peer fails`.
+- The DeepSeek side keeps moving too: `Preset DeepSeek V4.1 packed GPU
+  speculation`, `Add a serving option for adaptive verification scoring alpha`,
+  `Resume preparation checkpoints and share compiled forwards across ranks`.

specs/tech-architecture/flare-upstream.pin

@@ -0,0 +1 @@
+3b9e86a000000000000000000000000000000000 2026-10-22T15:48