upstream watch 10-15: v0.32.0rc2 imminent (flashinfer 0.7.1 + SM12x NVFP4 KV + DSpark scheduler fix), SparkRing 2026.10.4 validates the MiMo rebase image + adds TP2-DCP2, DGPP adds Qwen3.8-27B, Nemotron-3.5-Bolt 24GB single-Spark, Ling-3.1-flash 572B day-0 NVFP4, net-sircl DAC plugin, CRS812 fabric resolution

specs/tech-architecture/upstream-watch-2026-10-15.md

@@ -0,0 +1,126 @@
+# Upstream watch — 2026-10-15
+
+Delta pass over the same sources as [`upstream-watch-2026-10-14.md`](upstream-watch-2026-10-14.md)
+(one day, but the ecosystem is moving fast). Baseline changed since then:
+kindling 2x re-pinned to **vLLM v0.31.0** (`spark-glm53:v10`, bench 95/100),
+gr8 + DGPP already deployed and quality-gated (89 / 88 def / 86 low),
+TEB pin now **v2.8.0** (97 scenarios).
+
+## 1. Engine stack
+
+### vLLM — v0.32.0 is imminent (the upgrade to plan)
+- Stable is still **v0.31.0** (our base — current). `v0.32.0rc2` cut
+  **2026-10-15 09:50 UTC** (`7960eeb9de29`); 0.31.1 skipped (rc0 only, 10-06).
+- What 0.32.0 carries that matters to GB10/our lanes (merged since 0.31.0):
+  - **#60948 FlashInfer → 0.7.1** (10-14) — the pairing we were waiting for;
+    0.7.1 has quantized expert-parallel MoE on SM12x, compact GLM53-NoPE
+    caches, DS4.1 dual-cache sparse MLA (see 10-14 spec).
+  - **#46963 NVFP4 KV cache on SM12x via FlashInfer** (10-07) +
+    **#60452 NVFP4 KV decode with XQA on SM12x** (10-10) — KV capacity lever
+    for the kindling lanes (NVFP4 KV ≈ 2× FP8 pool).
+  - **#59468 don't reserve DFlash/DSpark draft slots in the scheduler** (10-15)
+    — scheduler fix on our exact spec-decode path.
+  - **#59511 [Perf][DFlash] capture context combine + anchor in draft CUDA
+    graph** (10-14) — decode perf on our drafting path.
+  - #61437 SM121 TP=2 QSA projection plans; #61274 auto-disable DeepGEMM on
+    SM12x; #57378 NVML-memory fallback (UMA-adjacent); #58128 FP8 KV for
+    MiMo-V2.6-Flash.
+  - Structured-output hardening: #60359/#61047/#60748/#59061/#58862.
+- **Action:** when v0.32.0 stable lands (days), re-pin the kindling base
+  v0.31.0 → v0.32.0 and rebase the 38 experimental overlays (delta is much
+  smaller than the nightly→0.31.0 jump — same procedure, now checklist-ized
+  in `docker-images/glm53-kindling/AGENTS.md`). Expect flashinfer
+  0.7.0.post1 → 0.7.1 pin bump too. Bench-gate vs the 95/100 anchor.
+
+### flashinfer
+- Stable **0.7.1** (10-09); 0.7.2 still rc3 (10-13). vLLM main pins 0.7.1.
+
+### lab (local-inference-lab) — r40 still pending, but watch the new channel
+- No new source.lock rev: vllm fork tip still `ff292a9c63` (10-04), LMCache
+  fork 09-14. Their flashinfer fork is where the action moved (b12x paged
+  attention, DSA top-k, BF16-GEMM paths — commits 10-15).
+- **New info source: `rtx6kpro` → `daily-summaries/YYYY-MM/*.md`** — the lab
+  publishes daily community/Discord digests (latest covers 10-14). Notable
+  from the 10-14 issue: LIL stable build broke on `b12x.loader._csf_source`
+  (fixed next pull); LMCache L1/L2 occasional misses force full prefill on
+  GLM-5.3-Flash; dealignai UNCENSORED-NVFP4 needs
+  `*.self_attn.in_proj_qkvgfab` added to `quantization_config.ignore`.
+
+### SparkRing 2026.10.4 (10-14) — validates the MiMo rebase, adds DCP2
+- Release image = `dev-20261013-dcp2prefill-status038-sircl-tp6`
+  (`@sha256:e21ba936d4fd…`) — **exactly the image pre-pulled on dgx3/4/6/8**
+  for the MiMo rebase. All 2026.10.2 profiles admitted unchanged.
+- New research-only profiles: **`glm53-flash-nvfp4-spark-tp2-dcp2`** (GLM-5.3-Flash
+  CSF on 2 Sparks with decode-context-parallel 2 — KV split across the pair,
+  ~2× context, 16 concurrent) and `glm53-nvfp4-csf-tp6` (full GLM-5.3 on 6
+  Sparks, not yet run on hardware). They use LIL **CSF checkpoints**
+  (`GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD`, `GLM-5.3-NVFP4-CSF`,
+  `Qwen3.8-Flash-Next-NVFP4-MXFP8-CSF-QAD`) under the LIL license.
+- **Action:** proceed with the MiMo 4x rebase on this image (plan unchanged).
+  DCP2 is a candidate datapoint for the 2x lane (context ceiling) — but it
+  needs the CSF checkpoint + their toolchain, so treat as a separate
+  experiment lane, not a kindling change.
+
+### net-sircl (new, via lab digest 10-14)
+- Plugin: NCCL between Sparks over **direct cables** (no switch) via a .so +
+  topology file + 4 env vars + one fabric-tool NIC-forwarding setup.
+  Relevant to `2nodes-dac` — worth a fabric experiment when the pair is free.
+
+## 2. Engines we've tested — deltas
+
+- **gr8** (kindlingai): active daily — S2 scheduler (prefill quantum to the
+  shortest prompt, notice-target sizing, cancel attribution), **host-heap
+  return after weight load** (their analogue of our UMA/page-cache issue),
+  boot-time drafting-cost overrides (#25, 10-15). Re-test after a quiet week.
+- **DGPP** (HawkBearPig): now actively adding **Qwen3.8-27B** — YaRN rope
+  ramp + 768K template check, prefill-attention TMA rewrites (10-15).
+  Direct A/B candidate for `qwen38-27b-nvfp4-unsloth` (our 21.7 t/s winner);
+  their earlier claim: +40% over published vLLM on Qwen single-node.
+- **TensorFold** (new stack family, thread 384789 + MiaAI/gilesco repos):
+  EXL3 GLM-5.3-Flash on 2× Spark with NVMe weight streaming + SSD cache
+  offload; "1B tokens, several days, no issues" (ajvazan); MiaAI stack pins
+  an old TensorFold, gilesco fork backports the latest; python branch frozen
+  pending CUDA engine v1.0.10 (PR 804). Full GLM-5.3 (753B) 3-bit EXL3 TP4
+  @43 t/s code single-stream (thread 385042). Watch PR 804; not adoptable
+  yet (frozen python branch, EXL3 quality untested by us).
+- **TAI engine**: 96/100 on current TEB (sparkdash standard settings,
+  jcagle #840) — highest community quality report on the GLM-5.3 line;
+  engine/repo not identified in-thread — track.
+
+## 3. Fabric (CRS812 thread 385776 — resolved)
+- 185 Gbps achieved after **rebooting with the cables connected** (the
+  regression was kernel 7.0.0 + cold-plugged ports); 176/200 after
+  `ethtool` rate fix. 4 logical interfaces (separate subnets) only needed to
+  max PCIe — with ≤4 nodes one interface per node is right.
+- **Action:** compare against our own fabric-check AG numbers; if a Spark
+  pair was cabled while powered off, a reboot is the first fix.
+
+## 4. Models
+
+| Model | Note | Action |
+|---|---|---|
+| **NVIDIA Nemotron-3.5-Bolt-74B-A9B** (10-15) | 74B/A9B; official **24 GB GGUF** fits ONE Spark; DSpark-GGUF drafts exist; "intended for 2-bit" | cheap single-node test lane (A9B decode should be fast); quality unknown vs Qwen3.8-Flash-Next |
+| **Ling-3.1-flash** (inclusionAI) | on HF now; **572B** total (bigger than the 10-14 estimate), AA ~41, MIT; day-0 `cbfight/Ling-3.1-flash-NVFP4-lossless8` | 4x-lane candidate (NVFP4 ≈ GLM-5.3-Flash class size); watch for Spark recipes |
+| GLM-5.3-Flash **EXL3 wave** | turboderp 2.10bpw self-calibrated (2.5K dl), yamz-labs (+uncensored), cbert33 "DGX-Sliced", AxonZeta Focus 2.05bpw | feeds TensorFold/veloGB10; quality-gate before any lane swap |
+| LIL **CSF checkpoints** (3) | GLM-5.3-Flash / GLM-5.3 / Qwen3.8-Flash-Next CSF variants for sparkring profiles | LIL-licensed; only via sparkring profiles |
+| MiMo-V2.6 re-cuts (10-09) | unchanged since 10-14 spec | covered by the rebase task |
+
+## 5. Harness
+- Community on current TEB: "high reasoning no longer enough for 90+"
+  (re-baseline discipline matters; our anchors are v2.8.0/97-scenario).
+- TAI 96/100 and MiaAI C8 numbers are same-harness-comparable only if they
+  stamp the version — ask before cross-comparing.
+
+## Recommended order (updated)
+
+1. **MiMo 4x rebase** — 2026.10.4 validates the pre-pulled image; do it when
+   dgx3/4/6/8 free (they're off now).
+2. **v0.32.0 stable → kindling base re-pin** — flashinfer 0.7.1 + SM12x
+   NVFP4 KV + DSpark scheduler/DFlash graph fixes in one bump; bench-gate
+   vs 95/100.
+3. **DGPP Qwen3.8-27B A/B** vs `qwen38-27b-nvfp4-unsloth` (their 768K YaRN
+   lands this week; single-node, cheap).
+4. **Nemotron-3.5-Bolt 24GB** single-Spark quick lane (new, free capacity).
+5. **net-sircl** DAC experiment on `2nodes-dac` (fabric headroom check).
+6. Watch: lab r40 cut, TensorFold CUDA PR 804, TAI engine identity,
+   Ling-3.1-flash Spark recipes, v0.32.0rc notes until stable.