upstream-watch-2026-10-14.md 8.3 KB
Upstream watch — 2026-10-14
Research pass over the sources we track (NVIDIA forums, engine repos, model
repos, AA) since our last cuts (GLM-5.3-Flash r39 = 2026-09-21, kindling
spark-glm53:v10 = 2026-09-28, sparkring = 2026-09-29, qwen38 = vLLM
0.27.1, dsv41 = tonyd2wild 09-10). Purpose: pick upgrades and new test
candidates for the lanes.
1. Engine stack upgrades
FlashInfer absorbs b12x (P1 — the big one)
local-inference-lab/b12xarchived 2026-10-06; README replaced by a FlashInfer migration guide. Kernels + preparation/autotune/compile/ workspace/checkpoint infra now live atflashinfer/experimental/b12x(merge: flashinfer PR #5767, commitfc8fdc17f70, 2026-10-06).- FlashInfer v0.7.1 (2026-10-09) / v0.7.2rc3 (10-13) highlights: quantized
expert-parallel MoE on SM12x (GB10!), fused Kimi-K3 spec verification,
cuDNN linear-attention prefill on Blackwell, DeepSeek-V4.1 dual-cache
sparse MLA, compact GLM53-NoPE caches.
flashinfer-python[b12x]extra; needs torch ≥2.12 + CUTLASS DSL 4.7.1. - Our pins: flashinfer 0.6.15/0.6.17 + standalone b12x (r39:
c2dc1cf022). - Action: r40 profile once the lab cuts a vLLM+flashinfer-0.7.x pairing
(jovian vLLM tip is 10-04
ff292a9c63— predates the merge, so their next cut is the one to take). Watchlocal-inference-lab/vllmtags. - The
schema 22tuning-decision invalidation (b12x 10-06) means autotune caches from old cuts should be dropped on upgrade.
vLLM mainline (P2 for qwen38 lanes)
- v0.28 (08-26) → v0.29 (09-09) → v0.30 (09-22) → v0.31.0 (10-05). We serve qwen38 on 0.27.1. Real recipes already run 0.30/0.31 on Spark: MiaAI 2x Flash-Next (0.30, TP2+EP+MTP3), chisel-spark-recipes (0.31.0).
- vLLM #58578 / PR #59740: reduced-vocabulary draft head (50,858-token drafter slice, 0.5 GB instead of 2.5 GB) — packaged as a plugin in yuanlin2004/chisel-spark-recipes for Qwen3.8-27B-FP8 (23.1 tok/s c1).
- Action: rebase qwen38-flash-next / qwen38-27b lanes on 0.31.0 (or the recipe's pinned stack) and re-bench; evaluate the draft-head plugin.
sparkring rebase (P2)
ghcr.io/fujitsupolycom/sparkringmoved:dev-20261013-dcp2prefill- status038-sircl-tp6(10-13!) — DCP2 prefill, TP6, sircl 0.6.0, status038; akrakenline since 10-01. Our recipe pins the 09-29 digest (71d41057…).- MiMo-V2.6 model repos re-cut 2026-10-09 (Flash-MOPD, Pro-MOPD, Flash-RL, Pro-RL). Our lanes serve the older checkpoints.
- Action: re-pin the MiMo lanes to the newest image + 10-09 checkpoints, re-bench (quality may move — RL is new for Flash).
2. New engines (test candidates — evaluate, don't adopt blindly)
kindlingai/gr8 (P1 interest — same org as our kindling stack)
- From-scratch CUDA engine, full GLM-5.3 (600B, 256 experts) on 3-6
Sparks, own RoCE-v2 verbs fabric (no vLLM/PyTorch/NCCL), OpenAI API,
one-line
./gr8 glm53 up. Pushed 2026-10-14; beta. - Model:
kindlingai/glm-5.3-exlr8-k2-k3-k4(updated 10-14, 300K dl). - Numbers (their RigMark runs, clocks capped 2200): TP=4 decode 49.3/30.0 t/s code/prose, aggregate 64 @4 streams, prefill 8K cold 1,374 t/s, KV pool 310K tok (415K with int4 KV). DFlash2+MTP drafting.
- The license-locked full model becomes servable — flag: Z.ai license is
license:other; quants redistribute weights, commercial use unclear. - Action: install + 4x lane test + tool-eval quality gate. Compare with GLM-5.3-Flash (AA 41.8) vs full GLM-5.3 (AA 44.8, max-thinking 50.3).
DGPP (HawkBearPig/dgpp) (P2)
- C++/CUDA engine for Spark, 120★, pushed 10-14, 107-post forum thread (383406). OpenAI API, 1/2/4-node templates.
- Matrix: GLM-5.3-Flash unsloth FP8 w4; GLM-5.3-Flash NVFP4-FP8 hybrid w2/w4; full GLM-5.3 int4/int8 w4 (272K FP8 KV); MiMo-V2.6-Flash-RL w2/w4; DS4.1-Flash w4; Qwen3.8-Flash-Next NVFP4 w1/w2 (mmap n-gram, decode graphs, fp8 dense — +40% over published vLLM on Qwen single-node); GLM-4.7.
- Numbers (5-class spread, warm decode): Flash-Next NVFP4 w2 62-75 t/s; GLM-5.3-Flash NVFP4 w4 50-58 t/s; full GLM-5.3 w4 25-29 t/s (42-47 @C4).
- Action: bench DGPP vs our vLLM lanes on (a) Qwen3.8-Flash-Next w1, (b) MiMo-RL w2 — quality via tool-eval-bench, same harness both sides.
veloGB10 v0.7.0 (sf-stav) (P3 — watch)
- Rust engine, EXL3-focused; Qwen3.8-Flash-Next EXL3 3.05bpw TP=2 @186 t/s (their claim); NVFP4 supported but unoptimized; TP=4 pending; no vision.
- Model: doth4580/Qwen3.8-Flash-Next-EXL3-3.05bpw. Re-test when TP4/NVFP4 mature.
dsgx10 (forum 385803) (P3 — watch)
- From-scratch DS4.1-Flash engine for 4×GB10 switchless ring; +5-70% over knapcio v2.2/v2.4 baselines, bit-exact, built with Claude. One post so far (10-13); no repo link yet. Track the thread.
3. Model landscape (AA / HF)
| Model | AA index | License | Size | Note |
|---|---|---|---|---|
| MiMo-V2.6-Pro | 46.3 | MIT | ~1TB | overkill, one-shot (unchanged) |
| Muse Wonder Full (Meta) | 56 | ? | ? | 10-12, likely closed/large — check |
| Kimi K3 (max) | (top open) | other | 2.78 TB | too big |
| Grok 4.7 | 46 | closed | — | — |
| Qwen3.8 Max (0902) | 45 | closed | — | — |
| GLM-5.3 (full) | 44.8 | license:other | 600B | gr8/DGPP make it servable; license q. |
| GLM-5.3-Flash | 41.8 | MIT | 320B | our flagship; zai repo touched 10-12 (minor) |
| Ling 3.1 Flash (InclusionAI) | 41 | MIT | ~350-400B? (512/8 experts, 262K ctx) | released ~10-13; watch for day-0 quants/recipes |
| MiMo-V2.6-Flash | 38 | MIT | 311B | re-cut 10-09; RL variant new |
| DeepSeek V4.1 Flash | 39 (max) | MIT | — | our dsv41 lanes |
| Qwen3.8-Flash-Next | 40 | Apache | 125B A6B | the coder model; see new lanes |
Qwen3.8-Flash-Next 2x TP2+EP lane (P2 — new lane)
- Forum 385731 (MiaAI recipe, vLLM 0.30): 43-46 t/s single, 174 t/s aggregate @8-16 conc, 2.9K t/s prefill, 158/164 HumanEval, 74.2% aider polyglot, 262K ctx, MTP=3, fp8 KV. Soak: 2-h agent run zero errors.
- Our flash-next recipes are single-node. Action: 2x lane with
--tensor-parallel-size 2 --enable-expert-parallel, MTP=3; bench with tool-eval-bench + concurrency sweep.
Full GLM-5.3 on the 4-node lane (P2 — new capability)
- Via gr8 (exlr8) or DGPP (int4/int8-mix). The presentation's "license-locked, unusable" stance should be revisited for personal/research use; commercial remains questionable. The 4x cluster is idle — this is the highest-ceiling quality upgrade available without new hardware.
4. Meta / infra notes
- MikroTik CRS812 200G thread (385776, 10-13): 4× Spark via CRS812 with 2×400G→2×200G breakout: NCCL AG 20.4 GB/s vs 25 theoretical (MTU 9000, NCCL 2.28.9 RoCE, FEC on; logical-interface bonding attempted). We run the same class of switch (SW2-MUC) — compare our own fabric-check AG numbers; the thread's resolution may apply to us.
- tool-eval-bench moved again: "with the most recent TEB,
highreasoning is no longer enough for 90+" (forum 381350 #827, 10-12); community reports 90-93 on current TEB where older pins read lower. Our pin: v2.7.0. Any cross-engine A/B must share a harness version; consider a re-baseline of the anchor lanes before comparing against community numbers. - RigMark (github.com/alexellis/rigmark) — the benchmark gr8 uses (reasoning_effort low, clock-capped, machine-readable kindling.json). Worth a look as a cross-engine comparison harness.
- eugr/spark-vllm-b12x nightlies continue (10-14); our qwen38-flash-next lane pins the 09-25 digest because of the garbling bug — consider testing a fresh pinned digest for the fix.
- The 16-node operator thread (385107) wants GLM-5.3-full NVFP4 16x, DS4.1+GLM-5.3-Flash 4+4, MiMo-Pro 16x on spark-vllm-docker — our shared recipe repo in the wild; their TP-over-switch questions may produce reusable data.
Recommended order
- r40 cut watch (flashinfer-0.7.x pairing) — wait for the lab's tag, then cut the profile for kindling/flash lanes.
- gr8 on the 4x lane — full GLM-5.3; quality-gate with tool-eval-bench.
- sparkring rebase + MiMo 10-09 re-cuts — cheap, existing lanes.
- Qwen3.8-Flash-Next 2x EP lane — the coder-endpoint use case.
- DGPP bench (Qwen w1, MiMo-RL w2) — cross-engine comparison.
- Watch: veloGB10 TP4, dsgx10 repo, Ling-3.1-flash quants, TEB re-baseline.