|
|
@@ -0,0 +1,147 @@
|
|
|
+# Upstream watch — 2026-10-14
|
|
|
+
|
|
|
+Research pass over the sources we track (NVIDIA forums, engine repos, model
|
|
|
+repos, AA) since our last cuts (GLM-5.3-Flash r39 = 2026-09-21, kindling
|
|
|
+`spark-glm53:v10` = 2026-09-28, sparkring = 2026-09-29, qwen38 = vLLM
|
|
|
+0.27.1, dsv41 = tonyd2wild 09-10). Purpose: pick upgrades and new test
|
|
|
+candidates for the lanes.
|
|
|
+
|
|
|
+## 1. Engine stack upgrades
|
|
|
+
|
|
|
+### FlashInfer absorbs b12x (P1 — the big one)
|
|
|
+- `local-inference-lab/b12x` **archived 2026-10-06**; README replaced by a
|
|
|
+ FlashInfer migration guide. Kernels + preparation/autotune/compile/
|
|
|
+ workspace/checkpoint infra now live at `flashinfer/experimental/b12x`
|
|
|
+ (merge: flashinfer PR #5767, commit `fc8fdc17f70`, 2026-10-06).
|
|
|
+- **FlashInfer v0.7.1 (2026-10-09)** / v0.7.2rc3 (10-13) highlights: quantized
|
|
|
+ **expert-parallel MoE on SM12x** (GB10!), fused Kimi-K3 spec verification,
|
|
|
+ cuDNN linear-attention prefill on Blackwell, **DeepSeek-V4.1 dual-cache
|
|
|
+ sparse MLA**, **compact GLM53-NoPE caches**. `flashinfer-python[b12x]`
|
|
|
+ extra; needs torch ≥2.12 + CUTLASS DSL 4.7.1.
|
|
|
+- Our pins: flashinfer 0.6.15/0.6.17 + standalone b12x (r39: `c2dc1cf022`).
|
|
|
+- **Action:** r40 profile once the lab cuts a vLLM+flashinfer-0.7.x pairing
|
|
|
+ (jovian vLLM tip is 10-04 `ff292a9c63` — predates the merge, so their next
|
|
|
+ cut is the one to take). Watch `local-inference-lab/vllm` tags.
|
|
|
+- The `schema 22` tuning-decision invalidation (b12x 10-06) means autotune
|
|
|
+ caches from old cuts should be dropped on upgrade.
|
|
|
+
|
|
|
+### vLLM mainline (P2 for qwen38 lanes)
|
|
|
+- v0.28 (08-26) → v0.29 (09-09) → **v0.30 (09-22)** → **v0.31.0 (10-05)**.
|
|
|
+ We serve qwen38 on 0.27.1. Real recipes already run 0.30/0.31 on Spark:
|
|
|
+ MiaAI 2x Flash-Next (0.30, TP2+EP+MTP3), chisel-spark-recipes (0.31.0).
|
|
|
+- vLLM #58578 / PR #59740: reduced-vocabulary draft head (50,858-token
|
|
|
+ drafter slice, 0.5 GB instead of 2.5 GB) — packaged as a plugin in
|
|
|
+ yuanlin2004/chisel-spark-recipes for Qwen3.8-27B-FP8 (23.1 tok/s c1).
|
|
|
+- **Action:** rebase qwen38-flash-next / qwen38-27b lanes on 0.31.0 (or the
|
|
|
+ recipe's pinned stack) and re-bench; evaluate the draft-head plugin.
|
|
|
+
|
|
|
+### sparkring rebase (P2)
|
|
|
+- `ghcr.io/fujitsupolycom/sparkring` moved: `dev-20261013-dcp2prefill-
|
|
|
+ status038-sircl-tp6` (10-13!) — DCP2 prefill, **TP6**, sircl 0.6.0,
|
|
|
+ status038; a `kraken` line since 10-01. Our recipe pins the 09-29 digest
|
|
|
+ (`71d41057…`).
|
|
|
+- **MiMo-V2.6 model repos re-cut 2026-10-09** (Flash-MOPD, Pro-MOPD,
|
|
|
+ Flash-RL, Pro-RL). Our lanes serve the older checkpoints.
|
|
|
+- **Action:** re-pin the MiMo lanes to the newest image + 10-09 checkpoints,
|
|
|
+ re-bench (quality may move — RL is new for Flash).
|
|
|
+
|
|
|
+## 2. New engines (test candidates — evaluate, don't adopt blindly)
|
|
|
+
|
|
|
+### kindlingai/gr8 (P1 interest — same org as our kindling stack)
|
|
|
+- From-scratch CUDA engine, **full GLM-5.3 (600B, 256 experts) on 3-6
|
|
|
+ Sparks**, own RoCE-v2 verbs fabric (no vLLM/PyTorch/NCCL), OpenAI API,
|
|
|
+ one-line `./gr8 glm53 up`. Pushed 2026-10-14; beta.
|
|
|
+- Model: `kindlingai/glm-5.3-exlr8-k2-k3-k4` (updated 10-14, 300K dl).
|
|
|
+- Numbers (their RigMark runs, clocks capped 2200): TP=4 decode 49.3/30.0
|
|
|
+ t/s code/prose, aggregate 64 @4 streams, prefill 8K cold 1,374 t/s, KV
|
|
|
+ pool 310K tok (415K with int4 KV). DFlash2+MTP drafting.
|
|
|
+- The license-locked full model becomes servable — flag: Z.ai license is
|
|
|
+ `license:other`; quants redistribute weights, commercial use unclear.
|
|
|
+- **Action:** install + 4x lane test + tool-eval quality gate. Compare with
|
|
|
+ GLM-5.3-Flash (AA 41.8) vs full GLM-5.3 (AA 44.8, max-thinking 50.3).
|
|
|
+
|
|
|
+### DGPP (HawkBearPig/dgpp) (P2)
|
|
|
+- C++/CUDA engine for Spark, 120★, pushed 10-14, 107-post forum thread
|
|
|
+ (383406). OpenAI API, 1/2/4-node templates.
|
|
|
+- Matrix: GLM-5.3-Flash unsloth FP8 w4; GLM-5.3-Flash NVFP4-FP8 hybrid w2/w4;
|
|
|
+ **full GLM-5.3 int4/int8 w4** (272K FP8 KV); **MiMo-V2.6-Flash-RL w2/w4**;
|
|
|
+ DS4.1-Flash w4; Qwen3.8-Flash-Next NVFP4 w1/w2 (mmap n-gram, decode
|
|
|
+ graphs, fp8 dense — +40% over published vLLM on Qwen single-node); GLM-4.7.
|
|
|
+- Numbers (5-class spread, warm decode): Flash-Next NVFP4 w2 62-75 t/s;
|
|
|
+ GLM-5.3-Flash NVFP4 w4 50-58 t/s; full GLM-5.3 w4 25-29 t/s (42-47 @C4).
|
|
|
+- **Action:** bench DGPP vs our vLLM lanes on (a) Qwen3.8-Flash-Next w1,
|
|
|
+ (b) MiMo-RL w2 — quality via tool-eval-bench, same harness both sides.
|
|
|
+
|
|
|
+### veloGB10 v0.7.0 (sf-stav) (P3 — watch)
|
|
|
+- Rust engine, EXL3-focused; Qwen3.8-Flash-Next EXL3 3.05bpw TP=2 @186 t/s
|
|
|
+ (their claim); NVFP4 supported but unoptimized; TP=4 pending; no vision.
|
|
|
+- Model: doth4580/Qwen3.8-Flash-Next-EXL3-3.05bpw. Re-test when TP4/NVFP4
|
|
|
+ mature.
|
|
|
+
|
|
|
+### dsgx10 (forum 385803) (P3 — watch)
|
|
|
+- From-scratch DS4.1-Flash engine for 4×GB10 switchless ring; +5-70% over
|
|
|
+ knapcio v2.2/v2.4 baselines, bit-exact, built with Claude. One post so
|
|
|
+ far (10-13); no repo link yet. Track the thread.
|
|
|
+
|
|
|
+## 3. Model landscape (AA / HF)
|
|
|
+
|
|
|
+| Model | AA index | License | Size | Note |
|
|
|
+|---|---|---|---|---|
|
|
|
+| MiMo-V2.6-Pro | 46.3 | MIT | ~1TB | overkill, one-shot (unchanged) |
|
|
|
+| **Muse Wonder Full (Meta)** | 56 | ? | ? | 10-12, likely closed/large — check |
|
|
|
+| **Kimi K3 (max)** | (top open) | other | 2.78 TB | too big |
|
|
|
+| **Grok 4.7** | 46 | closed | — | — |
|
|
|
+| **Qwen3.8 Max (0902)** | 45 | closed | — | — |
|
|
|
+| GLM-5.3 (full) | 44.8 | license:other | 600B | gr8/DGPP make it servable; license q. |
|
|
|
+| GLM-5.3-Flash | 41.8 | MIT | 320B | our flagship; zai repo touched 10-12 (minor) |
|
|
|
+| **Ling 3.1 Flash (InclusionAI)** | 41 | **MIT** | ~350-400B? (512/8 experts, 262K ctx) | released ~10-13; watch for day-0 quants/recipes |
|
|
|
+| MiMo-V2.6-Flash | 38 | MIT | 311B | re-cut 10-09; RL variant new |
|
|
|
+| DeepSeek V4.1 Flash | 39 (max) | MIT | — | our dsv41 lanes |
|
|
|
+| Qwen3.8-Flash-Next | 40 | Apache | 125B A6B | the coder model; see new lanes |
|
|
|
+
|
|
|
+### Qwen3.8-Flash-Next 2x TP2+EP lane (P2 — new lane)
|
|
|
+- Forum 385731 (MiaAI recipe, vLLM 0.30): 43-46 t/s single, **174 t/s
|
|
|
+ aggregate @8-16 conc**, 2.9K t/s prefill, 158/164 HumanEval, **74.2%
|
|
|
+ aider polyglot**, 262K ctx, MTP=3, fp8 KV. Soak: 2-h agent run zero errors.
|
|
|
+- Our flash-next recipes are single-node. **Action:** 2x lane with
|
|
|
+ `--tensor-parallel-size 2 --enable-expert-parallel`, MTP=3; bench with
|
|
|
+ tool-eval-bench + concurrency sweep.
|
|
|
+
|
|
|
+### Full GLM-5.3 on the 4-node lane (P2 — new capability)
|
|
|
+- Via gr8 (exlr8) or DGPP (int4/int8-mix). The presentation's "license-locked,
|
|
|
+ unusable" stance should be revisited for personal/research use; commercial
|
|
|
+ remains questionable. The 4x cluster is idle — this is the highest-ceiling
|
|
|
+ quality upgrade available without new hardware.
|
|
|
+
|
|
|
+## 4. Meta / infra notes
|
|
|
+
|
|
|
+- **MikroTik CRS812 200G thread (385776, 10-13):** 4× Spark via CRS812 with
|
|
|
+ 2×400G→2×200G breakout: NCCL AG 20.4 GB/s vs 25 theoretical (MTU 9000,
|
|
|
+ NCCL 2.28.9 RoCE, FEC on; logical-interface bonding attempted). We run the
|
|
|
+ same class of switch (SW2-MUC) — compare our own fabric-check AG numbers;
|
|
|
+ the thread's resolution may apply to us.
|
|
|
+- **tool-eval-bench moved again:** "with the most recent TEB, `high` reasoning
|
|
|
+ is no longer enough for 90+" (forum 381350 #827, 10-12); community reports
|
|
|
+ 90-93 on current TEB where older pins read lower. Our pin: v2.7.0. Any
|
|
|
+ cross-engine A/B must share a harness version; consider a re-baseline of
|
|
|
+ the anchor lanes before comparing against community numbers.
|
|
|
+- **RigMark** (github.com/alexellis/rigmark) — the benchmark gr8 uses
|
|
|
+ (reasoning_effort low, clock-capped, machine-readable kindling.json).
|
|
|
+ Worth a look as a cross-engine comparison harness.
|
|
|
+- eugr/spark-vllm-b12x nightlies continue (10-14); our qwen38-flash-next
|
|
|
+ lane pins the 09-25 digest because of the garbling bug — consider testing
|
|
|
+ a fresh pinned digest for the fix.
|
|
|
+- The 16-node operator thread (385107) wants GLM-5.3-full NVFP4 16x,
|
|
|
+ DS4.1+GLM-5.3-Flash 4+4, MiMo-Pro 16x on spark-vllm-docker — our shared
|
|
|
+ recipe repo in the wild; their TP-over-switch questions may produce
|
|
|
+ reusable data.
|
|
|
+
|
|
|
+## Recommended order
|
|
|
+
|
|
|
+1. **r40 cut watch** (flashinfer-0.7.x pairing) — wait for the lab's tag, then
|
|
|
+ cut the profile for kindling/flash lanes.
|
|
|
+2. **gr8 on the 4x lane** — full GLM-5.3; quality-gate with tool-eval-bench.
|
|
|
+3. **sparkring rebase + MiMo 10-09 re-cuts** — cheap, existing lanes.
|
|
|
+4. **Qwen3.8-Flash-Next 2x EP lane** — the coder-endpoint use case.
|
|
|
+5. **DGPP bench** (Qwen w1, MiMo-RL w2) — cross-engine comparison.
|
|
|
+6. Watch: veloGB10 TP4, dsgx10 repo, Ling-3.1-flash quants, TEB re-baseline.
|
