|
@@ -0,0 +1,65 @@
|
|
|
|
|
+# Night shift — 2026-09-28
|
|
|
|
|
+
|
|
|
|
|
+## MiMo long-context cliff: vLLM A/B lane (the main event)
|
|
|
|
|
+
|
|
|
|
|
+Background: sglang 4x decode falls to 30% of d0 at 256K (77.6 → 23.5 t/s
|
|
|
|
|
+slope sweep); GA layers' BF16 KV + kernel inefficiency. Operator asked for
|
|
|
|
|
+the vLLM mirror (`mimo26-flash-mopd-4x-vllm`), modeled on lil-catalog.
|
|
|
|
|
+
|
|
|
|
|
+### The four r39 walls (19:0x–19:1x)
|
|
|
|
|
+Four boots, four distinct failures — all in the fork's MiMo-unqualified gap:
|
|
|
|
|
+EAGLE3 hook vs the Omni wrapper (patched), b12x rejecting the draft's
|
|
|
|
|
+non-causal attention (triton workaround), `ModelOptMxFp8LinearMethod`
|
|
|
|
|
+hard-import (patched), then fatal: KV-layout validator refuses MiMo's mixed
|
|
|
|
|
+page sizes (SWA-128 + GA) because b12x declares LBNHC only. Spec written as
|
|
|
|
|
+PARKED: `specs/bugs/BUG-2026-09-28-vllm-mimo-dflash-parked.md`.
|
|
|
|
|
+
|
|
|
|
|
+### Resolved by the ecosystem the same evening
|
|
|
|
|
+- `local-inference-lab/vllm@dev/karmic-kraken` (eugr b12x nightly source)
|
|
|
|
|
+ merged the MiMo series 09-22: aux-state loading, **block-outermost KV
|
|
|
|
|
+ layouts for mixed page sizes**, **b12x non-causal**, eagle3 language-model
|
|
|
|
|
+ unwrap, lazy mxfp8 guards. CI ships `eugr/spark-vllm-b12x:latest` arm64
|
|
|
|
|
+ (v0.1.dev21546+g502d6cb5a) — pulled on dgx2, save/load-fabric'd to all
|
|
|
|
|
+ nodes (~25 GB each).
|
|
|
|
|
+- madeby561/vllm rc4 (b12x MiMo, FP8 KV, adaptive verification) is
|
|
|
|
|
+ **linux/amd64 only** — not runnable on GB10; its compose kept as intel.
|
|
|
|
|
+- Lane booted clean on the nightly: KV pool 8.57M tok, DFlash graphs on
|
|
|
|
|
+ B12X, API serving. Our wrapper image + 2 patches retired (upstream
|
|
|
|
|
+ equivalents shipped).
|
|
|
|
|
+
|
|
|
|
|
+### Garble and its root cause (21:15–20:3x)
|
|
|
|
|
+First serving = multilingual junk in the reasoning channel (spec on AND
|
|
|
|
|
+spec off — same garbage). Bisection found `--load-format instanttensor`
|
|
|
|
|
+mis-maps MiMo's split-GDN weight naming: the lab launcher always loads MiMo
|
|
|
|
|
+via the **b12x loader** (`VLLM_PLUGINS=b12x_loader`), full Omni +
|
|
|
|
|
+`--mm-encoder-tp-mode weights` (not `--language-model-only`), explicit bf16
|
|
|
|
|
+KV. Switching the contract → clean English/Russian immediately. DFlash
|
|
|
|
|
+re-enabled per the Pro-TP8 launcher config (adds `kv_cache_dtype:auto`).
|
|
|
|
|
+
|
|
|
|
|
+### Quality (harness v2.7.1.dev7 — same as today's sglang anchor)
|
|
|
|
|
+`2026-09-28_203145_vllm-kk-nightly-dflash`: **168/184 (91/100)** — exact
|
|
|
|
|
+parity with sglang's 168/184 (91/100) from earlier today.
|
|
|
|
|
+
|
|
|
|
|
+### Long-context slope (`~/.cache/sm121/bench/slope-28x-vllm`)
|
|
|
|
|
+pp2048 tg128 c1, runs=2; sglang from `slope-28x`:
|
|
|
|
|
+
|
|
|
|
|
+| depth | sglang | vLLM |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| d0 | 77.6 | 51.1 |
|
|
|
|
|
+| 8192 | 76.2 | 54.0 |
|
|
|
|
|
+| 32768 | 57.8 | 53.6 |
|
|
|
|
|
+| 131072 | 33.7 | 33.7 |
|
|
|
|
|
+| 262144 | **23.5** | **37.8** |
|
|
|
|
|
+
|
|
|
|
|
+**The cliff is flattened** (256K/d0: 0.30 → 0.74) — exactly what the
|
|
|
|
|
+split-KV/verify line promised — at the cost of −34% d0 decode. If d0 decode
|
|
|
|
|
+is improved (acceptance, b12x decode tuning), vLLM outright wins at long
|
|
|
|
|
+context. Next dials: spec acceptance stats, GMU raise (nightly boots with
|
|
|
|
|
+plenty of KV headroom), and whether `--max-num-scheduled-tokens` deserves
|
|
|
|
|
+the 2048 the lab uses for interactive lanes.
|
|
|
|
|
+
|
|
|
|
|
+### Ops notes
|
|
|
|
|
+- `sm121 serve` occupancy guard caught the crash-looped vLLM lane once;
|
|
|
|
|
+ `sm121 stop --recipe` + rerun handled it.
|
|
|
|
|
+- Bench/recipe churn today all pushed (see `git log` 2026-09-28).
|
|
|
|
|
+- sglang 4x can be restored in ~2 min if the operator wants a fallback.
|
