2026-09-28.md 3.2 KB
Night shift — 2026-09-28
MiMo long-context cliff: vLLM A/B lane (the main event)
Background: sglang 4x decode falls to 30% of d0 at 256K (77.6 → 23.5 t/s
slope sweep); GA layers' BF16 KV + kernel inefficiency. Operator asked for
the vLLM mirror (mimo26-flash-mopd-4x-vllm), modeled on lil-catalog.
The four r39 walls (19:0x–19:1x)
Four boots, four distinct failures — all in the fork's MiMo-unqualified gap:
EAGLE3 hook vs the Omni wrapper (patched), b12x rejecting the draft's
non-causal attention (triton workaround), ModelOptMxFp8LinearMethod
hard-import (patched), then fatal: KV-layout validator refuses MiMo's mixed
page sizes (SWA-128 + GA) because b12x declares LBNHC only. Spec written as
PARKED: specs/bugs/BUG-2026-09-28-vllm-mimo-dflash-parked.md.
Resolved by the ecosystem the same evening
local-inference-lab/vllm@dev/karmic-kraken(eugr b12x nightly source) merged the MiMo series 09-22: aux-state loading, block-outermost KV layouts for mixed page sizes, b12x non-causal, eagle3 language-model unwrap, lazy mxfp8 guards. CI shipseugr/spark-vllm-b12x:latestarm64 (v0.1.dev21546+g502d6cb5a) — pulled on dgx2, save/load-fabric'd to all nodes (~25 GB each).- madeby561/vllm rc4 (b12x MiMo, FP8 KV, adaptive verification) is linux/amd64 only — not runnable on GB10; its compose kept as intel.
- Lane booted clean on the nightly: KV pool 8.57M tok, DFlash graphs on B12X, API serving. Our wrapper image + 2 patches retired (upstream equivalents shipped).
Garble and its root cause (21:15–20:3x)
First serving = multilingual junk in the reasoning channel (spec on AND
spec off — same garbage). Bisection found --load-format instanttensor
mis-maps MiMo's split-GDN weight naming: the lab launcher always loads MiMo
via the b12x loader (VLLM_PLUGINS=b12x_loader), full Omni +
--mm-encoder-tp-mode weights (not --language-model-only), explicit bf16
KV. Switching the contract → clean English/Russian immediately. DFlash
re-enabled per the Pro-TP8 launcher config (adds kv_cache_dtype:auto).
Quality (harness v2.7.1.dev7 — same as today's sglang anchor)
2026-09-28_203145_vllm-kk-nightly-dflash: 168/184 (91/100) — exact
parity with sglang's 168/184 (91/100) from earlier today.
Long-context slope (~/.cache/sm121/bench/slope-28x-vllm)
pp2048 tg128 c1, runs=2; sglang from slope-28x:
| depth | sglang | vLLM |
|---|---|---|
| d0 | 77.6 | 51.1 |
| 8192 | 76.2 | 54.0 |
| 32768 | 57.8 | 53.6 |
| 131072 | 33.7 | 33.7 |
| 262144 | 23.5 | 37.8 |
The cliff is flattened (256K/d0: 0.30 → 0.74) — exactly what the
split-KV/verify line promised — at the cost of −34% d0 decode. If d0 decode
is improved (acceptance, b12x decode tuning), vLLM outright wins at long
context. Next dials: spec acceptance stats, GMU raise (nightly boots with
plenty of KV headroom), and whether --max-num-scheduled-tokens deserves
the 2048 the lab uses for interactive lanes.
Ops notes
sm121 serveoccupancy guard caught the crash-looped vLLM lane once;sm121 stop --recipe+ rerun handled it.- Bench/recipe churn today all pushed (see
git log2026-09-28). - sglang 4x can be restored in ~2 min if the operator wants a fallback.