Night shift — 2026-09-28

MiMo long-context cliff: vLLM A/B lane (the main event)

Background: sglang 4x decode falls to 30% of d0 at 256K (77.6 → 23.5 t/s slope sweep); GA layers' BF16 KV + kernel inefficiency. Operator asked for the vLLM mirror (mimo26-flash-mopd-4x-vllm), modeled on lil-catalog.

The four r39 walls (19:0x–19:1x)

Four boots, four distinct failures — all in the fork's MiMo-unqualified gap: EAGLE3 hook vs the Omni wrapper (patched), b12x rejecting the draft's non-causal attention (triton workaround), ModelOptMxFp8LinearMethod hard-import (patched), then fatal: KV-layout validator refuses MiMo's mixed page sizes (SWA-128 + GA) because b12x declares LBNHC only. Spec written as PARKED: specs/bugs/BUG-2026-09-28-vllm-mimo-dflash-parked.md.

Resolved by the ecosystem the same evening

local-inference-lab/vllm@dev/karmic-kraken (eugr b12x nightly source) merged the MiMo series 09-22: aux-state loading, block-outermost KV layouts for mixed page sizes, b12x non-causal, eagle3 language-model unwrap, lazy mxfp8 guards. CI ships eugr/spark-vllm-b12x:latest arm64 (v0.1.dev21546+g502d6cb5a) — pulled on dgx2, save/load-fabric'd to all nodes (~25 GB each).
madeby561/vllm rc4 (b12x MiMo, FP8 KV, adaptive verification) is linux/amd64 only — not runnable on GB10; its compose kept as intel.
Lane booted clean on the nightly: KV pool 8.57M tok, DFlash graphs on B12X, API serving. Our wrapper image + 2 patches retired (upstream equivalents shipped).

Garble and its root cause (21:15–20:3x)

First serving = multilingual junk in the reasoning channel (spec on AND spec off — same garbage). Bisection found --load-format instanttensor mis-maps MiMo's split-GDN weight naming: the lab launcher always loads MiMo via the b12x loader (VLLM_PLUGINS=b12x_loader), full Omni + --mm-encoder-tp-mode weights (not --language-model-only), explicit bf16 KV. Switching the contract → clean English/Russian immediately. DFlash re-enabled per the Pro-TP8 launcher config (adds kv_cache_dtype:auto).

Quality (harness v2.7.1.dev7 — same as today's sglang anchor)

2026-09-28_203145_vllm-kk-nightly-dflash: 168/184 (91/100) — exact parity with sglang's 168/184 (91/100) from earlier today.

Long-context slope (`~/.cache/sm121/bench/slope-28x-vllm`)

pp2048 tg128 c1, runs=2; sglang from slope-28x:

depth	sglang	vLLM
d0	77.6	51.1
8192	76.2	54.0
32768	57.8	53.6
131072	33.7	33.7
262144	23.5	37.8

The cliff is flattened (256K/d0: 0.30 → 0.74) — exactly what the split-KV/verify line promised — at the cost of −34% d0 decode. If d0 decode is improved (acceptance, b12x decode tuning), vLLM outright wins at long context. Next dials: spec acceptance stats, GMU raise (nightly boots with plenty of KV headroom), and whether --max-num-scheduled-tokens deserves the 2048 the lab uses for interactive lanes.

Ops notes

sm121 serve occupancy guard caught the crash-looped vLLM lane once; sm121 stop --recipe + rerun handled it.
Bench/recipe churn today all pushed (see git log 2026-09-28).
sglang 4x can be restored in ~2 min if the operator wants a fallback.

SparkRing tp4 contract adoption (21:1x–23:3x)

FujitsuPolycom/sparkring ships an implemented mimo-v26-flash-mopd-tp4 profile with measured matrices (52/216/334 decode @1/8/16, 4,025 prefill). Adopted the serve contract into our lane: dflash num_spec 5 (not 7) with safetensors draft load, max-num-seqs 16 / batched-tokens 8192, bounce-mode b12x loader, --compilation-config fuse_act_quant + capture sizes→96, env keys V2_MODEL_RUNNER, A16 lm-head (+MXFP8 off), BLHNC, B12X_POLICY_MODE=auto, RoCE allreduce sizes, NCCL_IB_EXTENDED_IPV4_GIDS. Deliberately NOT ported: their switchless-ring transport (ring HCA maps, SWITCHLESS_RING_ONLY, OPPOSITE_PATHS) — our TP4 is switched; TP2 would be DAC-ring-compatible. Their 20 GiB KV pin dropped: GB10 auto-sizing keeps ~54 GiB pool (6.93M tok after the larger cudagraph set), pinning would strand ~34 GiB.

Quality after adoption: 2026-09-28_232811_vllm-sparkring-contract: 170/184 (92/100) — best MiMo score so far (sglang 168, vLLM-dflash7 168).

Slope c1/c2 (slope-c12.log): c1 50.3→28.7 t/s d0→256K (flat, cliff fixed). New finding — c2 long-context collapse: 30.3 @32K, 5.6 @131K, 2.1 @256K. Repro: c2 pp2048 tg128 at 131K+. Morning: hunt (b12x split-KV c2 path? preemption? capture-size table for spec+2 streams?).

Shutdown for the night per operator; sm121 power on --cluster 4nodes-switch in the morning (mikrotik relays cut by power off, restored by power on).

2026-09-28.md 4.6 KB Permalink Lịch sử Raw