# Night shift — 2026-09-28

## MiMo long-context cliff: vLLM A/B lane (the main event)

Background: sglang 4x decode falls to 30% of d0 at 256K (77.6 → 23.5 t/s
slope sweep); GA layers' BF16 KV + kernel inefficiency. Operator asked for
the vLLM mirror (`mimo26-flash-mopd-4x-vllm`), modeled on lil-catalog.

### The four r39 walls (19:0x–19:1x)
Four boots, four distinct failures — all in the fork's MiMo-unqualified gap:
EAGLE3 hook vs the Omni wrapper (patched), b12x rejecting the draft's
non-causal attention (triton workaround), `ModelOptMxFp8LinearMethod`
hard-import (patched), then fatal: KV-layout validator refuses MiMo's mixed
page sizes (SWA-128 + GA) because b12x declares LBNHC only. Spec written as
PARKED: `specs/bugs/BUG-2026-09-28-vllm-mimo-dflash-parked.md`.

### Resolved by the ecosystem the same evening
- `local-inference-lab/vllm@dev/karmic-kraken` (eugr b12x nightly source)
  merged the MiMo series 09-22: aux-state loading, **block-outermost KV
  layouts for mixed page sizes**, **b12x non-causal**, eagle3 language-model
  unwrap, lazy mxfp8 guards. CI ships `eugr/spark-vllm-b12x:latest` arm64
  (v0.1.dev21546+g502d6cb5a) — pulled on dgx2, save/load-fabric'd to all
  nodes (~25 GB each).
- madeby561/vllm rc4 (b12x MiMo, FP8 KV, adaptive verification) is
  **linux/amd64 only** — not runnable on GB10; its compose kept as intel.
- Lane booted clean on the nightly: KV pool 8.57M tok, DFlash graphs on
  B12X, API serving. Our wrapper image + 2 patches retired (upstream
  equivalents shipped).

### Garble and its root cause (21:15–20:3x)
First serving = multilingual junk in the reasoning channel (spec on AND
spec off — same garbage). Bisection found `--load-format instanttensor`
mis-maps MiMo's split-GDN weight naming: the lab launcher always loads MiMo
via the **b12x loader** (`VLLM_PLUGINS=b12x_loader`), full Omni +
`--mm-encoder-tp-mode weights` (not `--language-model-only`), explicit bf16
KV. Switching the contract → clean English/Russian immediately. DFlash
re-enabled per the Pro-TP8 launcher config (adds `kv_cache_dtype:auto`).

### Quality (harness v2.7.1.dev7 — same as today's sglang anchor)
`2026-09-28_203145_vllm-kk-nightly-dflash`: **168/184 (91/100)** — exact
parity with sglang's 168/184 (91/100) from earlier today.

### Long-context slope (`~/.cache/sm121/bench/slope-28x-vllm`)
pp2048 tg128 c1, runs=2; sglang from `slope-28x`:

| depth | sglang | vLLM |
|---|---|---|
| d0 | 77.6 | 51.1 |
| 8192 | 76.2 | 54.0 |
| 32768 | 57.8 | 53.6 |
| 131072 | 33.7 | 33.7 |
| 262144 | **23.5** | **37.8** |

**The cliff is flattened** (256K/d0: 0.30 → 0.74) — exactly what the
split-KV/verify line promised — at the cost of −34% d0 decode. If d0 decode
is improved (acceptance, b12x decode tuning), vLLM outright wins at long
context. Next dials: spec acceptance stats, GMU raise (nightly boots with
plenty of KV headroom), and whether `--max-num-scheduled-tokens` deserves
the 2048 the lab uses for interactive lanes.

### Ops notes
- `sm121 serve` occupancy guard caught the crash-looped vLLM lane once;
  `sm121 stop --recipe` + rerun handled it.
- Bench/recipe churn today all pushed (see `git log` 2026-09-28).
- sglang 4x can be restored in ~2 min if the operator wants a fallback.

## SparkRing tp4 contract adoption (21:1x–23:3x)

FujitsuPolycom/sparkring ships an implemented `mimo-v26-flash-mopd-tp4`
profile with measured matrices (52/216/334 decode @1/8/16, 4,025 prefill).
Adopted the **serve contract** into our lane: dflash num_spec 5 (not 7) with
safetensors draft load, max-num-seqs 16 / batched-tokens 8192, bounce-mode
b12x loader, `--compilation-config` fuse_act_quant + capture sizes→96, env
keys V2_MODEL_RUNNER, A16 lm-head (+MXFP8 off), BLHNC, B12X_POLICY_MODE=auto,
RoCE allreduce sizes, NCCL_IB_EXTENDED_IPV4_GIDS. Deliberately NOT ported:
their switchless-ring transport (ring HCA maps, SWITCHLESS_RING_ONLY,
OPPOSITE_PATHS) — our TP4 is switched; TP2 would be DAC-ring-compatible.
Their 20 GiB KV pin dropped: GB10 auto-sizing keeps ~54 GiB pool (6.93M tok
after the larger cudagraph set), pinning would strand ~34 GiB.

Quality after adoption: `2026-09-28_232811_vllm-sparkring-contract`:
**170/184 (92/100)** — best MiMo score so far (sglang 168, vLLM-dflash7 168).

Slope c1/c2 (`slope-c12.log`): c1 50.3→28.7 t/s d0→256K (flat, cliff fixed).
**New finding — c2 long-context collapse**: 30.3 @32K, 5.6 @131K, 2.1 @256K.
Repro: c2 pp2048 tg128 at 131K+. Morning: hunt (b12x split-KV c2 path?
preemption? capture-size table for spec+2 streams?).

Shutdown for the night per operator; `sm121 power on --cluster
4nodes-switch` in the morning (mikrotik relays cut by power off, restored by
power on).