# SparkRing MiMo lanes (MOPD) — adoption, diagnosis, variants

Owner lanes: `recipes/mimo26-*-sparkring`. Operator directive 2026-09-29:
**SparkRing is the fast line**; MOPD only (RL retired — RL keeps training
artifacts); measure both c1 and c2 at every depth.

## 1. What we adopted

Image: `ghcr.io/fujitsupolycom/sparkring@sha256:c977a2d2…` (publication
`dev-20260928-plainstatus-cuda1342-nccl2323-status033`, image id
`4b7049d1e00f…`, base `eugr/spark-vllm-b12x:nightly-20260924`, CUDA 13.4.2,
NCCL 2.32.3, runtime-status 0.3.3). Anonymous pull verified; 14.2 GiB
compressed. Entrypoint is their `toolchain.py serve` wrapper — the harness
keeps their launcher contract, our `sm121` supplies model path, rendezvous
flags, rank, env.

Known wins of this line over the eugr nightly: B12X selection-cache
correction (restart 200–225 s), MiMo vision attention-sinks fix, host-to-device
staging fix, tool-result contract (400/500), `/v1/sparkring/status/view`
dashboard.

### Adopted serve contract (their tp4 qualified config → our recipe)

spec `num_speculative_tokens` 5, `kv_cache_dtype` bfloat16, draft load
safetensors, `--max-num-seqs 16`, `--max-num-batched-tokens 8192`, bounce
loader `{"read_mode":"bounce","io_threads":8}` (`--model-loader-extra-config`),
compilation-config fuse_act_quant + `cudagraph_mode FULL_AND_PIECEWISE` +
capture sizes to 96, GMU 0.85. Env: `VLLM_KV_CACHE_LAYOUT=BLHNC`,
`VLLM_LM_HEAD_A16=1`, `VLLM_MXFP8_LM_HEAD=0`, `VLLM_USE_V2_MODEL_RUNNER=1`,
`NCCL_IB_EXTENDED_IPV4_GIDS=1`, RoCE allreduce/allgather caps,
`NCCL_ALGO=Ring`, channels 4/4, `NCCL_P2P_LEVEL=SYS`, … (full recipes are
the source of truth).

Documented deviations (uniform across the three variants):
- their ring-only transport keys dropped on switched clusters
  (`NCCL_SWITCHLESS_RING_ONLY`, `B12X_ROCE_OPPOSITE_PATHS`/`PEER_HCA_MAP`)
  — their tp4 topology is `direct-cycle-4`, ours is the 200G switch fabric;
  HCA/GID/socket come from the cluster `[nccl]` block (f0 pair on switch,
  f1 pair on `2nodes-dac`);
- port 30000 (cluster default), launcher-generated rendezvous;
- model read from the mounted HF cache at the pinned snapshot (sha verified
  against their profile: config `61bea4a0…`, index `e16ec5b7…`);
- their KV pins NOT adopted (auto pool; operator policy — GB10 UMA auto
  gives plenty, pins strand headroom);
- `max-model-len 524288` prod parity (their profile ships 262144; our A/B
  found no behavioral difference — see §3);
- one conflict fix at transcription: their `--no-enable-flashinfer-autotune`
  and our nightly's `--kernel-config {…enable_flashinfer_autotune:false}`
  are mutually exclusive (engine ValueError) — their flag wins.

Image sync: `sm121 sync-image --recipe R --cluster C` — one `docker save` +
parallel fan-out, digest-referenced images re-attached via pull-by-digest
after load (save/load drops RepoDigest otherwise: `docker run <ref>` would
fail with "No such image" on the peer). Measured 29.5 GiB → 3 peers: 206 s.

## 2. Lane inventory

| recipe | cluster | nodes | state |
|---|---|---|---|
| `mimo26-flash-mopd-4x-sparkring` | dgx58-switch (bench) / any 4 | TP4 | **benched**: 165/184 (90/100) + honest depth curve (§3) |
| `mimo26-flash-mopd-2x-sparkring` | **2nodes-dac** (dgx1+dgx2) | TP2 | recipe + dry-run OK; their qualified `direct-pair-2` intent, `B12X_ROCE_PAIR_PATHS=2` kept, f1/GID-3 verified on-node |
| `mimo26-flash-mopd-2x-sparkring` | dgx58-switch (measured) / 2nodes-dac (intent) | TP2 | **benched**: 164/184 (89/100); c2 ≥ c1 up to 128K, 256K 27.4/25.0 |
| `mimo26-pro-mopd-8x-sparkring` | 8nodes-switch (all 8) | TP8 | **benched ×2**: 167/184 (91/100) on the Sep-29 image; **164/184 (89/100)** on the kraken image `b01442df` (2026-10-03, pluto/wired, PARTIAL — depth lost to the §4.1 client hang). First boot ~40 min incl. image sync + 394-candidate b12x autotune; KV pool 3.57M tokens (26.76 GiB auto) |

Facts for the Pro-MOPD 8x (verified 2026-09-29): snapshot
`XiaomiMiMo/MiMo-V2.6-Pro-MOPD @ adea8e2c5373` on the NAS (≈573.5 GB),
arch `MiMoV2ForCausalLM` — same as Flash, and `processor_config` is
**byte-equal** to Flash's, so their qualified Omni `hf-overrides` transfer
verbatim; `dflash/` draft present; quant = fp8 e4m3 + `mxfp4_block_size`
→ b12x MXFP4/MXFP8. Cache namespace `mimo-v26-pro-mopd-*` (compile caches
are per-model). Serve checklist: prod must be down for `8nodes-switch`;
image currently only on dgx5-8 → `--sync-image` ships it to dgx1-4.

KV expectations (auto pools): Flash TP2 ≈ 10-15 GiB (weights ~87 GiB/rank),
Pro TP8 ≈ 20-25 GiB (weights ~72 GiB/rank) — at ≈50K tok/GB that is
500-750K and 1M-1.2M tokens: fine for 256K×c2.

## 3. The "c2 long-context collapse" — closed as a measurement artifact

Symptom (2026-09-28/29, BOTH our nightly and the SparkRing image, identical
numbers): depth suite tg t/s at c2 = 13.7 (@65K), 5.7 (@131K), 2.1 (@256K)
while c1 stayed 37-47 everywhere.

Investigation trail (all findings kept because they are reusable):
1. Engine manual probes are healthy: c2 at real 62K/87K ctx = 40-53 t/s
   per stream; phase-pattern replica (Context Load → Inference) inference
   window 2-3 s; `return_token_ids`/`ignore_eos` payload toggles — no effect.
2. MML A/B (524288 vs their 262144): collapse bit-identical → not MML;
   KV usage during stall = 0.7% → not memory (operator called both).
3. Server-side tcpdump (`server→client`, split per chatcmpl id): one stream
   of each c2 pair trickled 19 s (2.5 tok/s) while its partner waited
   (`Waiting: 1 reqs`) then finished in 3 s; engine log windows showed
   `prompt 6759 tok/s, generation 0.8 tok/s` = a full ~67K monolith prefill
   in flight during the "decode" window.
4. Root cause: `tool-eval-bench` invokes llama-benchy with `no_cache=True`
   (→ `--no-cache`) and **without** `--enable-prefix-caching` (defaults
   `store_true`): the runner takes the Standard-Run branch — context+prompt
   as ONE ~67,595-token request, no warmup phase, full prefill every run.
   At c2 the two cold prefills serialize through the 8192-token chunked
   scheduler → exactly the admission block we measured; depth scaling of
   the artifact (2.4s→39s per pair) reproduces 13.7/5.7/2.1 to the digit.
   c1 was structurally immune (single stream interleaves its own prefill).
5. Proof: splitting their own code path yields 57.8-59.8 t/s at c2 @65K;
   the official CLI with one flag flips **13.7 → 52.1 t/s, TTFT 28.3s →
   1.5s** in a single run.

Fix shipped in `sm121 bench` (`d5a2032`): both the main perf phase and
`bench_depth` pass `--benchy-args=--enable-prefix-caching`. Historical
c2 depth numbers (both engines, both lanes) are void — re-baseline before
quoting. Upstream issue (ready to file — llama-benchy/tool-eval-bench):

> With defaults (`--no-cache`, no `--enable-prefix-caching`) every depth
> point measures two serialized cold ~67K prefills instead of decode: c2
> @65K/131K/256K reads 13.7/5.7/2.1 t/s against a healthy engine that does
> 40-50 t/s per stream in the same shape. One flag fixes it (13.7 → 52.1).
> Consider defaulting `enable_prefix_caching` on whenever `depth > 0`.

### Honest baseline (`mimo26-flash-mopd-4x-sparkring`, report
`2026-09-29_160255_sparkring-honest-baseline`)

Quality **165/184 (90/100)**; tg t/s warm-prefix:

| depth | c1 | c2 |
|---:|---:|---:|
| 0 | 67.3 | 72.2 |
| 4096 | 62.1 | 78.9 |
| 8192 | 61.5 | 73.8 |
| 16384 | 55.2 | 69.9 |
| 65536 | 46.5 | 58.5 |
| 131072 | 37.4 | 47.0 |
| 262144 | 44.4 | 44.6 |

c2 ≥ c1 at every depth — concurrency is fine (matches SparkRing's own
reports). Real effects worth further measurement: the flat-to-mild depth
slope (67→45 over 256K), the c2 advantage collapsing by 256K, and the
c1 128K dip (37.4 < 44.4 @256K) — one clean re-point needed to confirm.

## 4. 2026-10-02/03 pluto-wired campaign (kraken image, b01442df)

Bench client moved to **pluto.lan** (192.168.88.42, wired eth0) via
`sm121 bench --ssh-host pluto.lan` — a WiFi laptop client measurably
skews the perf table (kindling 4x A/B: pp2048@d0 3,667 wired vs ~2,700
WiFi, TTFT halved). TTB on pluto: `2.7.1.dev7+gbd35ba91b`.

Results (all pluto/wired, hardmode seed 42, perf c1/c2, depth 16K..256K c1/c2):

| lane | quality | notes |
|---|---|---|
| `mimo26-flash-mopd-4x-sparkring` (kraken) | **92/100 (169/184)** | full incl. depth; report `2026-10-02_*` (fef19db re-run); pp2048@d0 4,314 |
| `mimo26-flash-mopd-2x-sparkring` | **90/100 (166/184)** | report `2026-10-02_225515_mimo2x-pluto`; pp2048@d0 2,973, pp16K 2,302, pp65K 1,558, pp128K 1,023, pp256K 612 |
| `mimo26-pro-mopd-8x-sparkring` | **89/100 (164/184)** (Deploy 80, Quality 89, Resp 59, median turn 2.3s) | **PARTIAL**: main bench + shallow perf only — the depth suite client-hung (§4.1); report `2026-10-03_155239_mimopro8x-pluto3-partial.{md,json}`; pp2048@d0 2,386, d4K 2,270, d8K 2,086 |
| `mimo26-pro` 6x (TP6) | **not buildable** | model is 128 q-heads / 8 kv-heads (2⁷/2³): only TP∈{1,2,4,8} divide. vLLM's `divide()` rejects 6 at engine build. Kindling-style weight padding is NOT transferable to the main model: zero-padded KV heads enter the attention softmax (8→12 kv heads ≈ 1/3 of the denominator forced to e⁰) — fine for a verified drafter, direct quality loss for the served model. |

### 4.1 Depth-suite client hang (open, TTB-side)

Three consecutive depth runs against the **MiMo-Pro 8x** serve (2026-10-03
14:56 / 16:58 / 18:10 pluto) hung identically; kindling 6x (11:11) and
MiMo Flash 2x/4x depth runs the same days worked (16/16 in 13.5 min).

Forensics (run 3):
- Engine healthy: new 8-token chat → 0.3 s; nothing queued.
- GPU 0% on all 8 nodes; zero POSTs reaching the server for 30+ min.
- llama-benchy worker (11 threads): main thread idle in `do_epoll_wait`
  (empty asyncio loop), 10 OpenBLAS threads in futex; one ESTAB keep-alive
  to dgx1:30000 with 0 bytes queued both ways.
- No client output at all after the tool banner — not even
  "Measuring latency..." (the first `run_suite` print); stdout is pipe
  block-buffered, so absence of output ≠ early exit, but combined with the
  idle loop + zero requests the client is stuck **before the first depth
  request** (corpus/tokenizer load or session setup) with no CPU burn.
- `--no-warmup --skip-coherence --latency-mode generation` path; main bench
  (same TTB build, depths 0/4K/8K) worked 30 s earlier in the same run.

Suspect: client-side TTB/llama-benchy bug in the deep-depth invocation
(path `tool-eval-bench --perf-only --skip-tool-eval --depth 16384,...` via
`uvx llama-benchy`), not the cluster. Repro needed with the Flash 8x serve
(or kindling) to localize server-vs-harness; parked per operator.

## 5. Operating commands

```bash
# bench lane (dgx58-switch), image pre-warmed:
sm121 serve --recipe mimo26-flash-mopd-4x-sparkring --cluster dgx58-switch
sm121 bench --recipe mimo26-flash-mopd-4x-sparkring --cluster dgx58-switch \
  --label <name> --tokenizer /tmp/mimo-tok     # quality + perf c1/c2 + depth c1/c2

# 2-node DAC (needs dgx1+dgx2 free):
sm121 serve --recipe mimo26-flash-mopd-2x-sparkring --cluster 2nodes-dac

# Pro 8x (needs the whole fleet, image sync included):
sm121 serve --recipe mimo26-pro-mopd-8x-sparkring --cluster 8nodes-switch --sync-image
```

Diagnosis instruments used (reusable): raw TCP SSE observer
(`/tmp/wire-probe.py` pattern — recv() timestamps separate server flush
from client processing), node-side `tcpdump -tttt -A -vv 'tcp port 30000
and src <node>'` split per chatcmpl id, `VLLM_LOGGING_LEVEL=DEBUG` (drop
after use), and driver scripts driving llama-benchy's own classes
(`PromptGenerator`/`LLMClient`) to isolate library vs CLI behavior.

## 6. Open items

- **Depth-suite client hang** (§4.1): localize with a Flash-8x or kindling
  depth run from pluto; if it reproduces off-MiMo-Pro, file the upstream
  TTB issue with the forensics.
- Re-baseline the Pro 8x depth curve once the hang is resolved (the
  2026-09-29 c1/c2 depth numbers predate the kraken image).
- File the upstream bench issue; consider re-baselining the nightly lane
  with the fixed flag when `4nodes-switch` is free (for a like-for-like
  sparkring-vs-nightly pair, if still wanted).
- c1@128K dip: single re-point, then decide if it is capture-size related.
- Monitor cleanup: `/tmp/qwen_monitor.py` (Sep-26 leftover, 2-min qwen3.8
  probes against :30000 → 404 spam after dgx5 changed workload) killed
  2026-09-29; file kept for reference.