# SparkRing MiMo lanes (MOPD) — adoption, diagnosis, variants

Owner lanes: `recipes/mimo26-*-sparkring`. Operator directive 2026-09-29:
**SparkRing is the fast line**; MOPD only (RL retired — RL keeps training
artifacts); measure both c1 and c2 at every depth.

## 1. What we adopted

Image: `ghcr.io/fujitsupolycom/sparkring@sha256:c977a2d2…` (publication
`dev-20260928-plainstatus-cuda1342-nccl2323-status033`, image id
`4b7049d1e00f…`, base `eugr/spark-vllm-b12x:nightly-20260924`, CUDA 13.4.2,
NCCL 2.32.3, runtime-status 0.3.3). Anonymous pull verified; 14.2 GiB
compressed. Entrypoint is their `toolchain.py serve` wrapper — the harness
keeps their launcher contract, our `sm121` supplies model path, rendezvous
flags, rank, env.

Known wins of this line over the eugr nightly: B12X selection-cache
correction (restart 200–225 s), MiMo vision attention-sinks fix, host-to-device
staging fix, tool-result contract (400/500), `/v1/sparkring/status/view`
dashboard.

### Adopted serve contract (their tp4 qualified config → our recipe)

spec `num_speculative_tokens` 5, `kv_cache_dtype` bfloat16, draft load
safetensors, `--max-num-seqs 16`, `--max-num-batched-tokens 8192`, bounce
loader `{"read_mode":"bounce","io_threads":8}` (`--model-loader-extra-config`),
compilation-config fuse_act_quant + `cudagraph_mode FULL_AND_PIECEWISE` +
capture sizes to 96, GMU 0.85. Env: `VLLM_KV_CACHE_LAYOUT=BLHNC`,
`VLLM_LM_HEAD_A16=1`, `VLLM_MXFP8_LM_HEAD=0`, `VLLM_USE_V2_MODEL_RUNNER=1`,
`NCCL_IB_EXTENDED_IPV4_GIDS=1`, RoCE allreduce/allgather caps,
`NCCL_ALGO=Ring`, channels 4/4, `NCCL_P2P_LEVEL=SYS`, … (full recipes are
the source of truth).

Documented deviations (uniform across the three variants):
- their ring-only transport keys dropped on switched clusters
  (`NCCL_SWITCHLESS_RING_ONLY`, `B12X_ROCE_OPPOSITE_PATHS`/`PEER_HCA_MAP`)
  — their tp4 topology is `direct-cycle-4`, ours is the 200G switch fabric;
  HCA/GID/socket come from the cluster `[nccl]` block (f0 pair on switch,
  f1 pair on `2nodes-dac`);
- port 30000 (cluster default), launcher-generated rendezvous;
- model read from the mounted HF cache at the pinned snapshot (sha verified
  against their profile: config `61bea4a0…`, index `e16ec5b7…`);
- their KV pins NOT adopted (auto pool; operator policy — GB10 UMA auto
  gives plenty, pins strand headroom);
- `max-model-len 524288` prod parity (their profile ships 262144; our A/B
  found no behavioral difference — see §3);
- one conflict fix at transcription: their `--no-enable-flashinfer-autotune`
  and our nightly's `--kernel-config {…enable_flashinfer_autotune:false}`
  are mutually exclusive (engine ValueError) — their flag wins.

Image sync: `sm121 sync-image --recipe R --cluster C` — one `docker save` +
parallel fan-out, digest-referenced images re-attached via pull-by-digest
after load (save/load drops RepoDigest otherwise: `docker run <ref>` would
fail with "No such image" on the peer). Measured 29.5 GiB → 3 peers: 206 s.

## 2. Lane inventory

| recipe | cluster | nodes | state |
|---|---|---|---|
| `mimo26-flash-mopd-4x-sparkring` | dgx58-switch (bench) / any 4 | TP4 | **benched**: 165/184 (90/100) + honest depth curve (§3) |
| `mimo26-flash-mopd-2x-sparkring` | **2nodes-dac** (dgx1+dgx2) | TP2 | recipe + dry-run OK; their qualified `direct-pair-2` intent, `B12X_ROCE_PAIR_PATHS=2` kept, f1/GID-3 verified on-node |
| `mimo26-pro-mopd-8x-sparkring` | 8nodes-switch (all 8) | TP8 | recipe + dry-run OK; no upstream Pro profile — our tp4 contract adapted (see below) |

Facts for the Pro-MOPD 8x (verified 2026-09-29): snapshot
`XiaomiMiMo/MiMo-V2.6-Pro-MOPD @ adea8e2c5373` on the NAS (≈573.5 GB),
arch `MiMoV2ForCausalLM` — same as Flash, and `processor_config` is
**byte-equal** to Flash's, so their qualified Omni `hf-overrides` transfer
verbatim; `dflash/` draft present; quant = fp8 e4m3 + `mxfp4_block_size`
→ b12x MXFP4/MXFP8. Cache namespace `mimo-v26-pro-mopd-*` (compile caches
are per-model). Serve checklist: prod must be down for `8nodes-switch`;
image currently only on dgx5-8 → `--sync-image` ships it to dgx1-4.

KV expectations (auto pools): Flash TP2 ≈ 10-15 GiB (weights ~87 GiB/rank),
Pro TP8 ≈ 20-25 GiB (weights ~72 GiB/rank) — at ≈50K tok/GB that is
500-750K and 1M-1.2M tokens: fine for 256K×c2.

## 3. The "c2 long-context collapse" — closed as a measurement artifact

Symptom (2026-09-28/29, BOTH our nightly and the SparkRing image, identical
numbers): depth suite tg t/s at c2 = 13.7 (@65K), 5.7 (@131K), 2.1 (@256K)
while c1 stayed 37-47 everywhere.

Investigation trail (all findings kept because they are reusable):
1. Engine manual probes are healthy: c2 at real 62K/87K ctx = 40-53 t/s
   per stream; phase-pattern replica (Context Load → Inference) inference
   window 2-3 s; `return_token_ids`/`ignore_eos` payload toggles — no effect.
2. MML A/B (524288 vs their 262144): collapse bit-identical → not MML;
   KV usage during stall = 0.7% → not memory (operator called both).
3. Server-side tcpdump (`server→client`, split per chatcmpl id): one stream
   of each c2 pair trickled 19 s (2.5 tok/s) while its partner waited
   (`Waiting: 1 reqs`) then finished in 3 s; engine log windows showed
   `prompt 6759 tok/s, generation 0.8 tok/s` = a full ~67K monolith prefill
   in flight during the "decode" window.
4. Root cause: `tool-eval-bench` invokes llama-benchy with `no_cache=True`
   (→ `--no-cache`) and **without** `--enable-prefix-caching` (defaults
   `store_true`): the runner takes the Standard-Run branch — context+prompt
   as ONE ~67,595-token request, no warmup phase, full prefill every run.
   At c2 the two cold prefills serialize through the 8192-token chunked
   scheduler → exactly the admission block we measured; depth scaling of
   the artifact (2.4s→39s per pair) reproduces 13.7/5.7/2.1 to the digit.
   c1 was structurally immune (single stream interleaves its own prefill).
5. Proof: splitting their own code path yields 57.8-59.8 t/s at c2 @65K;
   the official CLI with one flag flips **13.7 → 52.1 t/s, TTFT 28.3s →
   1.5s** in a single run.

Fix shipped in `sm121 bench` (`d5a2032`): both the main perf phase and
`bench_depth` pass `--benchy-args=--enable-prefix-caching`. Historical
c2 depth numbers (both engines, both lanes) are void — re-baseline before
quoting. Upstream issue (ready to file — llama-benchy/tool-eval-bench):

> With defaults (`--no-cache`, no `--enable-prefix-caching`) every depth
> point measures two serialized cold ~67K prefills instead of decode: c2
> @65K/131K/256K reads 13.7/5.7/2.1 t/s against a healthy engine that does
> 40-50 t/s per stream in the same shape. One flag fixes it (13.7 → 52.1).
> Consider defaulting `enable_prefix_caching` on whenever `depth > 0`.

### Honest baseline (`mimo26-flash-mopd-4x-sparkring`, report
`2026-09-29_160255_sparkring-honest-baseline`)

Quality **165/184 (90/100)**; tg t/s warm-prefix:

| depth | c1 | c2 |
|---:|---:|---:|
| 0 | 67.3 | 72.2 |
| 4096 | 62.1 | 78.9 |
| 8192 | 61.5 | 73.8 |
| 16384 | 55.2 | 69.9 |
| 65536 | 46.5 | 58.5 |
| 131072 | 37.4 | 47.0 |
| 262144 | 44.4 | 44.6 |

c2 ≥ c1 at every depth — concurrency is fine (matches SparkRing's own
reports). Real effects worth further measurement: the flat-to-mild depth
slope (67→45 over 256K), the c2 advantage collapsing by 256K, and the
c1 128K dip (37.4 < 44.4 @256K) — one clean re-point needed to confirm.

## 4. Operating commands

```bash
# bench lane (dgx58-switch), image pre-warmed:
sm121 serve --recipe mimo26-flash-mopd-4x-sparkring --cluster dgx58-switch
sm121 bench --recipe mimo26-flash-mopd-4x-sparkring --cluster dgx58-switch \
  --label <name> --tokenizer /tmp/mimo-tok     # quality + perf c1/c2 + depth c1/c2

# 2-node DAC (needs dgx1+dgx2 free):
sm121 serve --recipe mimo26-flash-mopd-2x-sparkring --cluster 2nodes-dac

# Pro 8x (needs the whole fleet, image sync included):
sm121 serve --recipe mimo26-pro-mopd-8x-sparkring --cluster 8nodes-switch --sync-image
```

Diagnosis instruments used (reusable): raw TCP SSE observer
(`/tmp/wire-probe.py` pattern — recv() timestamps separate server flush
from client processing), node-side `tcpdump -tttt -A -vv 'tcp port 30000
and src <node>'` split per chatcmpl id, `VLLM_LOGGING_LEVEL=DEBUG` (drop
after use), and driver scripts driving llama-benchy's own classes
(`PromptGenerator`/`LLMClient`) to isolate library vs CLI behavior.

## 5. Open items

- File the upstream bench issue; consider re-baselining the nightly lane
  with the fixed flag when `4nodes-switch` is free (for a like-for-like
  sparkring-vs-nightly pair, if still wanted).
- Pro 8x fleet run: operator coordination + first quality campaign.
- c1@128K dip: single re-point, then decide if it is capture-size related.
- Monitor cleanup: `/tmp/qwen_monitor.py` (Sep-26 leftover, 2-min qwen3.8
  probes against :30000 → 404 spam after dgx5 changed workload) killed
  2026-09-29; file kept for reference.