SparkRing MiMo lanes (MOPD) — adoption, diagnosis, variants

Owner lanes: recipes/mimo26-*-sparkring. Operator directive 2026-09-29: SparkRing is the fast line; MOPD only (RL retired — RL keeps training artifacts); measure both c1 and c2 at every depth.

1. What we adopted

Current image (2026-10-09): ghcr.io/fujitsupolycom/sparkring@sha256:71d41057…5620ad (publication dev-20261004-kraken-cuda1342-nccl2323-status034, karmic-kraken base). All three mimo26-*-sparkring recipes point at it; the older digests below (c977a2d2, b01442df) are the historical campaign images.

Image (original adoption): ghcr.io/fujitsupolycom/sparkring@sha256:c977a2d2… (publication dev-20260928-plainstatus-cuda1342-nccl2323-status033, image id 4b7049d1e00f…, base eugr/spark-vllm-b12x:nightly-20260924, CUDA 13.4.2, NCCL 2.32.3, runtime-status 0.3.3). Anonymous pull verified; 14.2 GiB compressed. Entrypoint is their toolchain.py serve wrapper — the harness keeps their launcher contract, our sm121 supplies model path, rendezvous flags, rank, env.

Known wins of this line over the eugr nightly: B12X selection-cache correction (restart 200–225 s), MiMo vision attention-sinks fix, host-to-device staging fix, tool-result contract (400/500), /v1/sparkring/status/view dashboard.

Adopted serve contract (their tp4 qualified config → our recipe)

spec num_speculative_tokens 5, kv_cache_dtype bfloat16, draft load safetensors, --max-num-seqs 16, --max-num-batched-tokens 8192, bounce loader {"read_mode":"bounce","io_threads":8} (--model-loader-extra-config), compilation-config fuse_act_quant + cudagraph_mode FULL_AND_PIECEWISE + capture sizes to 96, GMU 0.85. Env: VLLM_KV_CACHE_LAYOUT=BLHNC, VLLM_LM_HEAD_A16=1, VLLM_MXFP8_LM_HEAD=0, VLLM_USE_V2_MODEL_RUNNER=1, NCCL_IB_EXTENDED_IPV4_GIDS=1, RoCE allreduce/allgather caps, NCCL_ALGO=Ring, channels 4/4, NCCL_P2P_LEVEL=SYS, … (full recipes are the source of truth).

Documented deviations (uniform across the three variants):

their ring-only transport keys dropped on switched clusters (NCCL_SWITCHLESS_RING_ONLY, B12X_ROCE_OPPOSITE_PATHS/PEER_HCA_MAP) — their tp4 topology is direct-cycle-4, ours is the 200G switch fabric; HCA/GID/socket come from the cluster [nccl] block (f0 pair on switch, f1 pair on 2nodes-dac);
port 30000 (cluster default), launcher-generated rendezvous;
model read from the mounted HF cache at the pinned snapshot (sha verified against their profile: config 61bea4a0…, index e16ec5b7…);
their KV pins NOT adopted (auto pool; operator policy — GB10 UMA auto gives plenty, pins strand headroom);
max-model-len 524288 prod parity (their profile ships 262144; our A/B found no behavioral difference — see §3);
one conflict fix at transcription: their --no-enable-flashinfer-autotune and our nightly's --kernel-config {…enable_flashinfer_autotune:false} are mutually exclusive (engine ValueError) — their flag wins.

Image sync: sm121 sync-image --recipe R --cluster C — one docker save + parallel fan-out, digest-referenced images re-attached via pull-by-digest after load (save/load drops RepoDigest otherwise: docker run <ref> would fail with "No such image" on the peer). Measured 29.5 GiB → 3 peers: 206 s.

2. Lane inventory

recipe	cluster	nodes	state
`mimo26-flash-mopd-4x-sparkring`	dgx58-switch (bench) / any 4	TP4	benched: 165/184 (90/100) + honest depth curve (§3)
`mimo26-flash-mopd-2x-sparkring`	2nodes-dac (dgx1+dgx2)	TP2	recipe + dry-run OK; their qualified `direct-pair-2` intent, `B12X_ROCE_PAIR_PATHS=2` kept, f1/GID-3 verified on-node
`mimo26-flash-mopd-2x-sparkring`	dgx58-switch (measured) / 2nodes-dac (intent)	TP2	benched: 164/184 (89/100); c2 ≥ c1 up to 128K, 256K 27.4/25.0
`mimo26-pro-mopd-8x-sparkring`	8nodes-switch (all 8)	TP8	benched ×2: 167/184 (91/100) on the Sep-29 image; 164/184 (89/100) on the kraken image `b01442df` (2026-10-03, pluto/wired, PARTIAL — depth lost to the §4.1 client hang). First boot ~40 min incl. image sync + 394-candidate b12x autotune; KV pool 3.57M tokens (26.76 GiB auto)

Facts for the Pro-MOPD 8x (verified 2026-09-29): snapshot XiaomiMiMo/MiMo-V2.6-Pro-MOPD @ adea8e2c5373 on the NAS (≈573.5 GB), arch MiMoV2ForCausalLM — same as Flash, and processor_config is byte-equal to Flash's, so their qualified Omni hf-overrides transfer verbatim; dflash/ draft present; quant = fp8 e4m3 + mxfp4_block_size → b12x MXFP4/MXFP8. Cache namespace mimo-v26-pro-mopd-* (compile caches are per-model). Serve checklist: prod must be down for 8nodes-switch; image currently only on dgx5-8 → --sync-image ships it to dgx1-4.

KV expectations (auto pools): Flash TP2 ≈ 10-15 GiB (weights ~87 GiB/rank), Pro TP8 ≈ 20-25 GiB (weights ~72 GiB/rank) — at ≈50K tok/GB that is 500-750K and 1M-1.2M tokens: fine for 256K×c2.

3. The "c2 long-context collapse" — closed as a measurement artifact

Symptom (2026-09-28/29, BOTH our nightly and the SparkRing image, identical numbers): depth suite tg t/s at c2 = 13.7 (@65K), 5.7 (@131K), 2.1 (@256K) while c1 stayed 37-47 everywhere.

Investigation trail (all findings kept because they are reusable):

Engine manual probes are healthy: c2 at real 62K/87K ctx = 40-53 t/s per stream; phase-pattern replica (Context Load → Inference) inference window 2-3 s; return_token_ids/ignore_eos payload toggles — no effect.
MML A/B (524288 vs their 262144): collapse bit-identical → not MML; KV usage during stall = 0.7% → not memory (operator called both).
Server-side tcpdump (server→client, split per chatcmpl id): one stream of each c2 pair trickled 19 s (2.5 tok/s) while its partner waited (Waiting: 1 reqs) then finished in 3 s; engine log windows showed prompt 6759 tok/s, generation 0.8 tok/s = a full ~67K monolith prefill in flight during the "decode" window.
Root cause: tool-eval-bench invokes llama-benchy with no_cache=True (→ --no-cache) and without --enable-prefix-caching (defaults store_true): the runner takes the Standard-Run branch — context+prompt as ONE ~67,595-token request, no warmup phase, full prefill every run. At c2 the two cold prefills serialize through the 8192-token chunked scheduler → exactly the admission block we measured; depth scaling of the artifact (2.4s→39s per pair) reproduces 13.7/5.7/2.1 to the digit. c1 was structurally immune (single stream interleaves its own prefill).
Proof: splitting their own code path yields 57.8-59.8 t/s at c2 @65K; the official CLI with one flag flips 13.7 → 52.1 t/s, TTFT 28.3s → 1.5s in a single run.

Fix shipped in sm121 bench (d5a2032): both the main perf phase and bench_depth pass --benchy-args=--enable-prefix-caching. Historical c2 depth numbers (both engines, both lanes) are void — re-baseline before quoting. Upstream issue (ready to file — llama-benchy/tool-eval-bench):

With defaults (--no-cache, no --enable-prefix-caching) every depth point measures two serialized cold ~67K prefills instead of decode: c2 @65K/131K/256K reads 13.7/5.7/2.1 t/s against a healthy engine that does 40-50 t/s per stream in the same shape. One flag fixes it (13.7 → 52.1). Consider defaulting enable_prefix_caching on whenever depth > 0.

Honest baseline (`mimo26-flash-mopd-4x-sparkring`, report

2026-09-29_160255_sparkring-honest-baseline)

Quality 165/184 (90/100); tg t/s warm-prefix:

depth	c1	c2
0	67.3	72.2
4096	62.1	78.9
8192	61.5	73.8
16384	55.2	69.9
65536	46.5	58.5
131072	37.4	47.0
262144	44.4	44.6

c2 ≥ c1 at every depth — concurrency is fine (matches SparkRing's own reports). Real effects worth further measurement: the flat-to-mild depth slope (67→45 over 256K), the c2 advantage collapsing by 256K, and the c1 128K dip (37.4 < 44.4 @256K) — one clean re-point needed to confirm.

4. 2026-10-02/03 pluto-wired campaign (kraken image, `b01442df`)

Bench client moved to pluto.lan (192.168.88.42, wired eth0) via sm121 bench --ssh-host pluto.lan — a WiFi laptop client measurably skews the perf table (kindling 4x A/B: pp2048@d0 3,667 wired vs ~2,700 WiFi, TTFT halved). TTB on pluto: 2.7.1.dev7+gbd35ba91b.

Results (all pluto/wired, hardmode seed 42, perf c1/c2, depth 16K..256K c1/c2):

lane	quality	notes
`mimo26-flash-mopd-4x-sparkring` (kraken)	92/100 (169/184)	full incl. depth; report `2026-10-02_*` (`fef19db` re-run); pp2048@d0 4,314
`mimo26-flash-mopd-2x-sparkring`	90/100 (166/184)	report `2026-10-02_225515_mimo2x-pluto`; pp2048@d0 2,973, pp16K 2,302, pp65K 1,558, pp128K 1,023, pp256K 612
`mimo26-pro-mopd-8x-sparkring`	89/100 (164/184) (Deploy 80, Quality 89, Resp 59, median turn 2.3s)	PARTIAL: main bench + shallow perf only — the depth suite client-hung (§4.1); report `2026-10-03_155239_mimopro8x-pluto3-partial.{md,json}`; pp2048@d0 2,386, d4K 2,270, d8K 2,086
`mimo26-pro` 6x (TP6)	not buildable	model is 128 q-heads / 8 kv-heads (2⁷/2³): only TP∈{1,2,4,8} divide. vLLM's `divide()` rejects 6 at engine build. Kindling-style weight padding is NOT transferable to the main model: zero-padded KV heads enter the attention softmax (8→12 kv heads ≈ 1/3 of the denominator forced to e⁰) — fine for a verified drafter, direct quality loss for the served model.

4.1 Depth-suite client hang (open, TTB-side)

Three consecutive depth runs against the MiMo-Pro 8x serve (2026-10-03 14:56 / 16:58 / 18:10 pluto) hung identically; kindling 6x (11:11) and MiMo Flash 2x/4x depth runs the same days worked (16/16 in 13.5 min).

Forensics (run 3):

Engine healthy: new 8-token chat → 0.3 s; nothing queued.
GPU 0% on all 8 nodes; zero POSTs reaching the server for 30+ min.
llama-benchy worker (11 threads): main thread idle in do_epoll_wait (empty asyncio loop), 10 OpenBLAS threads in futex; one ESTAB keep-alive to dgx1:30000 with 0 bytes queued both ways.
No client output at all after the tool banner — not even "Measuring latency..." (the first run_suite print); stdout is pipe block-buffered, so absence of output ≠ early exit, but combined with the idle loop + zero requests the client is stuck before the first depth request (corpus/tokenizer load or session setup) with no CPU burn.
--no-warmup --skip-coherence --latency-mode generation path; main bench (same TTB build, depths 0/4K/8K) worked 30 s earlier in the same run.

Suspect: client-side TTB/llama-benchy bug in the deep-depth invocation (path tool-eval-bench --perf-only --skip-tool-eval --depth 16384,... via uvx llama-benchy), not the cluster. Repro needed with the Flash 8x serve (or kindling) to localize server-vs-harness; parked per operator.

5. Operating commands

# bench lane (dgx58-switch), image pre-warmed:
sm121 serve --recipe mimo26-flash-mopd-4x-sparkring --cluster dgx58-switch
sm121 bench --recipe mimo26-flash-mopd-4x-sparkring --cluster dgx58-switch \
  --label <name> --tokenizer /tmp/mimo-tok     # quality + perf c1/c2 + depth c1/c2

# 2-node DAC (needs dgx1+dgx2 free):
sm121 serve --recipe mimo26-flash-mopd-2x-sparkring --cluster 2nodes-dac

# Pro 8x (needs the whole fleet, image sync included):
sm121 serve --recipe mimo26-pro-mopd-8x-sparkring --cluster 8nodes-switch --sync-image

Diagnosis instruments used (reusable): raw TCP SSE observer (/tmp/wire-probe.py pattern — recv() timestamps separate server flush from client processing), node-side tcpdump -tttt -A -vv 'tcp port 30000 and src <node>' split per chatcmpl id, VLLM_LOGGING_LEVEL=DEBUG (drop after use), and driver scripts driving llama-benchy's own classes (PromptGenerator/LLMClient) to isolate library vs CLI behavior.

6. Open items

Depth-suite client hang (§4.1): localize with a Flash-8x or kindling depth run from pluto; if it reproduces off-MiMo-Pro, file the upstream TTB issue with the forensics.
Re-baseline the Pro 8x depth curve once the hang is resolved (the 2026-09-29 c1/c2 depth numbers predate the kraken image).
File the upstream bench issue; consider re-baselining the nightly lane with the fixed flag when 4nodes-switch is free (for a like-for-like sparkring-vs-nightly pair, if still wanted).
c1@128K dip: single re-point, then decide if it is capture-size related.
Monitor cleanup: /tmp/qwen_monitor.py (Sep-26 leftover, 2-min qwen3.8 probes against :30000 → 404 spam after dgx5 changed workload) killed 2026-09-29; file kept for reference.

sparkring-mimo-lanes.md 13 KB Lịch sử Raw