SparkRing MiMo lanes (MOPD) — adoption, diagnosis, variants

Owner lanes: recipes/mimo26-*-sparkring. Operator directive 2026-09-29: SparkRing is the fast line; MOPD only (RL retired — RL keeps training artifacts); measure both c1 and c2 at every depth.

1. What we adopted

Image: ghcr.io/fujitsupolycom/sparkring@sha256:c977a2d2… (publication dev-20260928-plainstatus-cuda1342-nccl2323-status033, image id 4b7049d1e00f…, base eugr/spark-vllm-b12x:nightly-20260924, CUDA 13.4.2, NCCL 2.32.3, runtime-status 0.3.3). Anonymous pull verified; 14.2 GiB compressed. Entrypoint is their toolchain.py serve wrapper — the harness keeps their launcher contract, our sm121 supplies model path, rendezvous flags, rank, env.

Known wins of this line over the eugr nightly: B12X selection-cache correction (restart 200–225 s), MiMo vision attention-sinks fix, host-to-device staging fix, tool-result contract (400/500), /v1/sparkring/status/view dashboard.

Adopted serve contract (their tp4 qualified config → our recipe)

spec num_speculative_tokens 5, kv_cache_dtype bfloat16, draft load safetensors, --max-num-seqs 16, --max-num-batched-tokens 8192, bounce loader {"read_mode":"bounce","io_threads":8} (--model-loader-extra-config), compilation-config fuse_act_quant + cudagraph_mode FULL_AND_PIECEWISE + capture sizes to 96, GMU 0.85. Env: VLLM_KV_CACHE_LAYOUT=BLHNC, VLLM_LM_HEAD_A16=1, VLLM_MXFP8_LM_HEAD=0, VLLM_USE_V2_MODEL_RUNNER=1, NCCL_IB_EXTENDED_IPV4_GIDS=1, RoCE allreduce/allgather caps, NCCL_ALGO=Ring, channels 4/4, NCCL_P2P_LEVEL=SYS, … (full recipes are the source of truth).

Documented deviations (uniform across the three variants):

their ring-only transport keys dropped on switched clusters (NCCL_SWITCHLESS_RING_ONLY, B12X_ROCE_OPPOSITE_PATHS/PEER_HCA_MAP) — their tp4 topology is direct-cycle-4, ours is the 200G switch fabric; HCA/GID/socket come from the cluster [nccl] block (f0 pair on switch, f1 pair on 2nodes-dac);
port 30000 (cluster default), launcher-generated rendezvous;
model read from the mounted HF cache at the pinned snapshot (sha verified against their profile: config 61bea4a0…, index e16ec5b7…);
their KV pins NOT adopted (auto pool; operator policy — GB10 UMA auto gives plenty, pins strand headroom);
max-model-len 524288 prod parity (their profile ships 262144; our A/B found no behavioral difference — see §3);
one conflict fix at transcription: their --no-enable-flashinfer-autotune and our nightly's --kernel-config {…enable_flashinfer_autotune:false} are mutually exclusive (engine ValueError) — their flag wins.

Image sync: sm121 sync-image --recipe R --cluster C — one docker save + parallel fan-out, digest-referenced images re-attached via pull-by-digest after load (save/load drops RepoDigest otherwise: docker run <ref> would fail with "No such image" on the peer). Measured 29.5 GiB → 3 peers: 206 s.

2. Lane inventory

recipe	cluster	nodes	state
`mimo26-flash-mopd-4x-sparkring`	dgx58-switch (bench) / any 4	TP4	benched: 165/184 (90/100) + honest depth curve (§3)
`mimo26-flash-mopd-2x-sparkring`	2nodes-dac (dgx1+dgx2)	TP2	recipe + dry-run OK; their qualified `direct-pair-2` intent, `B12X_ROCE_PAIR_PATHS=2` kept, f1/GID-3 verified on-node
`mimo26-pro-mopd-8x-sparkring`	8nodes-switch (all 8)	TP8	recipe + dry-run OK; no upstream Pro profile — our tp4 contract adapted (see below)

Facts for the Pro-MOPD 8x (verified 2026-09-29): snapshot XiaomiMiMo/MiMo-V2.6-Pro-MOPD @ adea8e2c5373 on the NAS (≈573.5 GB), arch MiMoV2ForCausalLM — same as Flash, and processor_config is byte-equal to Flash's, so their qualified Omni hf-overrides transfer verbatim; dflash/ draft present; quant = fp8 e4m3 + mxfp4_block_size → b12x MXFP4/MXFP8. Cache namespace mimo-v26-pro-mopd-* (compile caches are per-model). Serve checklist: prod must be down for 8nodes-switch; image currently only on dgx5-8 → --sync-image ships it to dgx1-4.

KV expectations (auto pools): Flash TP2 ≈ 10-15 GiB (weights ~87 GiB/rank), Pro TP8 ≈ 20-25 GiB (weights ~72 GiB/rank) — at ≈50K tok/GB that is 500-750K and 1M-1.2M tokens: fine for 256K×c2.

3. The "c2 long-context collapse" — closed as a measurement artifact

Symptom (2026-09-28/29, BOTH our nightly and the SparkRing image, identical numbers): depth suite tg t/s at c2 = 13.7 (@65K), 5.7 (@131K), 2.1 (@256K) while c1 stayed 37-47 everywhere.

Investigation trail (all findings kept because they are reusable):

Engine manual probes are healthy: c2 at real 62K/87K ctx = 40-53 t/s per stream; phase-pattern replica (Context Load → Inference) inference window 2-3 s; return_token_ids/ignore_eos payload toggles — no effect.
MML A/B (524288 vs their 262144): collapse bit-identical → not MML; KV usage during stall = 0.7% → not memory (operator called both).
Server-side tcpdump (server→client, split per chatcmpl id): one stream of each c2 pair trickled 19 s (2.5 tok/s) while its partner waited (Waiting: 1 reqs) then finished in 3 s; engine log windows showed prompt 6759 tok/s, generation 0.8 tok/s = a full ~67K monolith prefill in flight during the "decode" window.
Root cause: tool-eval-bench invokes llama-benchy with no_cache=True (→ --no-cache) and without --enable-prefix-caching (defaults store_true): the runner takes the Standard-Run branch — context+prompt as ONE ~67,595-token request, no warmup phase, full prefill every run. At c2 the two cold prefills serialize through the 8192-token chunked scheduler → exactly the admission block we measured; depth scaling of the artifact (2.4s→39s per pair) reproduces 13.7/5.7/2.1 to the digit. c1 was structurally immune (single stream interleaves its own prefill).
Proof: splitting their own code path yields 57.8-59.8 t/s at c2 @65K; the official CLI with one flag flips 13.7 → 52.1 t/s, TTFT 28.3s → 1.5s in a single run.

Fix shipped in sm121 bench (d5a2032): both the main perf phase and bench_depth pass --benchy-args=--enable-prefix-caching. Historical c2 depth numbers (both engines, both lanes) are void — re-baseline before quoting. Upstream issue (ready to file — llama-benchy/tool-eval-bench):

With defaults (--no-cache, no --enable-prefix-caching) every depth point measures two serialized cold ~67K prefills instead of decode: c2 @65K/131K/256K reads 13.7/5.7/2.1 t/s against a healthy engine that does 40-50 t/s per stream in the same shape. One flag fixes it (13.7 → 52.1). Consider defaulting enable_prefix_caching on whenever depth > 0.

Honest baseline (`mimo26-flash-mopd-4x-sparkring`, report

2026-09-29_160255_sparkring-honest-baseline)

Quality 165/184 (90/100); tg t/s warm-prefix:

depth	c1	c2
0	67.3	72.2
4096	62.1	78.9
8192	61.5	73.8
16384	55.2	69.9
65536	46.5	58.5
131072	37.4	47.0
262144	44.4	44.6

c2 ≥ c1 at every depth — concurrency is fine (matches SparkRing's own reports). Real effects worth further measurement: the flat-to-mild depth slope (67→45 over 256K), the c2 advantage collapsing by 256K, and the c1 128K dip (37.4 < 44.4 @256K) — one clean re-point needed to confirm.

4. Operating commands

# bench lane (dgx58-switch), image pre-warmed:
sm121 serve --recipe mimo26-flash-mopd-4x-sparkring --cluster dgx58-switch
sm121 bench --recipe mimo26-flash-mopd-4x-sparkring --cluster dgx58-switch \
  --label <name> --tokenizer /tmp/mimo-tok     # quality + perf c1/c2 + depth c1/c2

# 2-node DAC (needs dgx1+dgx2 free):
sm121 serve --recipe mimo26-flash-mopd-2x-sparkring --cluster 2nodes-dac

# Pro 8x (needs the whole fleet, image sync included):
sm121 serve --recipe mimo26-pro-mopd-8x-sparkring --cluster 8nodes-switch --sync-image

Diagnosis instruments used (reusable): raw TCP SSE observer (/tmp/wire-probe.py pattern — recv() timestamps separate server flush from client processing), node-side tcpdump -tttt -A -vv 'tcp port 30000 and src <node>' split per chatcmpl id, VLLM_LOGGING_LEVEL=DEBUG (drop after use), and driver scripts driving llama-benchy's own classes (PromptGenerator/LLMClient) to isolate library vs CLI behavior.

5. Open items

File the upstream bench issue; consider re-baselining the nightly lane with the fixed flag when 4nodes-switch is free (for a like-for-like sparkring-vs-nightly pair, if still wanted).
Pro 8x fleet run: operator coordination + first quality campaign.
c1@128K dip: single re-point, then decide if it is capture-size related.
Monitor cleanup: /tmp/qwen_monitor.py (Sep-26 leftover, 2-min qwen3.8 probes against :30000 → 404 spam after dgx5 changed workload) killed 2026-09-29; file kept for reference.

sparkring-mimo-lanes.md 8.8 KB 履歴 Raw