Morning shift 2026-09-29 (vLLM MiMo lane)

Launcher: stop→serve "occupied" race, fixed

Empirical repro: stop reported success 8 s before labeled containers were actually gone on 3/4 nodes (graceful teardown window) — the back-to-back serve then saw three nodes occupied and bailed. Fixed in stop.rs (remote command now waits for labeled containers gone in ANY state, explicitly rms stragglers, then verifies running-only) and in orchestrate (up to 3 re-probes on the short-free verdict). End-to-end verified: stop→serve back-to-back, zero occupied errors. 354 tests green.

JIT/compile cache persistence

TORCHINDUCTOR_CACHE_DIR/TRITON_CACHE_DIR env + jit-121f/triton mount added to the vLLM MiMo lane (inductor was writing to ephemeral /tmp/torchinductor_root, so the persistent vllm compile cache dangled at lost cubins → TritonBundler warnings every boot). One-time purge of the stale host-side torch_compile_cache (~100 MB/node) done. Warnings still printed this boot (warm-memory reuse across the running container); next restart should be clean.

State left for production

mimo26-flash-mopd-4x-vllm serving on all 4 nodes (sparkring serve contract, benched 170/184 = 92/100 this morning; cache work since then is JIT-only). API http://192.168.88.14:30000, model mimo-v2.6-flash. Open item for later: c2 long-context throughput collapse (5.6 t/s @131K, 2.1 @256K) — noted below, not touched.

Evening shift — SparkRing adoption, c2 case CLOSED, new variants

Operator: SparkRing is the fast line (nightly comparison dropped), MOPD only from now on (RL retired — training artifacts), and measure c1+c2 at depth.

SparkRing lane stood up and benched

mimo26-flash-mopd-4x-sparkring (installer image c977a2d2, their qualified tp4 contract) — image pushed to all 4 dgx58 nodes (sync rewrite exercised: digest-ref RepoDigest re-attach after save/load, parallel fan-out 206 s), cold boot 26 min, warm restart ~14 min. Campaign 2026-09-29_160255_sparkring-honest-baseline: 165/184 (90/100) plus the first honest c1/c2 depth curve: c2 ≥ c1 everywhere (72.2 @d0 → 44.6 @256K).

The c2 "collapse" (yesterday's open item) — measurement artifact, closed

The 13.7/5.7/2.1 t/s c2 depth rows were never the engine: tool-eval-bench default-invokes llama-benchy with --no-cache and NO --enable-prefix-caching, so the runner concatenates context+prompt into a single ~67K request and serializes two cold prefills through the chunked scheduler (wire capture: one stream trickles 19 s at 2.5 tok/s while its partner waits; engine windows read prompt 6759 t/s + generation 0.8 t/s). One flag flips c2@65K 13.7 → 52.1 t/s. sm121 bench now passes it for the main perf phase and the depth suite (d5a2032); all historical c2 depth numbers are void. Full story + upstream-issue draft: specs/tech-architecture/sparkring-mimo-lanes.md. Morning note "c2 long-context throughput collapse (5.6 @131K, 2.1 @256K)" is superseded by this section.

Also fixed along the way (kept for reuse): sync digest-ref probe/attach, kernel-config ↔ --no-enable-flashinfer-autotune mutual exclusion, MML A/B negative (524288 vs 262144 → bit-identical), VLLM_LOGGING_LEVEL debug session, and the /tmp/qwen_monitor.py zombie (Sep-26 control probes → 404 spam on dgx5) killed.

New variants prepared (dry-run green)

mimo26-flash-mopd-2x-sparkring → cluster 2nodes-dac: their qualified direct-pair-2 tp2 profile transcribed; f1 ports carry the DAC 100.x subnet and their IPv4 RoCE GID is index 3 (verified on dgx1); B12X_ROCE_PAIR_PATHS=2 kept, ring keys still dropped.
mimo26-pro-mopd-8x-sparkring → cluster 8nodes-switch: Pro-MOPD (573.5 GB, arch/processor verified byte-compatible with their qualified overrides, dflash/ draft present) on the tp8 contract — needs fleet coordination + image sync to dgx1-4. RL recipes stay as history only — operator: RL no longer used.

2026-09-29.md 3.9 KB Pysyvä linkki Historia Raaka