# Morning shift 2026-09-29 (vLLM MiMo lane)

## Launcher: stop→serve "occupied" race, fixed

Empirical repro: `stop` reported success 8 s before labeled containers were
actually gone on 3/4 nodes (graceful teardown window) — the back-to-back
`serve` then saw three nodes `occupied` and bailed. Fixed in `stop.rs`
(remote command now waits for labeled containers gone in ANY state,
explicitly `rm`s stragglers, then verifies running-only) and in
`orchestrate` (up to 3 re-probes on the short-free verdict). End-to-end
verified: stop→serve back-to-back, zero occupied errors. 354 tests green.

## JIT/compile cache persistence

`TORCHINDUCTOR_CACHE_DIR`/`TRITON_CACHE_DIR` env + `jit-121f/triton` mount
added to the vLLM MiMo lane (inductor was writing to ephemeral
`/tmp/torchinductor_root`, so the persistent vllm compile cache dangled at
lost cubins → TritonBundler warnings every boot). One-time purge of the stale
host-side `torch_compile_cache` (~100 MB/node) done. Warnings still printed
this boot (warm-memory reuse across the running container); next restart
should be clean.

## State left for production

`mimo26-flash-mopd-4x-vllm` serving on all 4 nodes (sparkring serve
contract, benched 170/184 = 92/100 this morning; cache work since then is
JIT-only). API http://192.168.88.14:30000, model `mimo-v2.6-flash`.
Open item for later: c2 long-context throughput collapse (5.6 t/s @131K,
2.1 @256K) — noted below, not touched.


## Evening shift — SparkRing adoption, c2 case CLOSED, new variants

Operator: SparkRing is the fast line (nightly comparison dropped), MOPD only
from now on (RL retired — training artifacts), and measure c1+c2 at depth.

### SparkRing lane stood up and benched

`mimo26-flash-mopd-4x-sparkring` (installer image c977a2d2, their qualified
tp4 contract) — image pushed to all 4 dgx58 nodes (sync rewrite exercised:
digest-ref RepoDigest re-attach after save/load, parallel fan-out 206 s),
cold boot 26 min, warm restart ~14 min. Campaign
`2026-09-29_160255_sparkring-honest-baseline`: **165/184 (90/100)** plus the
first honest c1/c2 depth curve: c2 ≥ c1 everywhere (72.2 @d0 → 44.6 @256K).

### The c2 "collapse" (yesterday's open item) — measurement artifact, closed

The 13.7/5.7/2.1 t/s c2 depth rows were never the engine: `tool-eval-bench`
default-invokes llama-benchy with `--no-cache` and NO
`--enable-prefix-caching`, so the runner concatenates context+prompt into a
single ~67K request and serializes two cold prefills through the chunked
scheduler (wire capture: one stream trickles 19 s at 2.5 tok/s while its
partner waits; engine windows read prompt 6759 t/s + generation 0.8 t/s).
One flag flips c2@65K 13.7 → 52.1 t/s. `sm121 bench` now passes it for the
main perf phase and the depth suite (`d5a2032`); all historical c2 depth
numbers are void. Full story + upstream-issue draft:
`specs/tech-architecture/sparkring-mimo-lanes.md`.
Morning note "c2 long-context throughput collapse (5.6 @131K, 2.1 @256K)" is
superseded by this section.

Also fixed along the way (kept for reuse): sync digest-ref probe/attach,
`kernel-config` ↔ `--no-enable-flashinfer-autotune` mutual exclusion,
MML A/B negative (524288 vs 262144 → bit-identical), VLLM_LOGGING_LEVEL
debug session, and the `/tmp/qwen_monitor.py` zombie (Sep-26 control probes
→ 404 spam on dgx5) killed.

### New variants prepared (dry-run green)

- `mimo26-flash-mopd-2x-sparkring` → cluster `2nodes-dac`: their qualified
  `direct-pair-2` tp2 profile transcribed; f1 ports carry the DAC 100.x
  subnet and their IPv4 RoCE GID is index 3 (verified on dgx1);
  `B12X_ROCE_PAIR_PATHS=2` kept, ring keys still dropped.
- `mimo26-pro-mopd-8x-sparkring` → cluster `8nodes-switch`: Pro-MOPD
  (573.5 GB, arch/processor verified byte-compatible with their qualified
  overrides, dflash/ draft present) on the tp8 contract — needs fleet
  coordination + image sync to dgx1-4.
RL recipes stay as history only — operator: RL no longer used.