2026-09-29.md 1.4 KB
Morning shift 2026-09-29 (vLLM MiMo lane)
Launcher: stop→serve "occupied" race, fixed
Empirical repro: stop reported success 8 s before labeled containers were
actually gone on 3/4 nodes (graceful teardown window) — the back-to-back
serve then saw three nodes occupied and bailed. Fixed in stop.rs
(remote command now waits for labeled containers gone in ANY state,
explicitly rms stragglers, then verifies running-only) and in
orchestrate (up to 3 re-probes on the short-free verdict). End-to-end
verified: stop→serve back-to-back, zero occupied errors. 354 tests green.
JIT/compile cache persistence
TORCHINDUCTOR_CACHE_DIR/TRITON_CACHE_DIR env + jit-121f/triton mount
added to the vLLM MiMo lane (inductor was writing to ephemeral
/tmp/torchinductor_root, so the persistent vllm compile cache dangled at
lost cubins → TritonBundler warnings every boot). One-time purge of the stale
host-side torch_compile_cache (~100 MB/node) done. Warnings still printed
this boot (warm-memory reuse across the running container); next restart
should be clean.
State left for production
mimo26-flash-mopd-4x-vllm serving on all 4 nodes (sparkring serve
contract, benched 170/184 = 92/100 this morning; cache work since then is
JIT-only). API http://192.168.88.14:30000, model mimo-v2.6-flash.
Open item for later: c2 long-context throughput collapse (5.6 t/s @131K,
2.1 @256K) — noted below, not touched.