|
|
@@ -0,0 +1,29 @@
|
|
|
+# Morning shift 2026-09-29 (vLLM MiMo lane)
|
|
|
+
|
|
|
+## Launcher: stop→serve "occupied" race, fixed
|
|
|
+
|
|
|
+Empirical repro: `stop` reported success 8 s before labeled containers were
|
|
|
+actually gone on 3/4 nodes (graceful teardown window) — the back-to-back
|
|
|
+`serve` then saw three nodes `occupied` and bailed. Fixed in `stop.rs`
|
|
|
+(remote command now waits for labeled containers gone in ANY state,
|
|
|
+explicitly `rm`s stragglers, then verifies running-only) and in
|
|
|
+`orchestrate` (up to 3 re-probes on the short-free verdict). End-to-end
|
|
|
+verified: stop→serve back-to-back, zero occupied errors. 354 tests green.
|
|
|
+
|
|
|
+## JIT/compile cache persistence
|
|
|
+
|
|
|
+`TORCHINDUCTOR_CACHE_DIR`/`TRITON_CACHE_DIR` env + `jit-121f/triton` mount
|
|
|
+added to the vLLM MiMo lane (inductor was writing to ephemeral
|
|
|
+`/tmp/torchinductor_root`, so the persistent vllm compile cache dangled at
|
|
|
+lost cubins → TritonBundler warnings every boot). One-time purge of the stale
|
|
|
+host-side `torch_compile_cache` (~100 MB/node) done. Warnings still printed
|
|
|
+this boot (warm-memory reuse across the running container); next restart
|
|
|
+should be clean.
|
|
|
+
|
|
|
+## State left for production
|
|
|
+
|
|
|
+`mimo26-flash-mopd-4x-vllm` serving on all 4 nodes (sparkring serve
|
|
|
+contract, benched 170/184 = 92/100 this morning; cache work since then is
|
|
|
+JIT-only). API http://192.168.88.14:30000, model `mimo-v2.6-flash`.
|
|
|
+Open item for later: c2 long-context throughput collapse (5.6 t/s @131K,
|
|
|
+2.1 @256K) — noted below, not touched.
|
