 Alexander Korolev
|
e6204c6b95
kindling recipes: control paths are FILES, not the cache dirs — VLLM_{ADAPTIVE_K,DRAFT_TRUNC}_CONTROL pointed at the mounted dir roots, and their _read_control() does os.stat+open on the path (stat succeeds on a dir, open does not) => IsADirectoryError killed the engine core right after KV allocation; point them at <dir>/control (absent file = safe defaults: tau 0.3 / adapt, written file persists + live-control)
|
3 semanas atrás |
|
Alexander Korolev
|
571a5499f1
history: 2026-10-02 night — kindling 4x e2e (spec-config doubling, triple-quote form, upstream 27cf839 re-vendor, GB10 UMA gate coin flip fixed via GMU 0.86 + last-moment drop_caches mod), serve-boot-triage skill, headless dgx6/7/8, upstream sweep (MiMo/GLM/DFlash2 pins current, eugr nightly-20261001 new)
|
3 semanas atrás |
|
Alexander Korolev
|
2190f53141
kindling recipes: kill the GB10 UMA startup gate coin flip — GMU 0.88->0.86 (required 104.60 vs observed free 106.28-106.92 at worker init; the container stack refills ~11 GiB between the pre-container drop_caches and the check) + last-moment drop_caches mod right before the engine exec; the hard --kv-cache-memory pin is untouched so the KV pool is unchanged
|
3 semanas atrás |
|
Alexander Korolev
|
6cb6d346ed
skill: serve-boot-triage (log-preserving --keep launches, detached serve, per-node state matrix, hard rules incl. config IPs, GB10 failure patterns); revert the redundant recipe drop_caches hooks (launcher config default is already true)
|
3 semanas atrás |
|
Alexander Korolev
|
b23cc1ec1f
kindling recipes: drop_caches before_serve — the 0.88 GMU startup gate reads MemFree on GB10 UMA and random nodes carried ~15 GiB of weight/NFS page cache (106.53 free vs 107.03 required on rank1, the '2 of 4 nodes' boot failure); reclaim cache per node before containers start
|
3 semanas atrás |
|
Alexander Korolev
|
39c8ae1432
kindling: re-vendor to upstream 27cf839c2133 (spec-config moved to entrypoint/SPEC_EXTRA #64 — same bug we hit; megamoe routed-expert-0 fix #67; kindling.json results #68); README notes B-mode equivalence
|
3 semanas atrás |
|
Alexander Korolev
|
48af96bc5d
kindling recipes: close the --speculative-config triple-quote on the content line (mimo shape) — a trailing newline before the closing hit a different code path and the engine received apostrophe-wrapped JSON (argparse 'cannot be converted to loads'); form now byte-identical to the working mimo recipes, argv validated in-image against the real vLLM
|
3 semanas atrás |
|
Alexander Korolev
|
4c3f95d199
kindling recipes: fix doubled --speculative-config value (generator inserted the JSON twice; engine boot crashed with argparse 'cannot be converted' — caught by first e2e diag run)
|
3 semanas atrás |
|
Alexander Korolev
|
d222208590
fmt: cargo fmt the symlink-checksum test (missed before push)
|
3 semanas atrás |
|
Alexander Korolev
|
94b400a0be
build: compute_dir_checksum hashes symlinks by target text (EISDIR on dir-links killed sm121 build on the vendored kindling tree: vllm -> .submodules/spark-agent/vllm); upload tar dereferences separately
|
3 semanas atrás |
|
Alexander Korolev
|
e9a4ff28ef
kindling recipes: docker_image_dir=glm53-kindling (shared build context for all 3 TP variants)
|
3 semanas atrás |
|
Alexander Korolev
|
71163f27e6
kindlingai glm-5.3-flash-gx10: vendor their stack (mode B — image+patches under our rendezvous) + 3 transcribed recipes (TP4/TP2/TP6): entrypoint exec-args + all 9 experimental overlay envs 1:1, per-TP mimics of their case block (pins 26/8/26 GiB, MML 524288/163840/524288, batched 16384/8192/16384), DFlash2 spec + AdaptiveK + tp6 padded checkpoint; build.sh bakes the 39 overlay files into the image (no per-node mounts)
|
3 semanas atrás |
|
Alexander Korolev
|
6309233699
pro 8x campaign: MiMo-V2.6-Pro-MOPD TP8 (all 8, sparkring) — 167/184 (91/100), best quality lane; depth c2>=c1 up to 128K (34.8/28.6), 256K 34.6/33.5; KV 3.57M tokens; boot ~40 min (sync + first-time b12x autotune 394 candidates)
|
3 semanas atrás |
|
Alexander Korolev
|
739e9cc577
2x sparkring campaign (dgx58-switch, TP2): 164/184 (89/100); depth curve healthy — c2 >= c1 up to 128K (41.1/27.7), 256K 27.4/25.0; no collapse (warm-prefix flag)
|
3 semanas atrás |
|
Alexander Korolev
|
3ac2f09b57
served-model-name: simple lane names everywhere (mimo-v2.6-flash/pro, glm53) — drop TP/variant clutter copied from foreign sparkring recipes (operator: be simple)
|
3 semanas atrás |
|
Alexander Korolev
|
c9f74f0cb6
sparkring variants: 2x direct-pair (2nodes-dac, tp2 qualified shape + PAIR_PATHS, f1/GID3 verified) and 8x Pro-MOPD (TP8 contract, arch/processor/dflash verified); document lanes+bench-artifact case (specs/tech-architecture/sparkring-mimo-lanes.md); mark RL recipes retired; AGENTS: depth-suite c1+c2 + warm-prefix flag + sparkring block; shift log evening section
|
3 semanas atrás |
|
Alexander Korolev
|
2196a14c0f
sparkring lane: honest baseline campaign 165/184 (90/100) + full warm-prefix c1/c2 depth curve — c2 >= c1 at every depth (16K: 69.9, 64K: 58.5, 128K: 47.0, 256K: 44.6 t/s); the historic 13.7/5.6/2.1 collapse fully disappears
|
3 semanas atrás |
|
Alexander Korolev
|
d5a2032562
bench: warm-prefix depth measurement (--benchy-args=--enable-prefix-caching) — the c2 long-context 'collapse' (13.7/5.6/2.1) was llama-benchy's default cold-monolith branch, proven artifact (flag flips c2@65K 13.7 -> 52.1, TTFT 28.3s -> 1.5s); drop DEBUG from sparkring recipe (diagnosis closed)
|
3 semanas atrás |
|
Alexander Korolev
|
67d758a601
sparkring lane: temporary VLLM_LOGGING_LEVEL=DEBUG for c2-collapse diagnosis (prefill-miss trace)
|
3 semanas atrás |
|
Alexander Korolev
|
a46aa56797
sparkring lane: drop stale duplicated multimodal triad (kernel-config JSON leaked into --limit-mm-per-prompt after the earlier conflict fix)
|
3 semanas atrás |
|
Alexander Korolev
|
5d19f5ecc9
sparkring lane: MML A/B negative (collapse identical at 262144) — restore 524288 prod parity; kv usage 0.7% during stall excludes memory
|
3 semanas atrás |
|
Alexander Korolev
|
6044b3d669
sparkring lane: A/B max-model-len 524288 -> their qualified 262144 (c2 collapse axis; kv pin still off, pool auto per operator)
|
3 semanas atrás |
|
Alexander Korolev
|
10ca63d213
bench depth-suite: run c1 alongside c2 (operator: the suite was c2-only; c1 keeps flat while c2 collapses — both curves belong in the Throughput table)
|
3 semanas atrás |
|
Alexander Korolev
|
2467be1a9b
sparkring lane: drop kernel-config (mutually exclusive with their --no-enable-flashinfer-autotune); note engine's data->weights mm-encoder fallback
|
3 semanas atrás |
|
Alexander Korolev
|
c916344540
sync: re-attach RepoDigest after save/load via pull-by-digest (digest-only refs load untagged — docker run would fail on peers); inspect-miss = None not error
|
3 semanas atrás |
|
Alexander Korolev
|
3c1e2cd87c
sync: digest-ref probe tolerates absent image (inspect exit 1 = None, not ssh failure)
|
3 semanas atrás |
|
Alexander Korolev
|
cbf8c38243
sparkring lane: recipe transcribed from their published per-rank compose (installer image digest, toolchain serve, full env) with switch-only deviations noted
|
3 semanas atrás |
|
Alexander Korolev
|
6b16590b1f
sync: one docker save + parallel fabric fan-out (3x on a 4-node set), inner hop key = node-side ~/.ssh/<basename> (mac path broke dgx5-8), known_hosts lives until task end (10G fallback resurrected); sm121 sync-image command
|
3 semanas atrás |
|
Alexander Korolev
|
0ee032e4e6
bench: depth-suite phase (16K/64K/128K/256K @ c2, 2 repeats) appended to the campaign log; report/tests.rs split (perf_box_tests) for the line cap
|
3 semanas atrás |
|
Alexander Korolev
|
6b75b3427b
morning shift: stop/serve race fixed + JIT cache persistence notes; lane left live for prod
|
3 semanas atrás |
