 Alexander Korolev
|
7a8c147767
night shift 2026-10-20: FLARE 1017 tested and rejected (2x slower decode, crashes on agent prompts); Beam-1.0 NVFP4 lane added at 90/100 (fleet best); four-lane comparison recorded
|
2 hari lalu |
|
Alexander Korolev
|
5a4eec608c
spec: 1017 TensorFold lane is now an sm121 recipe; record the TF+vLLM blocker
|
4 hari lalu |
|
Alexander Korolev
|
198e6137dd
engine: tensorfold driver + tensorfold_1017_prepare built-in (NODE_RANK exported to hooks)
|
4 hari lalu |
|
Alexander Korolev
|
8f734c7926
spec: TensorFold 2x 1017 lane measured — 89/100 with half the nodes, decode 50.4 vs 39-43, but turn latency 6.7s vs 1.4s
|
4 hari lalu |
|
Alexander Korolev
|
b17dc421d2
spec: 1017 reasoning-effort A/B — 75 buys nothing over 100 on the new model (86 vs 87, same 1.4s turn); depth sweep closed
|
5 hari lalu |
|
Alexander Korolev
|
0deb95f896
spec: 1017 depth sweep (16K-256K) on the EP2/1M config — single-stream and suffix decode hold, concurrent deep prefills starve decode
|
5 hari lalu |
|
Alexander Korolev
|
000f022481
spec: dsv41-1017 tuning A/B (EP2 decode win, fp4-indexer negative, engram MAX_IDS vs chunked-prefill coupling, slow-NAS boot desync)
|
5 hari lalu |
|
Alexander Korolev
|
cebff9abb2
style: cargo fmt on src/power (occupancy-guard helpers) — no behavior change
|
5 hari lalu |
|
Alexander Korolev
|
d390fd2e87
audit: #15 resolved — the effort-mapping patch baked into the versioned dsv41-sglang-1017:v1 image (glm52 pattern), verified live in the running engine; TensorFold 1017 dual-Spark thread 386121 noted for the watch list
|
6 hari lalu |
|
Alexander Korolev
|
4a706d5739
audit: DFlash2 A/B blocked — eugr build rejects all draft attention backends under fp8 KV (4 backends tried); DFlash-for-GLM is a kindling-overlay capability; cross-option blocked by the lossless scale format
|
6 hari lalu |
|
Alexander Korolev
|
7ba7dde55b
audit: GLM 2x fully resolved through the launcher — missing distributed_backend (independent full-model loads = the 4 stalls), global-vs-local node-rank, cold GMU floor; lane now runlock-tracked and boots in ~4 min
|
6 hari lalu |
|
Alexander Korolev
|
d7a8ee224d
start: renumber node ranks within the launch set — a subset launch (dgx3+dgx4 = global ranks 2/3 from 4nodes-switch) sent --node-rank 2 with --nnodes 2 and vLLM rejects it ('must be between 0 and 1'); dist flags/head selection/headless-ness now use the serve-local index; container names/labels follow the launch set (_0/_1)
|
6 hari lalu |
|
Alexander Korolev
|
4029b286ae
fix start.rs bail! string quoting (nested double quotes broke the macro — 9770012 shipped broken); same guard
|
6 hari lalu |
|
Alexander Korolev
|
97700120af
serve: refuse multi-node recipes without distributed_backend — a recipe with required_nodes>1 but no distributed_backend launched N INDEPENDENT single-node engines (no --nnodes/--node-rank/--master-addr/--tensor-parallel-size), each trying to load the FULL checkpoint into one 121 GiB Spark: the actual root cause of the four glm53-flash-fp4-local-2x stalls (b12x routing OOM was the symptom). Guard bails loudly at serve time; the recipe fix (distributed_backend = "mp") restores the dist flags.
|
6 hari lalu |
|
Alexander Korolev
|
fd84b405e5
audit: GLM 2x lane RESOLVED via the card's manual flow — routing passed 42/42 (launcher died at 7/42), bench 92/100 ★★★★★; launcher-integration delta bisect is the remaining work item
|
6 hari lalu |
|
Alexander Korolev
|
2ce7181660
audit: GLM 2x stall #4 — checkpoint on node-local NVMe (mirror content-verified) still freezes the head during b12x shard routing; NFS-page-cache hypothesis dead, routing-phase UMA exhaustion intrinsic on 2x121GB; mirror completeness fix shipped (59291f3)
|
6 hari lalu |
|
Alexander Korolev
|
59291f31da
mirror: verify copied repo completeness by content (entry count + du -sb vs source), not just refs/main presence — a failed cp -a (GLM: 41/42 shards after the copy died) left a partial mirror that the old refs/main gate would trust and serve; now the gate re-verifies and re-copies the partial dir, and a post-copy mismatch fails loudly (MIRROR_VERIFY_FAILED) instead of silently serving a broken tree
|
6 hari lalu |
|
Alexander Korolev
|
8a683153dd
bench: add --node <name> — pin the bench target within a cluster (default: rank 0). A serve launched with 'serve --node' lives on a non-rank-0 node (solo Bolt on dgx3 inside 4nodes-switch), and benching the rank-0 head measured a different engine (the live kindling on dgx1 — model_not_available). pick_head tolerates the .lan suffix; web API passes node: None.
|
6 hari lalu |
|
Alexander Korolev
|
4bc842cd8a
audit: DeepSeek-V4.1-Flash-1017 vs Aiden SGLang 1.7.1 image — protocol encoder is jinja-free (tool-parser-keyed), DSML markers byte-identical; one real regression (REASONING_EFFORT_MAPPINGS shifted a tier: image 25/50/75/100 vs 1017 50/75/100); TC-89 verdict contradicts its own milestones (grader flake), TC-92 recurring DS4.1 disclosure gap; GLM 2x stall root-cause refined (card requires node-local checkpoint; mirror copy incomplete 41/42 + launcher partial-mirror check hole)
|
6 hari lalu |
|
Alexander Korolev
|
f0110cd14a
AGENTS.md: document occupancy-guarded shared extras in power off
|
6 hari lalu |
|
Alexander Korolev
|
c937008b42
power off: never cut a shared extra (fabric switch/fan) another cluster's live run needs
|
6 hari lalu |
|
Alexander Korolev
|
bde9a8c427
follow_with_liveness: don't mislabel a slow boot as 'engine down'
|
6 hari lalu |
|
Alexander Korolev
|
2fba3b05ec
AGENTS.md: document serve --cold + recipe cold_args
|
6 hari lalu |
|
Alexander Korolev
|
8f78bcdc45
serve --cold: recipe-declared cold_args overlay for first-launch safety
|
6 hari lalu |
|
Alexander Korolev
|
62bf7be418
power off --node: do NOT turn off cluster-level extras (fabric switch/fan)
|
6 hari lalu |
|
Alexander Korolev
|
0e826d4b35
bench: add --no-think and --timeout passthrough to tool-eval-bench — unblocks valid quality benches for thinking-by-default models (Ling, GLM-4.5-style) whose default reasoning blows the 120s/turn timeout and excludes most scenarios (the 10-16 fp8 Ling run scored 67/100 at 3.1% completion for exactly this reason)
|
6 hari lalu |
|
Alexander Korolev
|
8e3aff4749
upstream-watch 10-17: eugr/spark-vllm-b12x nightly (vLLM 0.1.dev21830, 10-15, 391 archs) checked — NO MuseSpark, NO Step5 (only MuseGlimmer + Step3 family). Confirms SGLang #44530 series is earliest path for Muse-816B. Bonus: b12x has DSpark speculators (Glm53DSpark, Qwen3OmniDSpark, K3DSpark, Gemma4DSpark)
|
6 hari lalu |
|
Alexander Korolev
|
183e6cfea5
upstream-watch 10-17: SGLang check — v0.5.21 has no MuseSpark/Step5 either; but MuseSpark landing via PR #44530 5-layer series (#44664 Docker + #44665 kernels merged, model layer in-flight) + merged cookbook; SGLang likely runs Muse-816B before vLLM. SGLang has bailing_moe_v3 (Ling) as an alt engine option
|
6 hari lalu |
|
Alexander Korolev
|
52fa899b5b
upstream-watch 10-17: Muse Spark Open 816B NVFP4 research — 497GB omni (text+img+video+audio), 1M ctx, NVFP4 g16; BLOCKED: MuseSpark arch not in vLLM v0.31.0/main (only MuseGlimmer 30B), PR #62064 is sm80+INT4 not NVFP4/SM121; ≥700GB VRAM; runnable Muse alt = Muse-Glimmer-30B (on NAS, vLLM-supported)
|
6 hari lalu |
|
Alexander Korolev
|
28ed284c75
Night-plan Task1: Ling fp4 38/54 done; NAS cold-read collapsed to 1.3MB/s (external, not our cluster) = blocker; converter fixes (copy_file_range NFS hang, 1-worker, NAS is the only bottleneck); resume command
|
6 hari lalu |
