|
|
@@ -65,6 +65,42 @@ does NOT yet support** — you MUST run the `eugr/spark-vllm-b12x` container
|
|
|
change). Our pin `3ad63d6b…`, HEAD observed `d9311e72…` — bump pending a
|
|
|
supervised re-bench.
|
|
|
|
|
|
+## Muse Spark Open 816B (meta-models/Muse-Spark-Open-816B-A42B-NVFP4 + DFlash2)
|
|
|
+Downloaded to NAS (that download is what saturated `nas.lan` cold-read to ~1.3 MB/s
|
|
|
+for an hour on 10-16 — NOT our cluster). Both repos present:
|
|
|
+`models--meta-models--Muse-Spark-Open-816B-A42B-NVFP4` (497 GB, 127 shards,
|
|
|
+snapshot `4239e4db…`) + `models--nvidia--Muse-Spark-Open-816B-A42B-DFlash2`
|
|
|
+(8.2 GB draft). Also incidental `models--meta-models--Muse-Glimmer-30B`.
|
|
|
+
|
|
|
+**Facts (config + card):** 816B total / 42B active, **natively multimodal omni**
|
|
|
+(text+image+video+audio → text; `MuseSparkForConditionalGeneration`,
|
|
|
+`model_type: muse_spark`, `auto_map: None`). text_config: 62 layers, hidden 8192,
|
|
|
+8 experts/token, moe_intermediate 4096, **ctx 1,048,576 (1M)**, vocab 202048.
|
|
|
+NVFP4 `group_size 16`, exclude `layers.0.`, `layers.61.`, `self_attn`, `lm_head`.
|
|
|
+Card: **NVFP4 needs ≥700 GB VRAM** (BF16 2 TB); frameworks SGLang/vLLM/TRT-LLM/
|
|
|
+Unsloth/llama.cpp; `reasoning_effort` low/med/high/xhigh (default **xhigh**);
|
|
|
+Temp 1.0 / top_p 1.0 / top_k off. DFlash2 draft = `DFlash2DraftModel`, qwen3-based,
|
|
|
+5 layers.
|
|
|
+
|
|
|
+**Feasibility on GB10 — BLOCKED (same class as Step-5):**
|
|
|
+- `MuseSparkForConditionalGeneration` is **NOT in vLLM v0.31.0 nor main** — the
|
|
|
+ registry only has `MuseGlimmer*` (the 30B model); only `muse_glimmer.py` exists.
|
|
|
+- The single MuseSpark PR (**#62064, open**) targets **Ampere (sm80) + AutoRound/
|
|
|
+ GPTQ INT4 experts** — a different quant (not our NVFP4) and arch (not SM121).
|
|
|
+ Even merged it would not run this NVFP4 checkpoint on GB10.
|
|
|
+- 497 GB weights is tight even on 8 Sparks (8×119 = 952 GB; 497 weights + KV +
|
|
|
+ multimodal encoders + activations); vision/audio towers + any sparse-attention
|
|
|
+ kernels have no SM121 build (Step-5 DSA precedent).
|
|
|
+- Multimodal omni → our `tool-eval-bench` is text-only; quality gate partial.
|
|
|
+→ **Watch item, not a night-shift lane.** Re-check when a vLLM PR lands MuseSpark
|
|
|
+with NVFP4 + SM121 (and a DFlash2-for-MuseSpark target). Track #62064.
|
|
|
+
|
|
|
+**Runnable Muse-family alternative NOW:** `meta-models/Muse-Glimmer-30B` IS
|
|
|
+supported in vLLM v0.31.0 (`MuseGlimmerForCausalLM` / `MuseGlimmerForConditionalGeneration`
|
|
|
++ `DFlashMuseGlimmerAssistantModel` speculator via `qwen3_dflash`) and is already
|
|
|
+on NAS. If we want a Muse lane this shift, Glimmer-30B (single Spark) is the one
|
|
|
+that actually loads — not the 816B.
|
|
|
+
|
|
|
## Ling-3.1-flash fp8→fp4 (our own converter — in progress)
|
|
|
No official 3.1-flash fp4 exists (inclusionAI has fp4 only for 3.0). We wrote a
|
|
|
converter (fp8-block routed experts → MXFP4, "even" scale mode, byte-exact vs
|
