|
|
@@ -0,0 +1,110 @@
|
|
|
+# Mistral Large 4 1T A52B NVFP4 — DGX Spark bring-up (2026-10-29/30)
|
|
|
+
|
|
|
+`mistralai/Mistral-Large-4-1T-A52B-NVFP4` — Mistral's 1T MoE (52B active), DeepSeek-style
|
|
|
+sparse MLA + DSA top-k indexer, cross-layer KV sharing, NoPE layers, embedded DSpark
|
|
|
+drafter (k=3), multimodal (Pixtral tower), native 1M context. Weights: 610 GB (568 GiB),
|
|
|
+NVFP4 experts + FP8 linears (`modelopt MIXED_PRECISION`). **The NVFP4 repo has no
|
|
|
+config.json** (only the FP8 checkpoint has one) — Mistral-native "consolidated" format.
|
|
|
+
|
|
|
+Lanes: `mistral-large4-nvfp4-8x-vllm`, `mistral-large4-nvfp4-8x-sglang` (8× DGX Spark,
|
|
|
+TP8, 512K context, DSpark spec k=3 both). Reports: `recipes/<lane>/reports/2026-10-30_*`.
|
|
|
+
|
|
|
+## Images
|
|
|
+
|
|
|
+Both vendor images ship arm64 variants — no source builds needed (the dsv41-style
|
|
|
+sm_121a rebuilds were NOT required; kernels JIT where missing):
|
|
|
+
|
|
|
+- `docker-images/mistral-large4-vllm` — thin FROM over
|
|
|
+ `vllm/vllm-openai@sha256:b046d936…` (the `mistral-large-4` tag, arm64 digest);
|
|
|
+ `ENTRYPOINT ["vllm","serve"]`, vLLM `0.1.dev23730+g64053a10b`, torch 2.14.0+cu130.
|
|
|
+- `docker-images/mistral-large4-sglang` — thin FROM over
|
|
|
+ `lmsysorg/sglang@sha256:2f0cd825…` (`mistral-large-4` tag, arm64); NVIDIA wrapper
|
|
|
+ `/opt/nvidia/nvidia_entrypoint.sh` + `sglang serve` CLI; torch 2.14.1+cu130,
|
|
|
+ mistral_common 1.13.0. **One baked patch** (`patch_barrier_timeout.py`): the
|
|
|
+ post-load `monitored_barrier` timeout becomes env-tunable
|
|
|
+ (`SGLANG_UNBALANCED_LOAD_TIMEOUT_S`, default 480).
|
|
|
+
|
|
|
+Recipe entrypoints: the launcher runs `--entrypoint <tail[0]> img <tail[1..]>`, so the
|
|
|
+recipes carry explicit entrypoints (`vllm serve` / the NVIDIA wrapper + `sglang serve`).
|
|
|
+**Gotcha (latent launcher bug)**: a recipe WITHOUT `entrypoint` for a positional-model
|
|
|
+engine composes `--entrypoint <model-id>` — broken since the 2026-09-26 command.rs
|
|
|
+change (commit `ad7069e`). Every new recipe must lead the tail with its program.
|
|
|
+
|
|
|
+## Boot ledger (SGLang lane, 6 boots — the hard-won flags)
|
|
|
+
|
|
|
+| Boot | Result | Cause |
|
|
|
+|---|---|---|
|
|
|
+| 1 | died in ~1 s | **containerd snapshotter lazy-unpack race**: `docker load` returns before the rootfs is materialized; `docker run` 35 s later exec'd a half-unpacked image (`stat …: no such file`), `--rm` deleted the evidence, rendezvous hung 30 min. **Launcher fix 0.2.5**: sync probes exec-readiness per peer (`/bin/sh` container start, retried ≤2 min) before declaring OK. |
|
|
|
+| 2 | head died at ~21 min, logs lost | KV-budget refusal at GMU 0.85 (see below); `--rm` destroyed the logs — all retries use `--keep` now. |
|
|
|
+| 3 | weights loaded, crash at autotune | default flashinfer MLA route autotunes `trtllm fmha` — **sm100-only** ("Unsupported architecture" on GB10/sm_121). |
|
|
|
+| 4 | rank 0 loaded (3.5 h), barrier killed ranks 2/5/6/7 | `UNBALANCED_MODEL_LOADING_TIMEOUT_S = 480` stock; rank-0 crawl was self-inflicted (my mid-load `drop_caches` wiped the warm NFS page cache → cold re-read of 71 GB). Barrier now 7200 s via the baked patch. |
|
|
|
+| 5 | weights loaded (80 min), crash at backend init | `flashinfer_sparse_mla` is **GLM-DSA-only** — the validator refuses any other `model_arch`. |
|
|
|
+| 6 | weights + autotune OK, crash in a warmup forward | NVFP4 MoE auto-routed to the **TensorRT-LLM runner** on SM12x → `'FusedMoE' object has no attribute 'g1_scale_c'` (upstream #41764/#45520: the auto rule compares `--quantization`, unset for checkpoint-quantized models). |
|
|
|
+| 7 | **READY** | `--moe-runner-backend flashinfer_cutlass` (the supported NVFP4 MoE runner on SM120/121). |
|
|
|
+
|
|
|
+### The flags that matter (SGLang)
|
|
|
+
|
|
|
+```
|
|
|
+--tokenizer-mode mistral # NVFP4 has no config.json: AutoTokenizer→AutoConfig
|
|
|
+ # dies ("couldn't find them in the cached files");
|
|
|
+ # mistral-common (tekken.json) instead
|
|
|
+--attention-backend dsa # DSA backend (MistralLarge4AttentionMLA extends
|
|
|
+ # DeepseekV2AttentionMLA)
|
|
|
+--dsa-prefill-backend tilelang # tilelang JIT-compiles for sm_121; FP8 KV REQUIRES
|
|
|
+--dsa-decode-backend tilelang # both backends tilelang (layouts cannot be mixed)
|
|
|
+--moe-runner-backend flashinfer_cutlass # NVFP4 MoE on SM120/121
|
|
|
+--watchdog-timeout 3600 # boot takes ~80 min
|
|
|
+SGLANG_UNBALANCED_LOAD_TIMEOUT_S=7200 # post-load barrier
|
|
|
+```
|
|
|
+
|
|
|
+### vLLM lane flags
|
|
|
+
|
|
|
+`--config-format mistral --load-format mistral --kv-cache-dtype fp8_ds_mla --dtype
|
|
|
+bfloat16 --enable-expert-parallel --enable-prefix-caching --mm-processor-cache-gb 0.0
|
|
|
+--tool-call-parser mistral --reasoning-parser mistral --enable-auto-tool-choice
|
|
|
+--speculative-config '{"method":"dspark","num_speculative_tokens":3}'` + GMU 0.89.
|
|
|
+
|
|
|
+**GMU boot-1 lesson**: at 0.85 the KV budget after weights (73.8 GiB) + graph
|
|
|
+profiling (5.0) is 23.63 GiB — a single 512K request needs 25 GiB ("estimated maximum
|
|
|
+model length is 495296"). 0.89 = +4.9 GiB and still under the GB10 admission ceiling
|
|
|
+(108.78 GiB free at startup). Explicit cudagraph capture sizes (the dsv41 boot-10
|
|
|
+lesson) backfired here: 29 exact sizes ate +3.6 GiB of graph memory — auto sizes win.
|
|
|
+
|
|
|
+### GB10 UMA page-cache accounting (verified live)
|
|
|
+
|
|
|
+The UMA driver's `mem_get_info` free ≈ **MemFree, not MemAvailable** — page cache from
|
|
|
+the NFS weight reads (~30 GB) counts as unavailable (`#48140` behaviour, SGLang side):
|
|
|
+GPU-free fell to ~1.6 GiB mid-load with `MemAvailable` still 31 GB, and a mid-load
|
|
|
+`drop_caches` released ~20 GiB instantly (verified). BUT wiping the warm NFS cache
|
|
|
+forces a cold re-read of 71 GB — rank 0's load went 40 min → 3.5 h. **Do NOT
|
|
|
+drop_caches mid-load**; keep the NFS cache warm and let the load converge. The
|
|
|
+post-load barrier must be longer than the slowest rank's crawl.
|
|
|
+
|
|
|
+## Bench (tool-eval-bench v3.0.0, seed 42, hardmode, c1/c2, from pluto)
|
|
|
+
|
|
|
+Same model id (`mistral-large-4`), 512K, DSpark k=3 on both.
|
|
|
+
|
|
|
+| Metric | vLLM (`…8x-vllm`) | SGLang (`…8x-sglang`) |
|
|
|
+|---|---:|---:|
|
|
|
+| Quality | **91**/100 (174/192) | **91**/100 (176/194) |
|
|
|
+| Deployability | 70 | **71** |
|
|
|
+| Responsiveness (median turn) | 6.9 s | **6.5 s** |
|
|
|
+| pp2048 @d0 | **1,069** | 897 |
|
|
|
+| pp2048 @d0 c2 | 837 | **925** |
|
|
|
+| tg @d0 c1/c2 | 24.8 / 33.0 | **31.9 / 41.7** |
|
|
|
+| ctx pp4096 @d4096 c1/c2 | 1,204/20.9 · 1,068/29.8 | 1,066/25.0 · 1,001/36.1 |
|
|
|
+| pp2048 @d4096 c1/c2 | **1,310**/20.9 · 980/30.7 | 1,077/**31.6** · 919/**43.7** |
|
|
|
+| ctx pp8192 @d8192 | **1,286** | 1,117 |
|
|
|
+| pp2048 @d8192 c1/c2 | **1,268**/22.0 · 984/31.0 | 1,047/**32.4** · 804/**38.1** |
|
|
|
+
|
|
|
+- **SGLang decode is faster everywhere** (+25-45% tg: 31.9 vs 24.8 c1; 41.7 vs 33.0 c2
|
|
|
+ at d0; 43.7 vs 30.7 at d4096 c2) — the tilelang DSA decode + DSPARK stack.
|
|
|
+- **vLLM prefill is faster** (+19-22% pp) — its TRITON_MLA/FlashInfer prefill path.
|
|
|
+- Quality identical (91/100). vLLM's TC-45 probe (`tool_choice`) failed with
|
|
|
+ `finish_reason=length` and was excluded (174/192); SGLang's endpoint enforces
|
|
|
+ `tool_choice` (176/194) — a real capability difference.
|
|
|
+- Safety-critical: vLLM 2 (TC-81 injection, TC-92 tenant isolation); SGLang 3 (the
|
|
|
+ same two + TC-74 batched sends).
|
|
|
+
|
|
|
+Bottom line: for agentic decode-heavy traffic SGLang is the faster engine; for
|
|
|
+prefill-heavy long-context batch vLLM. Both are production-viable at 512K.
|
