# Mistral Large 4 1T A52B NVFP4 — DGX Spark bring-up (2026-10-29/30)

`mistralai/Mistral-Large-4-1T-A52B-NVFP4` — Mistral's 1T MoE (52B active), DeepSeek-style
sparse MLA + DSA top-k indexer, cross-layer KV sharing, NoPE layers, embedded DSpark
drafter (k=3), multimodal (Pixtral tower), native 1M context. Weights: 610 GB (568 GiB),
NVFP4 experts + FP8 linears (`modelopt MIXED_PRECISION`). **The NVFP4 repo has no
config.json** (only the FP8 checkpoint has one) — Mistral-native "consolidated" format.

Lanes: `mistral-large4-nvfp4-8x-vllm`, `mistral-large4-nvfp4-8x-sglang` (8× DGX Spark,
TP8, 512K context, DSpark spec k=3 both). Reports: `recipes/<lane>/reports/2026-10-30_*`.

## Images

Both vendor images ship arm64 variants — no source builds needed (the dsv41-style
sm_121a rebuilds were NOT required; kernels JIT where missing):

- `docker-images/mistral-large4-vllm` — thin FROM over
  `vllm/vllm-openai@sha256:b046d936…` (the `mistral-large-4` tag, arm64 digest);
  `ENTRYPOINT ["vllm","serve"]`, vLLM `0.1.dev23730+g64053a10b`, torch 2.14.0+cu130.
- `docker-images/mistral-large4-sglang` — thin FROM over
  `lmsysorg/sglang@sha256:2f0cd825…` (`mistral-large-4` tag, arm64); NVIDIA wrapper
  `/opt/nvidia/nvidia_entrypoint.sh` + `sglang serve` CLI; torch 2.14.1+cu130,
  mistral_common 1.13.0. **One baked patch** (`patch_barrier_timeout.py`): the
  post-load `monitored_barrier` timeout becomes env-tunable
  (`SGLANG_UNBALANCED_LOAD_TIMEOUT_S`, default 480).

Recipe entrypoints: the launcher runs `--entrypoint <tail[0]> img <tail[1..]>`, so the
recipes carry explicit entrypoints (`vllm serve` / the NVIDIA wrapper + `sglang serve`).
**Gotcha (latent launcher bug)**: a recipe WITHOUT `entrypoint` for a positional-model
engine composes `--entrypoint <model-id>` — broken since the 2026-09-26 command.rs
change (commit `ad7069e`). Every new recipe must lead the tail with its program.

## Boot ledger (SGLang lane, 6 boots — the hard-won flags)

| Boot | Result | Cause |
|---|---|---|
| 1 | died in ~1 s | **containerd snapshotter lazy-unpack race**: `docker load` returns before the rootfs is materialized; `docker run` 35 s later exec'd a half-unpacked image (`stat …: no such file`), `--rm` deleted the evidence, rendezvous hung 30 min. **Launcher fix 0.2.5**: sync probes exec-readiness per peer (`/bin/sh` container start, retried ≤2 min) before declaring OK. |
| 2 | head died at ~21 min, logs lost | KV-budget refusal at GMU 0.85 (see below); `--rm` destroyed the logs — all retries use `--keep` now. |
| 3 | weights loaded, crash at autotune | default flashinfer MLA route autotunes `trtllm fmha` — **sm100-only** ("Unsupported architecture" on GB10/sm_121). |
| 4 | rank 0 loaded (3.5 h), barrier killed ranks 2/5/6/7 | `UNBALANCED_MODEL_LOADING_TIMEOUT_S = 480` stock; rank-0 crawl was self-inflicted (my mid-load `drop_caches` wiped the warm NFS page cache → cold re-read of 71 GB). Barrier now 7200 s via the baked patch. |
| 5 | weights loaded (80 min), crash at backend init | `flashinfer_sparse_mla` is **GLM-DSA-only** — the validator refuses any other `model_arch`. |
| 6 | weights + autotune OK, crash in a warmup forward | NVFP4 MoE auto-routed to the **TensorRT-LLM runner** on SM12x → `'FusedMoE' object has no attribute 'g1_scale_c'` (upstream #41764/#45520: the auto rule compares `--quantization`, unset for checkpoint-quantized models). |
| 7 | **READY** | `--moe-runner-backend flashinfer_cutlass` (the supported NVFP4 MoE runner on SM120/121). |

### The flags that matter (SGLang)

```
--tokenizer-mode mistral          # NVFP4 has no config.json: AutoTokenizer→AutoConfig
                                  # dies ("couldn't find them in the cached files");
                                  # mistral-common (tekken.json) instead
--attention-backend dsa           # DSA backend (MistralLarge4AttentionMLA extends
                                  # DeepseekV2AttentionMLA)
--dsa-prefill-backend tilelang    # tilelang JIT-compiles for sm_121; FP8 KV REQUIRES
--dsa-decode-backend tilelang     # both backends tilelang (layouts cannot be mixed)
--moe-runner-backend flashinfer_cutlass   # NVFP4 MoE on SM120/121
--watchdog-timeout 3600           # boot takes ~80 min
SGLANG_UNBALANCED_LOAD_TIMEOUT_S=7200     # post-load barrier
```

### vLLM lane flags

`--config-format mistral --load-format mistral --kv-cache-dtype fp8_ds_mla --dtype
bfloat16 --enable-expert-parallel --enable-prefix-caching --mm-processor-cache-gb 0.0
--tool-call-parser mistral --reasoning-parser mistral --enable-auto-tool-choice
--speculative-config '{"method":"dspark","num_speculative_tokens":3}'` + GMU 0.89.

**GMU boot-1 lesson**: at 0.85 the KV budget after weights (73.8 GiB) + graph
profiling (5.0) is 23.63 GiB — a single 512K request needs 25 GiB ("estimated maximum
model length is 495296"). 0.89 = +4.9 GiB and still under the GB10 admission ceiling
(108.78 GiB free at startup). Explicit cudagraph capture sizes (the dsv41 boot-10
lesson) backfired here: 29 exact sizes ate +3.6 GiB of graph memory — auto sizes win.

### GB10 UMA page-cache accounting (verified live)

The UMA driver's `mem_get_info` free ≈ **MemFree, not MemAvailable** — page cache from
the NFS weight reads (~30 GB) counts as unavailable (`#48140` behaviour, SGLang side):
GPU-free fell to ~1.6 GiB mid-load with `MemAvailable` still 31 GB, and a mid-load
`drop_caches` released ~20 GiB instantly (verified). BUT wiping the warm NFS cache
forces a cold re-read of 71 GB — rank 0's load went 40 min → 3.5 h. **Do NOT
drop_caches mid-load**; keep the NFS cache warm and let the load converge. The
post-load barrier must be longer than the slowest rank's crawl.

## Bench (tool-eval-bench v3.0.0, seed 42, hardmode, c1/c2, from pluto)

Same model id (`mistral-large-4`), 512K, DSpark k=3 on both.

| Metric | vLLM (`…8x-vllm`) | SGLang (`…8x-sglang`) |
|---|---:|---:|
| Quality | **91**/100 (174/192) | **91**/100 (176/194) |
| Deployability | 70 | **71** |
| Responsiveness (median turn) | 6.9 s | **6.5 s** |
| pp2048 @d0 | **1,069** | 897 |
| pp2048 @d0 c2 | 837 | **925** |
| tg @d0 c1/c2 | 24.8 / 33.0 | **31.9 / 41.7** |
| ctx pp4096 @d4096 c1/c2 | 1,204/20.9 · 1,068/29.8 | 1,066/25.0 · 1,001/36.1 |
| pp2048 @d4096 c1/c2 | **1,310**/20.9 · 980/30.7 | 1,077/**31.6** · 919/**43.7** |
| ctx pp8192 @d8192 | **1,286** | 1,117 |
| pp2048 @d8192 c1/c2 | **1,268**/22.0 · 984/31.0 | 1,047/**32.4** · 804/**38.1** |

- **SGLang decode is faster everywhere** (+25-45% tg: 31.9 vs 24.8 c1; 41.7 vs 33.0 c2
  at d0; 43.7 vs 30.7 at d4096 c2) — the tilelang DSA decode + DSPARK stack.
- **vLLM prefill is faster** (+19-22% pp) — its TRITON_MLA/FlashInfer prefill path.
- Quality identical (91/100). vLLM's TC-45 probe (`tool_choice`) failed with
  `finish_reason=length` and was excluded (174/192); SGLang's endpoint enforces
  `tool_choice` (176/194) — a real capability difference.
- Safety-critical: vLLM 2 (TC-81 injection, TC-92 tenant isolation); SGLang 3 (the
  same two + TC-74 batched sends).

Bottom line: for agentic decode-heavy traffic SGLang is the faster engine; for
prefill-heavy long-context batch vLLM. Both are production-viable at 512K.