# Ripple-65B NVFP4 on a single GB10 (dgx4), 2026-11-11 `ricursiveAI/Ripple-65B-Preview-NVFP4` runs on one DGX Spark. The vendor ships a hardware-locked SGLang path; vLLM support exists upstream but is not in any release yet. Numbers below are measured on dgx4, not vendor claims. ## The model | | | |---|---| | repo | `ricursiveAI/Ripple-65B-Preview-NVFP4`, public, Apache-2.0, created 04.11.2026 | | parameters | **39.6 B** dense (the "65B" name is marketing), 80 layers, hybrid attention 4 sliding : 1 full | | size on disk | **53.28 GiB** in 6 shards (byte-verified against HF's published sizes) | | quantisation | MLP **NVFP4** static group 16, attention/embed/lm_head **BF16**, BF16 KV cache | | draft | `ricursiveAI/Ripple-65B-Preview-DFlash2`, 4.04 B, 7.6 GB, unquantised | | licence note | HF tag says "8-bit"; that is wrong — read `quantization_config` | ## Why the vendor code is DGX-Spark-only `inference/local/spark.py` refuses any other device: ```python if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (12, 1): raise RuntimeError("Run this notebook on a DGX Spark / GB10 GPU") ``` It also patches SGLang internals for SM121: `enable_pdl=False` in the flashinfer softmax ("SM121 PDL softmax can produce NaNs"), KV scales forced to 1.0 for quantised exports with BF16 KV, and `--mem-fraction-static 0.78`. Code-writing helpers (`ripple_kernels.py`, `ripple_tree*.py`, `ripple_transport.py`) must sit next to `spark.py`. ## Reproduction (dgx4, Ubuntu 24.04, Python 3.12.3, 20 cores) ```bash python3 -m venv ~/venvs/ripple ~/venvs/ripple/bin/pip install --pre --extra-index-url https://download.pytorch.org/whl/cu130 \ "sglang==0.5.19" "torch==2.13.0+cu130" "transformers==5.12.1" \ "flashinfer-python==0.6.18" "ninja==1.13.2" "nest_asyncio==1.6.0" # checkpoints (skip the notebook, it is not needed to serve) ~/venvs/ripple/bin/python -c " from huggingface_hub import snapshot_download as s s('ricursiveAI/Ripple-65B-Preview-NVFP4', local_dir='/home/kilork/models/Ripple-65B-Preview-NVFP4', ignore_patterns=['*.mp4','*.ipynb']) s('ricursiveAI/Ripple-65B-Preview-DFlash2', local_dir='/home/kilork/models/Ripple-65B-Preview-DFlash2', ignore_patterns=['*.mp4','*.ipynb'])" export CUDA_HOME=/usr/local/cuda-13.0 ~/venvs/ripple/bin/python ~/models/Ripple-65B-Preview-NVFP4/inference/local/spark.py \ --target ~/models/Ripple-65B-Preview-NVFP4 \ --draft ~/models/Ripple-65B-Preview-DFlash2 \ --decoding tree --draft-tokens 32 \ --host 0.0.0.0 --port 8010 ``` First start compiles flashinfer's `fp4_gemm_cutlass` for sm121 into `~/.cache/sglang/.cache/flashinfer/0.6.18/121a/cached_ops/` with two `cicc` processes — about 10 minutes. Later starts load in ~35 s and capture CUDA graphs in ~2 min. Do not stop it with `pkill -f "spark.py"` from an ssh one-liner: the pattern matches the ssh shell and kills the session. Use `pkill -f "spark[.]py"`. ## Measured (dgx4, GB10, driver 580.178.04, 121 GiB unified) | decoding | code prompt (stream) | prose, 128 tok | 243-token prompt | TTFT | |---|---|---|---|---| | target only | 4.1 tok/s | 4.05 tok/s | 3.76 tok/s | 0.40 s | | chain, 16 draft tokens | 18.7 tok/s | 6.8 tok/s | 7.25 tok/s | 0.50 s | | **tree, 32 nodes (vendor default)** | **22.6 tok/s** | **10.4 tok/s** | **11.1 tok/s** | 0.50 s | 4 concurrent requests, tree decoding: **39.9 tok/s aggregate** (401 tokens in 10.05 s wall), 110 / 82 / 128 / 81 tokens per request. Memory: 103 GiB of 121 GiB with target + draft loaded (`--mem-fraction-static 0.78`); the target alone is 98 GiB. Load time 35 s from NVMe (1.5 GB/s). The spread between "code" and "prose" is acceptance rate, not a benchmark artefact: predictable text accepts 4–6 draft tokens per step, prose accepts far fewer. ## vLLM path (not taken, but viable) vLLM implements the architecture natively — `RicursiveForCausalLM` and `DFlash2RicursiveDraftModel` in `vllm/model_executor/models/registry.py`, with NVFP4, TP/PP, torch.compile, CUDA graphs, `--reasoning-parser ripple` / `--tool-call-parser ripple` and no `--trust-remote-code`. It merged **05.11.2026** (PR #65745), so it is not in `0.33.0` (02.11). Options when we want it: - nightly aarch64 wheel, e.g. `vllm-0.34.0rc1.dev210+gbafa5c6c3-cp38-abi3-manylinux_2_28_aarch64.whl` (229 MiB) from `https://wheels.vllm.ai/nightly/vllm/`; - or wait for `0.34.0`. `ModelOptNvFp4Config.get_min_capability()` is 75, so sm_121 is not gated out. The PR was validated on **B300** only, so NVFP4 kernel selection on GB10 is the open risk — flashinfer's sm121 cutlass FP4 path working under SGLang suggests it will land on a real kernel rather than a marlin fallback. ## Operational notes - dgx4 must have no prod4 ranks while this runs: it occupies 103 GiB of the 121 GiB unified memory. Stop it before the cluster comes back. - Weights live on dgx4's local NVMe (144 GB used of 1.8 T before, ~198 GB after). Movable to the NAS store later; cold start would pay 53 GiB over 1 GbE. - Context defaults to 131072 (`--context-length`), although the config claims 2 097 152 positions; SGLang corrected `max_total_tokens` down to its profiled 134 119 at load time.