ripple-65b-gb10.md 5.0 KB

Ripple-65B NVFP4 on a single GB10 (dgx4), 2026-11-11

ricursiveAI/Ripple-65B-Preview-NVFP4 runs on one DGX Spark. The vendor ships a hardware-locked SGLang path; vLLM support exists upstream but is not in any release yet. Numbers below are measured on dgx4, not vendor claims.

The model

repo ricursiveAI/Ripple-65B-Preview-NVFP4, public, Apache-2.0, created 04.11.2026
parameters 39.6 B dense (the "65B" name is marketing), 80 layers, hybrid attention 4 sliding : 1 full
size on disk 53.28 GiB in 6 shards (byte-verified against HF's published sizes)
quantisation MLP NVFP4 static group 16, attention/embed/lm_head BF16, BF16 KV cache
draft ricursiveAI/Ripple-65B-Preview-DFlash2, 4.04 B, 7.6 GB, unquantised
licence note HF tag says "8-bit"; that is wrong — read quantization_config

Why the vendor code is DGX-Spark-only

inference/local/spark.py refuses any other device:

if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (12, 1):
    raise RuntimeError("Run this notebook on a DGX Spark / GB10 GPU")

It also patches SGLang internals for SM121: enable_pdl=False in the flashinfer softmax ("SM121 PDL softmax can produce NaNs"), KV scales forced to 1.0 for quantised exports with BF16 KV, and --mem-fraction-static 0.78. Code-writing helpers (ripple_kernels.py, ripple_tree*.py, ripple_transport.py) must sit next to spark.py.

Reproduction (dgx4, Ubuntu 24.04, Python 3.12.3, 20 cores)

python3 -m venv ~/venvs/ripple
~/venvs/ripple/bin/pip install --pre --extra-index-url https://download.pytorch.org/whl/cu130 \
  "sglang==0.5.19" "torch==2.13.0+cu130" "transformers==5.12.1" \
  "flashinfer-python==0.6.18" "ninja==1.13.2" "nest_asyncio==1.6.0"

# checkpoints (skip the notebook, it is not needed to serve)
~/venvs/ripple/bin/python -c "
from huggingface_hub import snapshot_download as s
s('ricursiveAI/Ripple-65B-Preview-NVFP4', local_dir='/home/kilork/models/Ripple-65B-Preview-NVFP4', ignore_patterns=['*.mp4','*.ipynb'])
s('ricursiveAI/Ripple-65B-Preview-DFlash2', local_dir='/home/kilork/models/Ripple-65B-Preview-DFlash2', ignore_patterns=['*.mp4','*.ipynb'])"

export CUDA_HOME=/usr/local/cuda-13.0
~/venvs/ripple/bin/python ~/models/Ripple-65B-Preview-NVFP4/inference/local/spark.py \
  --target ~/models/Ripple-65B-Preview-NVFP4 \
  --draft  ~/models/Ripple-65B-Preview-DFlash2 \
  --decoding tree --draft-tokens 32 \
  --host 0.0.0.0 --port 8010

First start compiles flashinfer's fp4_gemm_cutlass for sm121 into ~/.cache/sglang/.cache/flashinfer/0.6.18/121a/cached_ops/ with two cicc processes — about 10 minutes. Later starts load in ~35 s and capture CUDA graphs in ~2 min.

Do not stop it with pkill -f "spark.py" from an ssh one-liner: the pattern matches the ssh shell and kills the session. Use pkill -f "spark[.]py".

Measured (dgx4, GB10, driver 580.178.04, 121 GiB unified)

decoding code prompt (stream) prose, 128 tok 243-token prompt TTFT
target only 4.1 tok/s 4.05 tok/s 3.76 tok/s 0.40 s
chain, 16 draft tokens 18.7 tok/s 6.8 tok/s 7.25 tok/s 0.50 s
tree, 32 nodes (vendor default) 22.6 tok/s 10.4 tok/s 11.1 tok/s 0.50 s

4 concurrent requests, tree decoding: 39.9 tok/s aggregate (401 tokens in 10.05 s wall), 110 / 82 / 128 / 81 tokens per request.

Memory: 103 GiB of 121 GiB with target + draft loaded (--mem-fraction-static 0.78); the target alone is 98 GiB. Load time 35 s from NVMe (1.5 GB/s).

The spread between "code" and "prose" is acceptance rate, not a benchmark artefact: predictable text accepts 4–6 draft tokens per step, prose accepts far fewer.

vLLM path (not taken, but viable)

vLLM implements the architecture natively — RicursiveForCausalLM and DFlash2RicursiveDraftModel in vllm/model_executor/models/registry.py, with NVFP4, TP/PP, torch.compile, CUDA graphs, --reasoning-parser ripple / --tool-call-parser ripple and no --trust-remote-code. It merged 05.11.2026 (PR #65745), so it is not in 0.33.0 (02.11). Options when we want it:

  • nightly aarch64 wheel, e.g. vllm-0.34.0rc1.dev210+gbafa5c6c3-cp38-abi3-manylinux_2_28_aarch64.whl (229 MiB) from https://wheels.vllm.ai/nightly/vllm/;
  • or wait for 0.34.0.

ModelOptNvFp4Config.get_min_capability() is 75, so sm_121 is not gated out. The PR was validated on B300 only, so NVFP4 kernel selection on GB10 is the open risk — flashinfer's sm121 cutlass FP4 path working under SGLang suggests it will land on a real kernel rather than a marlin fallback.

Operational notes

  • dgx4 must have no prod4 ranks while this runs: it occupies 103 GiB of the 121 GiB unified memory. Stop it before the cluster comes back.
  • Weights live on dgx4's local NVMe (144 GB used of 1.8 T before, ~198 GB after). Movable to the NAS store later; cold start would pay 53 GiB over 1 GbE.
  • Context defaults to 131072 (--context-length), although the config claims 2 097 152 positions; SGLang corrected max_total_tokens down to its profiled 134 119 at load time.