|
@@ -0,0 +1,105 @@
|
|
|
|
|
+# Ripple-65B NVFP4 on a single GB10 (dgx4), 2026-11-11
|
|
|
|
|
+
|
|
|
|
|
+`ricursiveAI/Ripple-65B-Preview-NVFP4` runs on one DGX Spark. The vendor ships a
|
|
|
|
|
+hardware-locked SGLang path; vLLM support exists upstream but is not in any
|
|
|
|
|
+release yet. Numbers below are measured on dgx4, not vendor claims.
|
|
|
|
|
+
|
|
|
|
|
+## The model
|
|
|
|
|
+
|
|
|
|
|
+| | |
|
|
|
|
|
+|---|---|
|
|
|
|
|
+| repo | `ricursiveAI/Ripple-65B-Preview-NVFP4`, public, Apache-2.0, created 04.11.2026 |
|
|
|
|
|
+| parameters | **39.6 B** dense (the "65B" name is marketing), 80 layers, hybrid attention 4 sliding : 1 full |
|
|
|
|
|
+| size on disk | **53.28 GiB** in 6 shards (byte-verified against HF's published sizes) |
|
|
|
|
|
+| quantisation | MLP **NVFP4** static group 16, attention/embed/lm_head **BF16**, BF16 KV cache |
|
|
|
|
|
+| draft | `ricursiveAI/Ripple-65B-Preview-DFlash2`, 4.04 B, 7.6 GB, unquantised |
|
|
|
|
|
+| licence note | HF tag says "8-bit"; that is wrong — read `quantization_config` |
|
|
|
|
|
+
|
|
|
|
|
+## Why the vendor code is DGX-Spark-only
|
|
|
|
|
+
|
|
|
|
|
+`inference/local/spark.py` refuses any other device:
|
|
|
|
|
+
|
|
|
|
|
+```python
|
|
|
|
|
+if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (12, 1):
|
|
|
|
|
+ raise RuntimeError("Run this notebook on a DGX Spark / GB10 GPU")
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+It also patches SGLang internals for SM121: `enable_pdl=False` in the flashinfer
|
|
|
|
|
+softmax ("SM121 PDL softmax can produce NaNs"), KV scales forced to 1.0 for
|
|
|
|
|
+quantised exports with BF16 KV, and `--mem-fraction-static 0.78`. Code-writing
|
|
|
|
|
+helpers (`ripple_kernels.py`, `ripple_tree*.py`, `ripple_transport.py`) must sit
|
|
|
|
|
+next to `spark.py`.
|
|
|
|
|
+
|
|
|
|
|
+## Reproduction (dgx4, Ubuntu 24.04, Python 3.12.3, 20 cores)
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+python3 -m venv ~/venvs/ripple
|
|
|
|
|
+~/venvs/ripple/bin/pip install --pre --extra-index-url https://download.pytorch.org/whl/cu130 \
|
|
|
|
|
+ "sglang==0.5.19" "torch==2.13.0+cu130" "transformers==5.12.1" \
|
|
|
|
|
+ "flashinfer-python==0.6.18" "ninja==1.13.2" "nest_asyncio==1.6.0"
|
|
|
|
|
+
|
|
|
|
|
+# checkpoints (skip the notebook, it is not needed to serve)
|
|
|
|
|
+~/venvs/ripple/bin/python -c "
|
|
|
|
|
+from huggingface_hub import snapshot_download as s
|
|
|
|
|
+s('ricursiveAI/Ripple-65B-Preview-NVFP4', local_dir='/home/kilork/models/Ripple-65B-Preview-NVFP4', ignore_patterns=['*.mp4','*.ipynb'])
|
|
|
|
|
+s('ricursiveAI/Ripple-65B-Preview-DFlash2', local_dir='/home/kilork/models/Ripple-65B-Preview-DFlash2', ignore_patterns=['*.mp4','*.ipynb'])"
|
|
|
|
|
+
|
|
|
|
|
+export CUDA_HOME=/usr/local/cuda-13.0
|
|
|
|
|
+~/venvs/ripple/bin/python ~/models/Ripple-65B-Preview-NVFP4/inference/local/spark.py \
|
|
|
|
|
+ --target ~/models/Ripple-65B-Preview-NVFP4 \
|
|
|
|
|
+ --draft ~/models/Ripple-65B-Preview-DFlash2 \
|
|
|
|
|
+ --decoding tree --draft-tokens 32 \
|
|
|
|
|
+ --host 0.0.0.0 --port 8010
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+First start compiles flashinfer's `fp4_gemm_cutlass` for sm121 into
|
|
|
|
|
+`~/.cache/sglang/.cache/flashinfer/0.6.18/121a/cached_ops/` with two `cicc`
|
|
|
|
|
+processes — about 10 minutes. Later starts load in ~35 s and capture CUDA graphs
|
|
|
|
|
+in ~2 min.
|
|
|
|
|
+
|
|
|
|
|
+Do not stop it with `pkill -f "spark.py"` from an ssh one-liner: the pattern
|
|
|
|
|
+matches the ssh shell and kills the session. Use `pkill -f "spark[.]py"`.
|
|
|
|
|
+
|
|
|
|
|
+## Measured (dgx4, GB10, driver 580.178.04, 121 GiB unified)
|
|
|
|
|
+
|
|
|
|
|
+| decoding | code prompt (stream) | prose, 128 tok | 243-token prompt | TTFT |
|
|
|
|
|
+|---|---|---|---|---|
|
|
|
|
|
+| target only | 4.1 tok/s | 4.05 tok/s | 3.76 tok/s | 0.40 s |
|
|
|
|
|
+| chain, 16 draft tokens | 18.7 tok/s | 6.8 tok/s | 7.25 tok/s | 0.50 s |
|
|
|
|
|
+| **tree, 32 nodes (vendor default)** | **22.6 tok/s** | **10.4 tok/s** | **11.1 tok/s** | 0.50 s |
|
|
|
|
|
+
|
|
|
|
|
+4 concurrent requests, tree decoding: **39.9 tok/s aggregate** (401 tokens in
|
|
|
|
|
+10.05 s wall), 110 / 82 / 128 / 81 tokens per request.
|
|
|
|
|
+
|
|
|
|
|
+Memory: 103 GiB of 121 GiB with target + draft loaded (`--mem-fraction-static
|
|
|
|
|
+0.78`); the target alone is 98 GiB. Load time 35 s from NVMe (1.5 GB/s).
|
|
|
|
|
+
|
|
|
|
|
+The spread between "code" and "prose" is acceptance rate, not a benchmark
|
|
|
|
|
+artefact: predictable text accepts 4–6 draft tokens per step, prose accepts far
|
|
|
|
|
+fewer.
|
|
|
|
|
+
|
|
|
|
|
+## vLLM path (not taken, but viable)
|
|
|
|
|
+
|
|
|
|
|
+vLLM implements the architecture natively — `RicursiveForCausalLM` and
|
|
|
|
|
+`DFlash2RicursiveDraftModel` in `vllm/model_executor/models/registry.py`, with
|
|
|
|
|
+NVFP4, TP/PP, torch.compile, CUDA graphs, `--reasoning-parser ripple` /
|
|
|
|
|
+`--tool-call-parser ripple` and no `--trust-remote-code`. It merged **05.11.2026**
|
|
|
|
|
+(PR #65745), so it is not in `0.33.0` (02.11). Options when we want it:
|
|
|
|
|
+
|
|
|
|
|
+- nightly aarch64 wheel, e.g. `vllm-0.34.0rc1.dev210+gbafa5c6c3-cp38-abi3-manylinux_2_28_aarch64.whl` (229 MiB) from `https://wheels.vllm.ai/nightly/vllm/`;
|
|
|
|
|
+- or wait for `0.34.0`.
|
|
|
|
|
+
|
|
|
|
|
+`ModelOptNvFp4Config.get_min_capability()` is 75, so sm_121 is not gated out. The
|
|
|
|
|
+PR was validated on **B300** only, so NVFP4 kernel selection on GB10 is the open
|
|
|
|
|
+risk — flashinfer's sm121 cutlass FP4 path working under SGLang suggests it will
|
|
|
|
|
+land on a real kernel rather than a marlin fallback.
|
|
|
|
|
+
|
|
|
|
|
+## Operational notes
|
|
|
|
|
+
|
|
|
|
|
+- dgx4 must have no prod4 ranks while this runs: it occupies 103 GiB of the 121
|
|
|
|
|
+ GiB unified memory. Stop it before the cluster comes back.
|
|
|
|
|
+- Weights live on dgx4's local NVMe (144 GB used of 1.8 T before, ~198 GB after).
|
|
|
|
|
+ Movable to the NAS store later; cold start would pay 53 GiB over 1 GbE.
|
|
|
|
|
+- Context defaults to 131072 (`--context-length`), although the config claims
|
|
|
|
|
+ 2 097 152 positions; SGLang corrected `max_total_tokens` down to its profiled
|
|
|
|
|
+ 134 119 at load time.
|