Explorar o código

docs(spark): Ripple-65B NVFP4 running on one GB10, measured

Vendor SGLang path (inference/local/spark.py, hardware-locked to sm_121), stack
pinned to sglang 0.5.19 / torch 2.13.0+cu130 / flashinfer 0.6.18, target 53.28 GiB
plus the 4.04 B DFlash2 draft on dgx4.

Measured: 4.05 tok/s target-only, 6.8 / 18.7 tok/s chain (prose / code), 10.4 /
22.6 tok/s tree, 39.9 tok/s aggregate at 4 concurrent, TTFT 0.4-0.5 s, 103 of
121 GiB resident.

Records the first-run flashinfer sm121 fp4_gemm_cutlass JIT (~10 min), the
self-killing pkill pattern, and the vLLM alternative: native Ricursive support
merged 2026-11-05 (PR #65745, not in 0.33.0), nightly aarch64 wheel available,
NVFP4 min capability 75 but validated on B300 only.
Alexander Korolev hai 1 semana
pai
achega
ae6ea2f26e
Modificáronse 1 ficheiros con 105 adicións e 0 borrados
  1. 105 0
      specs/tech-architecture/ripple-65b-gb10.md

+ 105 - 0
specs/tech-architecture/ripple-65b-gb10.md

@@ -0,0 +1,105 @@
+# Ripple-65B NVFP4 on a single GB10 (dgx4), 2026-11-11
+
+`ricursiveAI/Ripple-65B-Preview-NVFP4` runs on one DGX Spark. The vendor ships a
+hardware-locked SGLang path; vLLM support exists upstream but is not in any
+release yet. Numbers below are measured on dgx4, not vendor claims.
+
+## The model
+
+| | |
+|---|---|
+| repo | `ricursiveAI/Ripple-65B-Preview-NVFP4`, public, Apache-2.0, created 04.11.2026 |
+| parameters | **39.6 B** dense (the "65B" name is marketing), 80 layers, hybrid attention 4 sliding : 1 full |
+| size on disk | **53.28 GiB** in 6 shards (byte-verified against HF's published sizes) |
+| quantisation | MLP **NVFP4** static group 16, attention/embed/lm_head **BF16**, BF16 KV cache |
+| draft | `ricursiveAI/Ripple-65B-Preview-DFlash2`, 4.04 B, 7.6 GB, unquantised |
+| licence note | HF tag says "8-bit"; that is wrong — read `quantization_config` |
+
+## Why the vendor code is DGX-Spark-only
+
+`inference/local/spark.py` refuses any other device:
+
+```python
+if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (12, 1):
+    raise RuntimeError("Run this notebook on a DGX Spark / GB10 GPU")
+```
+
+It also patches SGLang internals for SM121: `enable_pdl=False` in the flashinfer
+softmax ("SM121 PDL softmax can produce NaNs"), KV scales forced to 1.0 for
+quantised exports with BF16 KV, and `--mem-fraction-static 0.78`. Code-writing
+helpers (`ripple_kernels.py`, `ripple_tree*.py`, `ripple_transport.py`) must sit
+next to `spark.py`.
+
+## Reproduction (dgx4, Ubuntu 24.04, Python 3.12.3, 20 cores)
+
+```bash
+python3 -m venv ~/venvs/ripple
+~/venvs/ripple/bin/pip install --pre --extra-index-url https://download.pytorch.org/whl/cu130 \
+  "sglang==0.5.19" "torch==2.13.0+cu130" "transformers==5.12.1" \
+  "flashinfer-python==0.6.18" "ninja==1.13.2" "nest_asyncio==1.6.0"
+
+# checkpoints (skip the notebook, it is not needed to serve)
+~/venvs/ripple/bin/python -c "
+from huggingface_hub import snapshot_download as s
+s('ricursiveAI/Ripple-65B-Preview-NVFP4', local_dir='/home/kilork/models/Ripple-65B-Preview-NVFP4', ignore_patterns=['*.mp4','*.ipynb'])
+s('ricursiveAI/Ripple-65B-Preview-DFlash2', local_dir='/home/kilork/models/Ripple-65B-Preview-DFlash2', ignore_patterns=['*.mp4','*.ipynb'])"
+
+export CUDA_HOME=/usr/local/cuda-13.0
+~/venvs/ripple/bin/python ~/models/Ripple-65B-Preview-NVFP4/inference/local/spark.py \
+  --target ~/models/Ripple-65B-Preview-NVFP4 \
+  --draft  ~/models/Ripple-65B-Preview-DFlash2 \
+  --decoding tree --draft-tokens 32 \
+  --host 0.0.0.0 --port 8010
+```
+
+First start compiles flashinfer's `fp4_gemm_cutlass` for sm121 into
+`~/.cache/sglang/.cache/flashinfer/0.6.18/121a/cached_ops/` with two `cicc`
+processes — about 10 minutes. Later starts load in ~35 s and capture CUDA graphs
+in ~2 min.
+
+Do not stop it with `pkill -f "spark.py"` from an ssh one-liner: the pattern
+matches the ssh shell and kills the session. Use `pkill -f "spark[.]py"`.
+
+## Measured (dgx4, GB10, driver 580.178.04, 121 GiB unified)
+
+| decoding | code prompt (stream) | prose, 128 tok | 243-token prompt | TTFT |
+|---|---|---|---|---|
+| target only | 4.1 tok/s | 4.05 tok/s | 3.76 tok/s | 0.40 s |
+| chain, 16 draft tokens | 18.7 tok/s | 6.8 tok/s | 7.25 tok/s | 0.50 s |
+| **tree, 32 nodes (vendor default)** | **22.6 tok/s** | **10.4 tok/s** | **11.1 tok/s** | 0.50 s |
+
+4 concurrent requests, tree decoding: **39.9 tok/s aggregate** (401 tokens in
+10.05 s wall), 110 / 82 / 128 / 81 tokens per request.
+
+Memory: 103 GiB of 121 GiB with target + draft loaded (`--mem-fraction-static
+0.78`); the target alone is 98 GiB. Load time 35 s from NVMe (1.5 GB/s).
+
+The spread between "code" and "prose" is acceptance rate, not a benchmark
+artefact: predictable text accepts 4–6 draft tokens per step, prose accepts far
+fewer.
+
+## vLLM path (not taken, but viable)
+
+vLLM implements the architecture natively — `RicursiveForCausalLM` and
+`DFlash2RicursiveDraftModel` in `vllm/model_executor/models/registry.py`, with
+NVFP4, TP/PP, torch.compile, CUDA graphs, `--reasoning-parser ripple` /
+`--tool-call-parser ripple` and no `--trust-remote-code`. It merged **05.11.2026**
+(PR #65745), so it is not in `0.33.0` (02.11). Options when we want it:
+
+- nightly aarch64 wheel, e.g. `vllm-0.34.0rc1.dev210+gbafa5c6c3-cp38-abi3-manylinux_2_28_aarch64.whl` (229 MiB) from `https://wheels.vllm.ai/nightly/vllm/`;
+- or wait for `0.34.0`.
+
+`ModelOptNvFp4Config.get_min_capability()` is 75, so sm_121 is not gated out. The
+PR was validated on **B300** only, so NVFP4 kernel selection on GB10 is the open
+risk — flashinfer's sm121 cutlass FP4 path working under SGLang suggests it will
+land on a real kernel rather than a marlin fallback.
+
+## Operational notes
+
+- dgx4 must have no prod4 ranks while this runs: it occupies 103 GiB of the 121
+  GiB unified memory. Stop it before the cluster comes back.
+- Weights live on dgx4's local NVMe (144 GB used of 1.8 T before, ~198 GB after).
+  Movable to the NAS store later; cold start would pay 53 GiB over 1 GbE.
+- Context defaults to 131072 (`--context-length`), although the config claims
+  2 097 152 positions; SGLang corrected `max_total_tokens` down to its profiled
+  134 119 at load time.