TP4 switchless-ring serving numbers (4× DGX Spark)

Independent reproduction of the shipped TP4 profile on a switchless 4-node ring (no RoCE switch). The runtime was already serving deepseek-v4.1-flash at the advertised envelope:

context 1,048,576
MAX_RUNNING_REQUESTS=8 (CUDA-graph decode batch 1–8)
pinned KV MAX_TOTAL_TOKENS=8,000,000
MEM_FRACTION_STATIC=0.80, chunked prefill 1024
DSpark k=5, MXFP4 experts + MXFP8 dense, Engram packed on local NVMe
NCCL_SWITCHLESS_RING_ONLY=1 (sparkring overlay, not LD_LIBRARY_PATH)

Raw JSON: results/tp4-switchless-ring-20260915.json. Harnesses: benchmarks/openai_serving.py (public /v1/chat/completions) and benchmarks/decode_window.py (idle C1–C8, character-scaled estimate of tokens 129–641).

This change also includes the opt-in launcher integration used by the live fleet: patched-NCCL overlay without duplicate runtimes, ring-only env on every rank, per-node preflight checks, GB10 UMA-safe serial weight loading, local-weight NFS_SHARE=0 operation/status, and a physical cabling/IP guide. The ring transport/overlay foundation derives from PR #3 by @Saolence and is rebased and extended here onto current main. The external NCCL transport patch remains credited to FujitsuPolycom/sparkring.

Hardware / software


Nodes	4× NVIDIA GB10 (DGX Spark), aarch64
Kernel / driver	`6.17.0-1031-nvidia` / `580.173.02` on all four
Fabric	switchless ring, 200 GbE CX7, MTU 9000
Image	`dsv41-4x-spark:local` (`lmsysorg/sglang:dev-dsv41`)
Measuring checkout	`68b34cf` (TP4 1M-needle README)
Weights	local copies, `NFS_SHARE=0`

Correctness (OpenAI path)

Check	Result
`GET /v1/models`	`deepseek-v4.1-flash`, `max_model_len=1048576`
`19+23`, thinking off	`42`, `finish_reason=stop`
thinking on	`OK99`, `reasoning_tokens=17`
tool call	`get_weather({"city":"Vigo"})`
8 concurrent streams	8/8 HTTP 200, queue 0

Decode-window sweep (idle cluster)

Greedy code generation, streaming, thinking off. Window 129–641 is estimated by scaling cumulative output characters to final completion-token usage; SSE chunks are not individual tokens. Historical TTFT values below were also character-scaled estimates. The corrected harness measures TTFT at the first nonempty content event. Wall-clock aggregate uses final token usage and elapsed time. Server idle (running=0 waiting=0) before each wave.

C	Aggregate tok/s	Median estimated window tok/s / stream	Estimated TTFT	Success
1	74.03	67.41	0.535 s	1/1
2	111.48	54.98	0.738 s	2/2
4	160.05	40.84	0.978 s	4/4
8	224.84	29.91	1.181 s	8/8

An earlier short-prompt report recorded about C1 ~68 / C8 ~201. Prompt, completion length and measurement method differ; these runs do not establish a topology advantage or isolate a DSpark acceptance-rate effect.

Saturation (8 long streams)

8 concurrent HTML/code generations, 6000 completion tokens each, thinking off:

48,000 completion tokens in 222.413 s → 215.81 tok/s wall-clock aggregate
SGLang decode-batch peak 258.63 tok/s
#running-req: 8, #queue-req: 0
~27 tok/s per stream

Long context (needle)

Prompt tokens	C	Correct	Wall
~26k	1	1/1	12.8 s
~26k	4	4/4	36.8 s
~107k	1	1/1	414.7 s

A full 1M needle was not re-run in this pass (the shipped TP4 profile already reports it passed). CONTEXT_LENGTH was not changed.

Caveats

A short-prose max_tokens=256 matrix taken while other requests held ~557k KV is not a speed number (C1 looked like 16 tok/s). Discard it.
Character-scaled decode-window tok/s is approximate and differs from end-to-end tok/s. Short completions are dominated by TTFT/prefill.
Live checkout was 68b34cf + DSpark k=5. main after PR #18 defaults DSpark k=3 and adds loop-abort / DSV41_MAX_NEW_TOKENS. These benches do not claim k=3 performance.
No A/B against a switched 4-node fabric.

Reproduce

# Idle cluster required; set API_KEY if the endpoint requires authentication.
# These commands use the current serving configuration (default k=3), whereas
# the recorded measurements used k=5.
BASE_URL=http://127.0.0.1:8888 python3 benchmarks/decode_window.py
BASE_URL=http://127.0.0.1:8888 python3 benchmarks/openai_serving.py --skip-needles

Measured on Carlos's 4× DGX Spark ring. Twitter: @mankalan_122

tp4-switchless-ring-results.md 4.5 KB История Исходник