recipes/docker-images/glm53-kindling/gate/bench

bench.py: decode speed with error bars

A decode benchmark built for speculative decoding, where speed depends on how easy each answer is to draft. A single run per prompt can't separate two setups closer than about 10%. This one repeats each workload with different prompts and reports how sure it is.

The approach follows RigMark: fixed prompts behind a per-run tag, warmups, and checks on the output. The code is our own.

Workloads

workload	what runs
code	one stream, 512 tokens, a different coding prompt each run
prose	one stream, 512 tokens, a different explanation prompt each run
structured	one stream, 512 tokens, a different number list each run
counting	one stream, count to 200; any wrong line fails the run
mixed4, mixed8	4 or 8 concurrent streams of code, prose and structured prompts

Every workload gets 2 warmup runs that aren't recorded, then 16 recorded runs (8 batches for mixed). Requests use temperature 0 with thinking off. Each carries a run tag fixed by the series name, the workload and the run number. So run 5 of code sends the same prompt and tag on every setup, and a server can't answer it from its prefix cache. Every run is also checked for Hangul and replacement characters, which is how corruption on this model has shown up before.

What it reports

For each run: output tokens, time to first token, decode time (first token to last), and the speculative-decoding counters from /metrics. Those give tokens accepted per verify step, which depends on how easy the output was to draft, and time per step, which depends on the setup and on how many drafts adaptive-k chose to verify.

For each workload:

• tok/s: total output tokens / total decode time, with a bootstrap 95% interval
• spread: the run-to-run standard deviation, as a share of the mean
• detects: the smallest difference an unpaired comparison of this many runs would find (95% confidence, 80% power). Pairing does much better, because the spread is mostly prompt-to-prompt difference.
• outliers: runs with a modified Z-score above 3.5

compare pairs two result files run by run, and reports B / A as a geometric-mean ratio with a bootstrap 95% interval and a two-sided exact Wilcoxon signed-rank p-value, Holm-corrected across workloads. It splits the ratio into tokens per step and time per step, so you can tell a faster setup from easier answers.

Usage

python3 gate/bench/bench.py run --label mysetup --out mysetup.json
python3 gate/bench/bench.py compare base.json mysetup.json

GATE_URL (default http://127.0.0.1:8002) and GATE_MODEL (default glm53) pick the server. Runs pair only within one --series. A full run takes about 15 minutes at TP=4.

Check: two boots of the same build

Two separate TP=4 boots of one build, compared with compare:

workload	B / A	95% interval
code	1.004	0.996 to 1.015
prose	1.004	1.001 to 1.007
structured	0.998	0.993 to 1.001
counting	1.003	1.000 to 1.009
mixed4	0.976	0.951 to 1.000
mixed8	1.001	0.968 to 1.036

None differ (Holm-corrected p ≥ 0.20). Single-stream code varies 15% from run to run, and pairing still puts two boots within about 1% of each other.

Results

tok/s, with the bootstrap 95% interval. Main at 4e63b64, image v23, boots that restored weight snapshots. TP=4 pools two boots (32 runs per workload, 16 batches for mixed). TP=3 and TP=2 are one boot each. Every run passed its output checks. TP=6 hasn't been measured this way.

workload	TP=2	TP=3	TP=4
code	66.2 (61.5 to 71.6)	88.0 (82.9 to 93.9)	117.3 (111.7 to 123.9)
prose	42.6 (41.7 to 43.6)	54.2 (53.3 to 55.1)	70.5 (69.5 to 71.6)
structured	94.7 (89.1 to 100.4)	121.4 (115.2 to 127.8)	164.3 (158.3 to 170.4)
counting	97.2 (96.0 to 98.5)	125.0 (122.8 to 127.2)	170.5 (168.5 to 172.4)
mixed, 4 streams	82.6 (79.0 to 86.1)	106.8 (100.6 to 113.7)	144.4 (139.3 to 149.7)
mixed, 8 streams	109.6 (103.0 to 117.6)	138.0 (129.6 to 146.1)	194.0 (188.5 to 200.2)