|
|
@@ -287,3 +287,18 @@ buffer shape-follow patches (patch_mm_hc_streams.py), placeholder
|
|
|
integration, AOT=0 knob for mm experiments. nid: fork A/B (aiden base)
|
|
|
still pending its pip-layout patch; dspark_gate patch now auto-finds the
|
|
|
fork's venv root.
|
|
|
+
|
|
|
+### dsv4vision tool-eval-bench (c1, 2026-09-01)
|
|
|
+
|
|
|
+`tool-eval-bench run --base-url http://dgx3.lan:30000/v1 --model deepseek-v4-flash-vision --seed 42 --hardmode --perf --concurrency 1` → report `/private/tmp/runs/2026/09/2026-09-01T17-46-28.764372Z_89b18250.md`:
|
|
|
+
|
|
|
+| recipe | quality | points | decode tg t/s c1 (d0/d4096/d8192) | pp2048 t/s | TTFT | deployability |
|
|
|
+|---|---|---|---|---|---|---|
|
|
|
+| `dsv4-flash-vision` (text lane, local-NVMe, DSpark-6) | **81** | 143/176 | — | — | — | — |
|
|
|
+
|
|
|
+perf block: tg 34.1 / 31.0 / 29.1, pp2048 3,907 / 2,714 / 2,449, TTFT 1.9-4.9 s (spec),
|
|
|
+scenario-median turn 2.4 s, deployability **74** (best of any ds4 lane) — long-context
|
|
|
+capability is the raw 1M-context draw. 3 safety-critical failures (TC-42 extra-param,
|
|
|
+TC-43 empty required param, TC-60 sleeper injection) — the day-0 0731-style tool
|
|
|
+formatting is the gap vs the GLM lanes (thinking-pair JSON fencing was already a
|
|
|
+known day-0 weakness). Quality 81 < sparkrun 89 < lab 92.
|
kilork