Переглянути джерело

bench instructions: keep json report (--json-file) alongside the md

Alexander Korolev 1 місяць тому
батько
коміт
98dd4e42b3
2 змінених файлів з 8 додано та 2 видалено
  1. 1 1
      AGENTS.md
  2. 7 1
      specs/tech-architecture/glm53-recipe-state.md

+ 1 - 1
AGENTS.md

@@ -518,4 +518,4 @@ the detect command per node and renders `NCCL_IB_GID_INDEX=auto`.
 - Use `rg` (ripgrep) for all code search — plain `grep -rn` across this tree aborts/hangs and is unusable; `rg` is fast and skips `.git`/gitignored paths
 - `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`
 - **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.
-- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir). Quality-gate semantics are version-sensitive (`--hardmode` = 69+19 = 88 scenarios in 2.6.1.dev45; the older 176-pt pool is retired) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; re-baseline the anchor lane before comparing new lanes.
+- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 69+19 = 88 scenarios in 2.6.1.dev45; the older 176-pt pool is retired) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; re-baseline the anchor lane before comparing new lanes.

+ 7 - 1
specs/tech-architecture/glm53-recipe-state.md

@@ -5,9 +5,15 @@ served by sm121 through the launcher, same benchmark invocation:
 
 ```
 tool-eval-bench run --base-url http://<head>:30000/v1 --model glm53 \
-  --seed 42 --hardmode --perf --concurrency 1,2
+  --seed 42 --hardmode --perf --concurrency 1,2 \
+  --json-file <abs repo path>/recipes/<lane>/reports/<UTC-stamp>_<label>.json
 ```
 
+Keep BOTH the rich `.md` report (copied from the tool's `runs/` output dir)
+and the raw JSON (`--json-file`; absolute path — the bench runs in a separate
+cwd, and the flag keeps stdout clean for the `.log`) in
+`recipes/<lane>/reports/` under the same base name.
+
 **Tool version pin (2026-09-05, verified against the installed tool):**
 tool-eval-bench 2.6.1.dev45+gcf54b4bfe (uv tool; git
 SeraphimSerapis/tool-eval-bench @ cf54b4bfe). Current pool semantics: