|
|
@@ -1062,8 +1062,8 @@ the detect command per node and renders `NCCL_IB_GID_INDEX=auto`.
|
|
|
- Use `rg` (ripgrep) for all code search — plain `grep -rn` across this tree aborts/hangs and is unusable; `rg` is fast and skips `.git`/gitignored paths
|
|
|
- `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`
|
|
|
- **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.
|
|
|
-- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 97 scenarios on v2.8.0, was 92 on 2.7.1.dev25, 88 on 2.7.0) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; the tool is installed from `SeraphimSerapis/tool-eval-bench` — current pin: **v2.8.0** (2026-10-14); re-baseline the anchor lane before comparing new lanes.
|
|
|
-- **Harness version — do not `uv tool upgrade` casually.** `tool-eval-bench` is a uv tool (`~/.local/share/uv/tools/tool-eval-bench`) installed from `SeraphimSerapis/tool-eval-bench.git`; the receipt is pinned at tag **v2.8.0** (2026-10-14; 97 scenarios; perf presets, session ids, shareable results). Prior pins: 2.7.1.dev7 (92 scen), 2.7.1.dev25 (the 8x 95/100 anchor, 2026-10-12), v2.7.0 `544afb576b3` (2026-09-21, first stable of dev65). Reports stamp the harness version in the header — **A/B runs must share a harness version**; pools differ across versions (88→92→97), so cross-version deltas are not comparable. To move the pin: `uv tool install --reinstall --force "tool-eval-bench[perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<tag-or-rev>"` then re-baseline the anchor lane. Reasoning effort: engine defaults (recipe templates) — explicit low-effort runs are separate labeled datapoints, not comparable to the standard ones.
|
|
|
+- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 97 scenarios on v3.0.0 and v2.8.0; was 92 on 2.7.1.dev25, 88 on 2.7.0) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; the tool is installed from `SeraphimSerapis/tool-eval-bench` — current pin: **v3.0.0** (2026-10-26, commit `d85126e`; 69 standard / 97 Hard Mode). Do not compare v2.8.0 reports with v3.0.0 reports; re-baseline the anchor lane first.
|
|
|
+- **Harness version — do not `uv tool upgrade` casually.** `tool-eval-bench` is a uv tool (`~/.local/share/uv/tools/tool-eval-bench`) installed from `SeraphimSerapis/tool-eval-bench.git`; the receipt is pinned at tag **v3.0.0** (2026-10-26, commit `d85126e`; 69 standard / 97 Hard Mode; web UI, `card`, `--perf-preset long`, `--needle`). Prior pins: v2.8.0 (2026-10-14; 97 scenarios; the `91/100`, `88/100`, `95/100` reports are v2.8.0), 2.7.1.dev7 (92 scen), 2.7.1.dev25 (the 8x 95/100 anchor, 2026-10-12), v2.7.0 `544afb576b3` (2026-09-21, first stable of dev65). Reports stamp the harness version in the header — **A/B runs must share a harness version**; pools and evaluators differ across versions (88→92→97), and v3.0.0 changes TC-21/TC-89 plus structured-output retries, so cross-version deltas are not comparable. To move the pin: `uv tool install --reinstall --force "tool-eval-bench[perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<tag-or-rev>"` then re-baseline the anchor lane. Reasoning effort: engine defaults (recipe templates) — explicit low-effort runs are separate labeled datapoints, not comparable to the standard ones.
|
|
|
- **Bulk data moves node-to-node, never through the Mac** — the Mac→node link
|
|
|
is ~32 MB/s (a 200 GB transfer via the Mac takes hours); node-to-node over the
|
|
|
200G fabric does ~547 MB/s per stream. Node-to-node SSH works (the shared
|
