|
|
@@ -918,8 +918,8 @@ the detect command per node and renders `NCCL_IB_GID_INDEX=auto`.
|
|
|
- Use `rg` (ripgrep) for all code search — plain `grep -rn` across this tree aborts/hangs and is unusable; `rg` is fast and skips `.git`/gitignored paths
|
|
|
- `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`
|
|
|
- **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.
|
|
|
-- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 69+19 = 88 scenarios in 2.6.1.dev45; the older 176-pt pool is retired) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; re-baseline the anchor lane before comparing new lanes.
|
|
|
-- **Harness version is pinned — do not `uv tool upgrade` casually.** `tool-eval-bench` is a uv tool (`~/.local/share/uv/tools/tool-eval-bench`) installed from `SeraphimSerapis/tool-eval-bench.git`; the receipt is rev-pinned at tag `v2.7.0` (object `544afb576b3`, 2026-09-21 — the first stable release of the dev65 line; `reasoning_truncated` is now a distinct failure kind). The previous pin was `6be685f0e` (v2.6.1.dev65, campaign baseline 2026-09-11 — the 167/176 record run is on it); dev65-era anchors read slightly differently on v2.7.0 until re-baselined. The 19 dev46→dev65 commits (09-06..09-08) were **scoring-credit fixes** (correct behavior was being failed), so dev46 baselines (all GLM lanes: r28 158, r31 159, sparkring) read slightly low next to dev65 runs; cross-harness deltas include that bias. Reports stamp the harness version in the header — **A/B runs must share a harness version**. To move the pin deliberately: `uv tool install --reinstall --force "tool-eval-bench[perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<tag-or-rev>"` then re-baseline the anchor lane.
|
|
|
+- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 97 scenarios on v2.8.0, was 92 on 2.7.1.dev25, 88 on 2.7.0) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; the tool is installed from `SeraphimSerapis/tool-eval-bench` — current pin: **v2.8.0** (2026-10-14); re-baseline the anchor lane before comparing new lanes.
|
|
|
+- **Harness version — do not `uv tool upgrade` casually.** `tool-eval-bench` is a uv tool (`~/.local/share/uv/tools/tool-eval-bench`) installed from `SeraphimSerapis/tool-eval-bench.git`; the receipt is pinned at tag **v2.8.0** (2026-10-14; 97 scenarios; perf presets, session ids, shareable results). Prior pins: 2.7.1.dev7 (92 scen), 2.7.1.dev25 (the 8x 95/100 anchor, 2026-10-12), v2.7.0 `544afb576b3` (2026-09-21, first stable of dev65). Reports stamp the harness version in the header — **A/B runs must share a harness version**; pools differ across versions (88→92→97), so cross-version deltas are not comparable. To move the pin: `uv tool install --reinstall --force "tool-eval-bench[perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<tag-or-rev>"` then re-baseline the anchor lane. Reasoning effort: engine defaults (recipe templates) — explicit low-effort runs are separate labeled datapoints, not comparable to the standard ones.
|
|
|
- **Bulk data moves node-to-node, never through the Mac** — the Mac→node link
|
|
|
is ~32 MB/s (a 200 GB transfer via the Mac takes hours); node-to-node over the
|
|
|
200G fabric does ~547 MB/s per stream. Node-to-node SSH works (the shared
|
