|
|
@@ -530,6 +530,30 @@ fork) measured ~2.3× slower on this box despite the forum's 52-87 tok/s claims
|
|
|
`specs/tech-architecture/qwen38-recipe-state.md` for the full measured matrix,
|
|
|
why each decision was made, and how to re-benchmark.
|
|
|
|
|
|
+### GLM-5.3-Flash NVFP4 re-cut (measured)
|
|
|
+
|
|
|
+`local-inference-lab/GLM-5.3-Flash-NVFP4` was **re-cut in place on 2026-09-16**
|
|
|
+(the old ~186 GB upload is gone): 199.4 GB = 36 expert shards + 4 nonexpert +
|
|
|
+4 MTP shards. Structure is identical to the Spark checkpoint (verified from
|
|
|
+`model.safetensors.index.json`): `model.language_model.layers.*` with SPLIT GDN
|
|
|
+naming, a `model.visual.*` tower (skip via `--language-model-only`), MTP head
|
|
|
+embedded at layer 45 — so the JJ-fork engine requirements are the same as the
|
|
|
+Spark lanes (r35 + `VLLM_DISABLED_KERNELS`; r38 pool-deadlock on 2x still
|
|
|
+applies). Lanes: `glm53-flash-nvfp4-lab` (2x, same dir as the old pre-re-cut
|
|
|
+lane — it had no reports/lock) and `glm53-flash-nvfp4-lab-4x` (4x).
|
|
|
+
|
|
|
+Measured 2026-09-20 (r35, harness dev65, seed 42, hardmode):
|
|
|
+- **2x: 164/176 (93/100) — the family best** (old 186 GB cut: 89/100; Spark r35
|
|
|
+ 2x: 92). The operator's "they improved it" hunch was right. But the re-cut is
|
|
|
+ heavier: ~99 GiB weights/rank leaves only **2.4 GiB KV at GMU 0.88**, which
|
|
|
+ cannot hold one 524K request (needs 3.73 GiB) → the 2x lane is capped at
|
|
|
+ **256K context** (needs ~1.9 GiB; bench contexts are far below that).
|
|
|
+- **4x: 159/176 (90/100)**, decode 54-68 t/s (2× the 2x), KV 49.3 GiB →
|
|
|
+ 7.35M-token pool @524K. The 4x loses 5 pts to the 2x (TC-43/45/49/68/80/88
|
|
|
+ flakes); TC-51 fails on both (known harness recurring).
|
|
|
+- Weights live on all 4 nodes' local NVMe (`hf-local`, snapshot
|
|
|
+ `175ae8ce`); dgx1 is the tightest host (~55 GB free after the model).
|
|
|
+
|
|
|
## Config fields
|
|
|
|
|
|
Config resolution: `--config <path>` > `./config.toml` (cwd) > the global
|
|
|
@@ -613,3 +637,21 @@ the detect command per node and renders `NCCL_IB_GID_INDEX=auto`.
|
|
|
- **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.
|
|
|
- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 69+19 = 88 scenarios in 2.6.1.dev45; the older 176-pt pool is retired) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; re-baseline the anchor lane before comparing new lanes.
|
|
|
- **Harness version is pinned — do not `uv tool upgrade` casually.** `tool-eval-bench` is a uv tool (`~/.local/share/uv/tools/tool-eval-bench`) installed from `SeraphimSerapis/tool-eval-bench.git`; the receipt is rev-pinned at `6be685f0e` (v2.6.1.dev65, the campaign baseline since 2026-09-11 — the 167/176 record run is on it). The 19 dev46→dev65 commits (09-06..09-08) were **scoring-credit fixes** (correct behavior was being failed), so dev46 baselines (all GLM lanes: r28 158, r31 159, sparkring) read slightly low next to dev65 runs; cross-harness deltas include that bias. Reports stamp the harness version in the header — **A/B runs must share a harness version**. To move the pin deliberately: `uv tool install --reinstall --force "tool-eval-bench[tui,perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<rev>"` then re-baseline the anchor lane.
|
|
|
+- **Bulk data moves node-to-node, never through the Mac** — the Mac→node link
|
|
|
+ is ~32 MB/s (a 200 GB transfer via the Mac takes hours); node-to-node over the
|
|
|
+ 200G fabric does ~547 MB/s per stream. Node-to-node SSH works (the shared
|
|
|
+ key is on every node; `ssh kilork@<200.x-ib-ip>` from a node; dgx4 has two
|
|
|
+ RoCE IPs — 200.19 *and* 200.20 — use the `ib_ip` from config). Pattern for a
|
|
|
+ large HF download: one `hf download` on the roomiest node (dgx4, 1.9 TB),
|
|
|
+ then each target PULLS: `nohup bash -c "ssh <node> 'tar -hcf - -C <modeldir> snapshots trees refs' | tar xf - -C <modeldir>" &` (`-h` dereferences so targets don't need the hub blob layout). `hf download` on a node = `~/.local/bin/hf` (dgx2) or `uv tool install huggingface_hub` (others).
|
|
|
+- **huggingface_hub ≥0.34 (the `hf` CLI) moved blobs to the hub level** —
|
|
|
+ `hub/blobs/<xx>/<sha>` shared across models, per-model dirs are symlink trees
|
|
|
+ (snapshot → per-model blob → hub blob). `du`/`find -type f` on a model dir
|
|
|
+ only sees the small real files (~19 GB of the 199 GB) — judge completeness
|
|
|
+ by `df` deltas or hub-level `du`. The 09-20 re-cut download migrated dgx4's
|
|
|
+ cache layout this way (the Spark model there is hub-blob-backed now).
|
|
|
+- **`pkill -f` over ssh self-matches** — the pattern also matches the remote
|
|
|
+ shell's own `-c` command line, so `pkill -f "hf download"` kills the script
|
|
|
+ itself mid-run (and a `nohup … hf download …` line in the same command makes
|
|
|
+ the later pkill match again). Use the `[b]racket` trick for the pattern and
|
|
|
+ run kill and launch in separate ssh calls.
|
