|
|
@@ -212,6 +212,8 @@ Boot D INFEASIBLE on 4x (OOM incident 14:37 UTC, nodes recovered).
|
|
|
### P2 — 8x trials (cables on site; nodes provisioned, cluster configured)
|
|
|
Gate: CLEARED — all 8 nodes up + validated, `8nodes-switch` in config,
|
|
|
health green. The launcher syncs missing images at serve time.
|
|
|
+STATUS (18:28 UTC): P2.3 first (DS4.1 8x in-memory engram) — up, quality
|
|
|
+gate clean, bench `8x` running.
|
|
|
1. **8x full GLM-5.3 MXFP4 @512K DCP1**: TP8 → 47 GiB weights/node → KV
|
|
|
budget ≈ 45+ GiB/rank at GMU 0.85 → a 512K request costs 26.3 GiB/rank
|
|
|
(52.7 KB/token/rank × 512K) and fits **without DCP** — sidesteps the
|
|
|
@@ -221,15 +223,33 @@ health green. The launcher syncs missing images at serve time.
|
|
|
the 4x cluster cannot deliver.
|
|
|
2. **8x lab GLM-5.3-NVFP4 + MTP k3** (manifest policy): 433 GB → 54.1 GiB/
|
|
|
node; the lab's own qualified full-model config; A/B vs (1).
|
|
|
-3. **8x DS4.1-Flash — no disk staging** (operator decision, code-verified):
|
|
|
- at TP8 the two 203 GB Engram tables fit in DRAM (1/8 of the table ≈
|
|
|
- 51 GiB/rank + weights within 121.6 GiB/node), so the 8x recipe drops
|
|
|
- `DSV41_ENGRAM_DISK`/`DSV41_ENGRAM_LOCAL`/`DSV41_ENGRAM_DIR` and the
|
|
|
- `/var/tmp/engram-local` mount entirely — the baked `engram.py` patch's
|
|
|
- disk mode is env-gated and defaults to off (in-memory path). The engine
|
|
|
- prints the per-rank layout at boot (the recipe header documents it), so
|
|
|
- nothing needs pre-staging. Aggregate KV ~2× (10.4M → ~20M tokens) — the
|
|
|
- capacity/1M-context lane for prod traffic.
|
|
|
+3. [running] **8x DS4.1-Flash — in-memory engram** (operator decision, code-
|
|
|
+ verified). Recipe `dsv41-flash-8x`: drops `DSV41_ENGRAM_DISK`/`LOCAL`/
|
|
|
+ `DIR` + the `/engram-local` mount (the baked `engram.py` patch defaults to
|
|
|
+ memory mode). Boot log (first attempt 15:57, second 16:10 after fixes):
|
|
|
+ - **GMU 0.91 → 0.85**: the lab policy's 0.91 fails the GB10 admission
|
|
|
+ (free 108.78 < 110.7 required; ~12.8 GiB baseline reservation); 0.85
|
|
|
+ is the GB10-qualified value.
|
|
|
+ - **dgx6-8 cache symlink**: `/home/kilork/.cache/huggingface` was a local
|
|
|
+ dir there (docker -v auto-created it empty as root) — ranks 5-7 died
|
|
|
+ with LocalEntryNotFoundError; fixed with `rmdir + ln -s /mnt/nas/ai/
|
|
|
+ huggingface` (dgx1-5 have the symlink natively).
|
|
|
+ - **Load = 122 min** (r0, `Model loading took 49.55 GiB and 7338 s`):
|
|
|
+ memory mode does a full sequential scan of the two 101.5 GB sparse
|
|
|
+ shards per rank (~203 GB reads/rank) at ~20-40 MB/s under 8-way NFS
|
|
|
+ contention. One-off cost; the launcher's 1800 s readiness wait is
|
|
|
+ irrelevant for cold 8x loads (probe the API manually).
|
|
|
+ - Up 18:28 UTC: KV pool **2,584,421 tokens @512K (4.93x)**, engram
|
|
|
+ 2×11.80 GiB/rank pinned, DSpark K5 graphs captured, ~24.5 GiB KV
|
|
|
+ (matches the 103.4 − 38.4 − 25.4 − 12 math).
|
|
|
+ - Quality gate PASSED: RU 300-word text clean (no garbling — the prod
|
|
|
+ image + in-memory engram is clean, cf. the lab-tree garbling verdict
|
|
|
+ in P1), identifiers 6/6. This also settles the "Boot D" question
|
|
|
+ from the r37 bisect: engram without the disk-read path is clean on
|
|
|
+ this image.
|
|
|
+ - Bench `dev68-dsv41-8x` running vs 4x prod anchor 167/176 (95),
|
|
|
+ 1,642 pp / 47.0 tg / 1,738 ms @d0 c1 (expect prefill up ~2x on 8
|
|
|
+ nodes, decode level, TTFT lower).
|
|
|
Non-goals at 8x: DCP8 (the DCP>1 crash applies; only 131K-style DCP1 is
|
|
|
qualified).
|
|
|
|
