|
|
@@ -65,6 +65,57 @@ into `recipes/<lane>/reports/`, prints the final score + perf rows.
|
|
|
(`--served-model-name` in args / `port`). The same commit moves the `stop`
|
|
|
handler from `main.rs` to `docker/stop.rs` (line-budget).
|
|
|
|
|
|
+## r28 full-stack evaluation + version profiles (evening shift)
|
|
|
+
|
|
|
+Operator request: a `build_profile` entity — a named pin set referenced from
|
|
|
+recipes instead of copy-pasting the whole `[build.versions]` table. The
|
|
|
+half-cut r28 recipe (vLLM advanced, b12x/LMCache left at r27) was exactly the
|
|
|
+failure mode this kills.
|
|
|
+
|
|
|
+- Feature (15e9b62): `build_profile = "<name>"` → `docker-images/<dir>/
|
|
|
+ profiles/<name>.env` (same KEY=VAL format); merge base < profile < recipe
|
|
|
+ `[build.versions]`; merged file written to the node before `build.sh` and
|
|
|
+ folded into the `sm121.checksum` label. `profiles/glm53-r28.env` carries
|
|
|
+ the full r28 lock (vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a —
|
|
|
+ all three forks cut simultaneously 09-06, strictly ahead of r27 pins).
|
|
|
+- **TOML context trap (b5f0662)**: the first `build_profile` edit landed
|
|
|
+ after the `[[extra_mounts]]` block → the bare key parsed into the LAST
|
|
|
+ mount table and serde silently ignored it; the build ran with base pins
|
|
|
+ while the recipe "had" the override. Bare keys must stay top-level
|
|
|
+ (tables like `[build.versions]` reset the context and are safe anywhere).
|
|
|
+ Verified end-to-end with a live `load_recipe` + `build_checksum` probe
|
|
|
+ before re-running the build; pitfall documented in both skills.
|
|
|
+- Build: ~6 min (buildx cache), label 14271940…; image verified in-container
|
|
|
+ (vllm r28, lmcache 934c287a, b12x 1.3.0 — the r28 b12x moved to static
|
|
|
+ versioning, so the scm pretend no longer applies; pin still enforced by the
|
|
|
+ HEAD+tree gates).
|
|
|
+- **Memory gate flake**: first serve died at worker init —
|
|
|
+ `Free memory 107.86 < 0.9×121.63 = 109.46 GiB` (the `gpu_memory_utilization`
|
|
|
+ admission gate, vLLM #48140): docker load of the 16 GB image + ~12 GiB of
|
|
|
+ starting vLLM processes pushed the check under the line. The launcher's
|
|
|
+ existing `drop_caches` (post-sync, pre-container, default on) ran but the
|
|
|
+ margin is still razor-thin; a retry with settled cache booted fine.
|
|
|
+ Follow-up considered: GMU 0.88 in the 4x recipes (gate → 107.0 GiB; KV is
|
|
|
+ explicitly pinned to 16 GiB so capacity is unchanged).
|
|
|
+- **Bench (full r28, seed 42, hardmode, c1,2): 155/176 (88/100)** vs r27
|
|
|
+ RoCE 160/176 and mixed-stack r28 159/176; median turn 3.3 s (r27 3.9 s —
|
|
|
+ the r28 scheduler win holds). 8 hard scenarios lost points vs r27 (TC-21/
|
|
|
+ 49/51/61/67/80/82/88); TC-80+TC-88 failed only on full r28. But: r27 itself
|
|
|
+ spread 155–162 across its A/B runs, and the r28 b12x M8/M1 changes are
|
|
|
+ layout-only ("no model math, scale precision, or output reduction is
|
|
|
+ changed"), so 155 is inside the observed run-to-run band. Re-run queued to
|
|
|
+ pin the variance before judging the stack.
|
|
|
+
|
|
|
+Pending: (1) b12x O_DIRECT loader test (`--load-format b12x`, recipe
|
|
|
+`glm53-flash-r28-4x-dcp4-b12xload` ready) — the r28 b12x added
|
|
|
+`B12xModelLoader` (O_DIRECT reads into CUDA managed memory; weight loads stop
|
|
|
+polluting the page cache — relevant to the memory gate above); (2) Discord
|
|
|
+env vars `VLLM_GLM53_SPLIT_TARGET_BLOCK_SIZE=2048` /
|
|
|
+`VLLM_GLM53_SPLIT_MAMBA_BLOCK_SIZE=256` — split GLM-5.3 KV into independent
|
|
|
+target-MLA (2048) + recurrent-Mamba (256) page pools, requires
|
|
|
+`mamba_cache_mode='align'`, fine-grained prefix reuse; research done in the
|
|
|
+fork source (`vllm/platforms/interface.py:811`), test pending.
|
|
|
+
|
|
|
## Ops notes / lessons
|
|
|
|
|
|
- Node clocks on the Sparks run +2 h ahead of UTC; the launcher's own INFO
|
