|
|
@@ -293,3 +293,34 @@ model-side, not lane-side, same families as other DS4.1 lanes.
|
|
|
|
|
|
Tooling: `bench --tokenizer` added (EXL3's served id isn't resolvable
|
|
|
offline in the local HF cache); tokenizer pulled from the lane mirror.
|
|
|
+
|
|
|
+## aiden16 cutover — localized Engram + the liveness bug the operator smelled
|
|
|
+
|
|
|
+EXL3 benched (92/100, quality-neutral vs manual staging) and stopped;
|
|
|
+aiden16 serves `4nodes-switch` with the Engram rows off the NAS for the
|
|
|
+first time. Recipe migration: `[[staging]]` sparse-copy → `/home/kilork/
|
|
|
+aiden-engram`, ro-mounted at `/engram-local`, `SPARK_ENGRAM_DIR` repointed.
|
|
|
+
|
|
|
+Two real bugs found on live cutover, both fixed:
|
|
|
+1. **sparse-copy had no safetensors header** — stage_sparse_layer copied
|
|
|
+ only the engram row ranges; the aiden overlay parses the header first
|
|
|
+ and died with "safetensors header is not JSON". Header is now copied
|
|
|
+ once per shard (repair fast path: header-only fix on existing trees,
|
|
|
+ no range recopy — dgx1/2 healed in seconds).
|
|
|
+2. **row-tiling mismatch** — our sparse-copy used `rows*rank//tp`, the
|
|
|
+ aiden wrapper tiles `base+rem` (they differ by one row when
|
|
|
+ rows%tp!=0, layer14 here), and the overlay cross-checks
|
|
|
+ manifest==SPARK_ENGRAM_ROWS. Formula aligned (fixture 1006 rows
|
|
|
+ verified == wrapper).
|
|
|
+
|
|
|
+And the liveness bug the operator suspected: `watch()` in readiness.rs
|
|
|
+counted the API-not-yet-open window (weight load, 17+ min) as death and
|
|
|
+bailed with "containers died mid-follow" while all four containers were
|
|
|
+Up. Watch now ignores connection-refused until the head API has answered
|
|
|
+(or listens) once. Fix `6ae3c51` — the same class as the earlier
|
|
|
+"serve exited but Docker is fine" observation.
|
|
|
+
|
|
|
+Live state: sglang healthy, model `deepseek-v4.1-flash`, spot: ' Paris.'
|
|
|
+clean, 300-token generation ≈ 75 t/s decode c1 (Aiden's band is
|
|
|
+107/87/36; the bench will tell). Re-bench of this lane on localized
|
|
|
+engram rows pending.
|
