|
|
@@ -235,3 +235,20 @@ pre-run image presence check (or node-to-node sync) + honest error.
|
|
|
### Config fix
|
|
|
`config.toml` 1node-4 node entity was `switch.dgx3` (copy-paste) → fixed to
|
|
|
`switch.dgx4` (the 1node-3 node keeps `switch.dgx3`).
|
|
|
+
|
|
|
+### EXL3 2x — ELOOP root cause + fix (serving)
|
|
|
+Root cause (after ~10 red herrings: NFS dentry races, concurrency, cgroups,
|
|
|
+launcher-flag diffs — all disproven by flag-for-flag replicas): the DS41
|
|
|
+runtime opens the shard files with **O_NOFOLLOW** (`ssd_vocab_rows.py:68`
|
|
|
+`os.open(self.path, O_RDONLY|O_CLOEXEC|O_NOFOLLOW)`). With symlink-based
|
|
|
+staging, the final path component IS a symlink → open(2) returns ELOOP
|
|
|
+("Too many levels of symbolic links") **deterministically** — the kernel's
|
|
|
+ELOOP for O_NOFOLLOW, not a loop at all.
|
|
|
+Fix: no symlinks in the container. New recipe field
|
|
|
+`extra_mounts_manifest = "/tmp/sm121-exl3-mounts.txt"`: the before-serve hook
|
|
|
+generates a per-node `host<TAB>container` list (one **bind per real file**,
|
|
|
+161 model + 10 draft entries, resolved to the final hub blob), the launcher
|
|
|
+reads it over SSH after the hooks and appends it as read-only binds. The
|
|
|
+container sees regular files only → O_NOFOLLOW is satisfied.
|
|
|
+Result: EXL3 2x boots (shards stream from the NAS, ~15 min), API live on
|
|
|
+dgx1:30000 as `deepseek-v41-flash-exl3`, smoke 127×89 + Paris correct.
|
