|
|
@@ -0,0 +1,97 @@
|
|
|
+# TensorFold 1.0.12 (Zig) — 4-node bring-up, 2026-10-21/22
|
|
|
+
|
|
|
+Goal: test the operator's thesis that our TensorFold numbers (89/100, 6.7 s turn
|
|
|
+before the session cache) were an artifact of **our pinned engine**, not of TF.
|
|
|
+Our recipe pins TF **0.6.0** (`x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark`); the
|
|
|
+public family has since moved to **1.0.12 / 1.2.4** with a native Zig engine.
|
|
|
+
|
|
|
+Target chosen: `bertholomus/deepseek-v4.1-tensorfold-tp4-4xgb10` — an exact
|
|
|
+4-node TP4 engine (`deepseek-v41-zig-tp4` @ `51726210`), 1M context, 16 streams
|
|
|
+over a shared 4.19M-position window, kept-prompt reuse, exact DSpark, vision,
|
|
|
+grammar/tool calls, EXL3 **4.0bpw** weights (288 GB).
|
|
|
+
|
|
|
+## What worked
|
|
|
+
|
|
|
+- **Engine build** — `zig build install native -Dtarget=aarch64-linux-gnu.2.28
|
|
|
+ -Doptimize=ReleaseFast -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121` inside our
|
|
|
+ `dsv41-tf-1017:v1` image (it *is* `nvcr.io/nvidia/pytorch:26.07-py3` + our
|
|
|
+ TF layers) with Zig 0.17.0. Clean build, ~2 min.
|
|
|
+- **Kit** — the git kit is complete (357 files) but the four RoPE tables and
|
|
|
+ `vision/gate_bias_vl.safetensors` are shipped only in the HF repo
|
|
|
+ (`bertholomus/DeepSeek-V4.1-Flash-TensorFold-TP4-4xGB10`, `kit/…`).
|
|
|
+ `kit/verify_kit.sh` then reports **362 of 362 files match MANIFEST**. The HF
|
|
|
+ kit is byte-identical to git's for `aot/aot.json` + `MANIFEST`.
|
|
|
+- **Weights** — `bertholomus/DeepSeek-V4.1-Flash-1017-EXL3-4.0bpw@a2f91e27`
|
|
|
+ (288 GB) on the NAS, plus `tools/serving_1017.py` (adds
|
|
|
+ `serving-1017.safetensors` with the MTP markov head and the F32 engram q/k,
|
|
|
+ plus `chat_template.jinja`).
|
|
|
+- **Per-rank cache** — `node_inputs.sh` → ~75.6 GiB `rank<g>of4-<hash>.bin` per
|
|
|
+ node (~20 min each, four in parallel) and `dsv41_token_map.json`
|
|
|
+ (129280 → 99092 ids). `tf-dsv41-check` reports **189,610 entries, 0
|
|
|
+ mismatches, 0 layout mismatches**.
|
|
|
+
|
|
|
+## The blocker
|
|
|
+
|
|
|
+`serve.sh start` gets as far as reading the rank cache (67.3 GiB in 13 s) and
|
|
|
+then dies:
|
|
|
+
|
|
|
+```
|
|
|
+tensorfold: the native CUDA engine cannot load /model with /kit/aot glue kernels (FileNotFound)
|
|
|
+```
|
|
|
+
|
|
|
+- With `TENSORFOLD_CUDA_KERNELS=native` the same error says "with **its own**
|
|
|
+ glue kernels" — so the missing file is on the model side, not the kit.
|
|
|
+- `tf-dsv41-load /model 0 4 --cache /cache/rank-cache` → `error: MissingEntry`.
|
|
|
+- The model directory is complete: 41/41 shards from the index, `config.json`,
|
|
|
+ `tokenizer.json`, `quantization_config.json`, `serving-1017.safetensors`.
|
|
|
+
|
|
|
+So the 4.0bpw checkpoint + engine at that commit disagree on **some entry** the
|
|
|
+loader requires (not a missing shard, not a kit file). Worth reporting upstream
|
|
|
+with the exact inputs above.
|
|
|
+
|
|
|
+## Deviations we had to make (keep if the lane is retried)
|
|
|
+
|
|
|
+1. **HF symlink trees break bind mounts.** `hf download` (hub ≥ 0.34) makes
|
|
|
+ snapshot files symlinks into `../../blobs/…`. Mounting the *snapshot* alone
|
|
|
+ leaves them dangling (`config.json` missing) — and the same applies to the
|
|
|
+ Engram source. Fix: flat directories of **hardlinks** (`ln -L`), zero extra
|
|
|
+ space: `/mnt/nas/ai/tf4/model` (288 GB) and `/mnt/nas/ai/tf4/engram`
|
|
|
+ (shards 47/48, 190 GB).
|
|
|
+2. **The image's entrypoint must be cleared.** `scripts/node_run.sh` does
|
|
|
+ `docker run … IMAGE bash -c "…"`; our image's entrypoint is
|
|
|
+ `python -m tensorfold`, so it ran `python -m tensorfold bash -c …` and died
|
|
|
+ instantly. `--entrypoint ""` (not `--entrypoint bash`, which makes the extra
|
|
|
+ `bash` a script filename) fixes it.
|
|
|
+3. **`node_inputs.sh`'s token map silently failed**: the host python has no
|
|
|
+ `tokenizers`, and the in-image fallback used the entrypoint-less invocation
|
|
|
+ above; it also needs `HOME=/tmp` with `--user`. Run it by hand:
|
|
|
+ `docker run --rm --network none -e HOME=/tmp -v MODEL:/model:ro -v CACHE:/cache --entrypoint python3 IMAGE /tools/make_token_map.py /model /cache/dsv41_token_map.json`
|
|
|
+4. **All four nodes need the image** — the rank-0-only copy left dgx6 without it
|
|
|
+ (`node_run.sh` runs it on every node).
|
|
|
+
|
|
|
+## Where the artifacts live now
|
|
|
+
|
|
|
+| what | where |
|
|
|
+|---|---|
|
|
|
+| recipe + engine sources | dgx5 `~/tf-tp4`, `~/tf-tp4/engine` |
|
|
|
+| engine binaries (17) | every node `~/tf4/engine` |
|
|
|
+| kit (verified, + RoPE/vision) | every node `~/tf4/kit` |
|
|
|
+| model (hardlink flat) | NAS `/mnt/nas/ai/tf4/model` |
|
|
|
+| Engram shards 47/48 | NAS `/mnt/nas/ai/tf4/engram` |
|
|
|
+| rank-cache + token map | every node `~/tf4/cache` |
|
|
|
+| site config | dgx5 `~/tf-tp4/config.env` (4.0bpw TP4, RoCE 2 rails) |
|
|
|
+| logs | `~/tf4/log/{tp4-lane,serve-start*}.log` |
|
|
|
+
|
|
|
+## Next attempt (if taken up)
|
|
|
+
|
|
|
+1. Ask upstream (or bisect) the `MissingEntry`: run `tf-dsv41-load` with
|
|
|
+ `--digests out.jsonl` to see *which* entry is missing, or try the 2.9bpw
|
|
|
+ MiaAI checkpoint (the family's older, widely used weights) on the same
|
|
|
+ engine.
|
|
|
+2. Alternatively build from branch `deepseek-v41-zig-tp2` (the 2x family,
|
|
|
+ `MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks`) — 196 GB weights, more
|
|
|
+ users, and `sehoon123`'s wide-copy-window patches on top. Two nodes only,
|
|
|
+ which also matches what we have left.
|
|
|
+3. Keep the measurement plan ready: pp2048, tg c1/c2, kept-prompt reuse
|
|
|
+ (second question over the same prefix), needle at 128K/512K, then
|
|
|
+ tool-eval-bench v2.8.0 for the quality number.
|
