specs: TensorFold 1.0.12 4-node bring-up — engine+kit+weights+rank-cache all verified, blocked on a loader MissingEntry (documented with the exact inputs and our deviations)

specs/tech-architecture/tensorfold-1.0.12-bringup.md

@@ -0,0 +1,97 @@
+# TensorFold 1.0.12 (Zig) — 4-node bring-up, 2026-10-21/22
+
+Goal: test the operator's thesis that our TensorFold numbers (89/100, 6.7 s turn
+before the session cache) were an artifact of **our pinned engine**, not of TF.
+Our recipe pins TF **0.6.0** (`x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark`); the
+public family has since moved to **1.0.12 / 1.2.4** with a native Zig engine.
+
+Target chosen: `bertholomus/deepseek-v4.1-tensorfold-tp4-4xgb10` — an exact
+4-node TP4 engine (`deepseek-v41-zig-tp4` @ `51726210`), 1M context, 16 streams
+over a shared 4.19M-position window, kept-prompt reuse, exact DSpark, vision,
+grammar/tool calls, EXL3 **4.0bpw** weights (288 GB).
+
+## What worked
+
+- **Engine build** — `zig build install native -Dtarget=aarch64-linux-gnu.2.28
+  -Doptimize=ReleaseFast -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121` inside our
+  `dsv41-tf-1017:v1` image (it *is* `nvcr.io/nvidia/pytorch:26.07-py3` + our
+  TF layers) with Zig 0.17.0. Clean build, ~2 min.
+- **Kit** — the git kit is complete (357 files) but the four RoPE tables and
+  `vision/gate_bias_vl.safetensors` are shipped only in the HF repo
+  (`bertholomus/DeepSeek-V4.1-Flash-TensorFold-TP4-4xGB10`, `kit/…`).
+  `kit/verify_kit.sh` then reports **362 of 362 files match MANIFEST**. The HF
+  kit is byte-identical to git's for `aot/aot.json` + `MANIFEST`.
+- **Weights** — `bertholomus/DeepSeek-V4.1-Flash-1017-EXL3-4.0bpw@a2f91e27`
+  (288 GB) on the NAS, plus `tools/serving_1017.py` (adds
+  `serving-1017.safetensors` with the MTP markov head and the F32 engram q/k,
+  plus `chat_template.jinja`).
+- **Per-rank cache** — `node_inputs.sh` → ~75.6 GiB `rank<g>of4-<hash>.bin` per
+  node (~20 min each, four in parallel) and `dsv41_token_map.json`
+  (129280 → 99092 ids). `tf-dsv41-check` reports **189,610 entries, 0
+  mismatches, 0 layout mismatches**.
+
+## The blocker
+
+`serve.sh start` gets as far as reading the rank cache (67.3 GiB in 13 s) and
+then dies:
+
+```
+tensorfold: the native CUDA engine cannot load /model with /kit/aot glue kernels (FileNotFound)
+```
+
+- With `TENSORFOLD_CUDA_KERNELS=native` the same error says "with **its own**
+  glue kernels" — so the missing file is on the model side, not the kit.
+- `tf-dsv41-load /model 0 4 --cache /cache/rank-cache` → `error: MissingEntry`.
+- The model directory is complete: 41/41 shards from the index, `config.json`,
+  `tokenizer.json`, `quantization_config.json`, `serving-1017.safetensors`.
+
+So the 4.0bpw checkpoint + engine at that commit disagree on **some entry** the
+loader requires (not a missing shard, not a kit file). Worth reporting upstream
+with the exact inputs above.
+
+## Deviations we had to make (keep if the lane is retried)
+
+1. **HF symlink trees break bind mounts.** `hf download` (hub ≥ 0.34) makes
+   snapshot files symlinks into `../../blobs/…`. Mounting the *snapshot* alone
+   leaves them dangling (`config.json` missing) — and the same applies to the
+   Engram source. Fix: flat directories of **hardlinks** (`ln -L`), zero extra
+   space: `/mnt/nas/ai/tf4/model` (288 GB) and `/mnt/nas/ai/tf4/engram`
+   (shards 47/48, 190 GB).
+2. **The image's entrypoint must be cleared.** `scripts/node_run.sh` does
+   `docker run … IMAGE bash -c "…"`; our image's entrypoint is
+   `python -m tensorfold`, so it ran `python -m tensorfold bash -c …` and died
+   instantly. `--entrypoint ""` (not `--entrypoint bash`, which makes the extra
+   `bash` a script filename) fixes it.
+3. **`node_inputs.sh`'s token map silently failed**: the host python has no
+   `tokenizers`, and the in-image fallback used the entrypoint-less invocation
+   above; it also needs `HOME=/tmp` with `--user`. Run it by hand:
+   `docker run --rm --network none -e HOME=/tmp -v MODEL:/model:ro -v CACHE:/cache --entrypoint python3 IMAGE /tools/make_token_map.py /model /cache/dsv41_token_map.json`
+4. **All four nodes need the image** — the rank-0-only copy left dgx6 without it
+   (`node_run.sh` runs it on every node).
+
+## Where the artifacts live now
+
+| what | where |
+|---|---|
+| recipe + engine sources | dgx5 `~/tf-tp4`, `~/tf-tp4/engine` |
+| engine binaries (17) | every node `~/tf4/engine` |
+| kit (verified, + RoPE/vision) | every node `~/tf4/kit` |
+| model (hardlink flat) | NAS `/mnt/nas/ai/tf4/model` |
+| Engram shards 47/48 | NAS `/mnt/nas/ai/tf4/engram` |
+| rank-cache + token map | every node `~/tf4/cache` |
+| site config | dgx5 `~/tf-tp4/config.env` (4.0bpw TP4, RoCE 2 rails) |
+| logs | `~/tf4/log/{tp4-lane,serve-start*}.log` |
+
+## Next attempt (if taken up)
+
+1. Ask upstream (or bisect) the `MissingEntry`: run `tf-dsv41-load` with
+   `--digests out.jsonl` to see *which* entry is missing, or try the 2.9bpw
+   MiaAI checkpoint (the family's older, widely used weights) on the same
+   engine.
+2. Alternatively build from branch `deepseek-v41-zig-tp2` (the 2x family,
+   `MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks`) — 196 GB weights, more
+   users, and `sehoon123`'s wide-copy-window patches on top. Two nodes only,
+   which also matches what we have left.
+3. Keep the measurement plan ready: pp2048, tg c1/c2, kept-prompt reuse
+   (second question over the same prefix), needle at 128K/512K, then
+   tool-eval-bench v2.8.0 for the quality number.