# TensorFold 1.0.12 (Zig) — 4-node bring-up, 2026-10-21/22

Goal: test the operator's thesis that our TensorFold numbers (89/100, 6.7 s turn
before the session cache) were an artifact of **our pinned engine**, not of TF.
Our recipe pins TF **0.6.0** (`x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark`); the
public family has since moved to **1.0.12 / 1.2.4** with a native Zig engine.

Target chosen: `bertholomus/deepseek-v4.1-tensorfold-tp4-4xgb10` — an exact
4-node TP4 engine (`deepseek-v41-zig-tp4` @ `51726210`), 1M context, 16 streams
over a shared 4.19M-position window, kept-prompt reuse, exact DSpark, vision,
grammar/tool calls, EXL3 **4.0bpw** weights (288 GB).

## What worked

- **Engine build** — `zig build install native -Dtarget=aarch64-linux-gnu.2.28
  -Doptimize=ReleaseFast -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121` inside our
  `dsv41-tf-1017:v1` image (it *is* `nvcr.io/nvidia/pytorch:26.07-py3` + our
  TF layers) with Zig 0.17.0. Clean build, ~2 min.
- **Kit** — the git kit is complete (357 files) but the four RoPE tables and
  `vision/gate_bias_vl.safetensors` are shipped only in the HF repo
  (`bertholomus/DeepSeek-V4.1-Flash-TensorFold-TP4-4xGB10`, `kit/…`).
  `kit/verify_kit.sh` then reports **362 of 362 files match MANIFEST**. The HF
  kit is byte-identical to git's for `aot/aot.json` + `MANIFEST`.
- **Weights** — `bertholomus/DeepSeek-V4.1-Flash-1017-EXL3-4.0bpw@a2f91e27`
  (288 GB) on the NAS, plus `tools/serving_1017.py` (adds
  `serving-1017.safetensors` with the MTP markov head and the F32 engram q/k,
  plus `chat_template.jinja`).
- **Per-rank cache** — `node_inputs.sh` → ~75.6 GiB `rank<g>of4-<hash>.bin` per
  node (~20 min each, four in parallel) and `dsv41_token_map.json`
  (129280 → 99092 ids). `tf-dsv41-check` reports **189,610 entries, 0
  mismatches, 0 layout mismatches**.

## The blocker

`serve.sh start` gets as far as reading the rank cache (67.3 GiB in 13 s) and
then dies:

```
tensorfold: the native CUDA engine cannot load /model with /kit/aot glue kernels (FileNotFound)
```

- With `TENSORFOLD_CUDA_KERNELS=native` the same error says "with **its own**
  glue kernels" — so the missing file is on the model side, not the kit.
- `tf-dsv41-load /model 0 4 --cache /cache/rank-cache` → `error: MissingEntry`.
- The model directory is complete: 41/41 shards from the index, `config.json`,
  `tokenizer.json`, `quantization_config.json`, `serving-1017.safetensors`.

So the 4.0bpw checkpoint + engine at that commit disagree on **some entry** the
loader requires (not a missing shard, not a kit file). Worth reporting upstream
with the exact inputs above.

## Deviations we had to make (keep if the lane is retried)

1. **HF symlink trees break bind mounts.** `hf download` (hub ≥ 0.34) makes
   snapshot files symlinks into `../../blobs/…`. Mounting the *snapshot* alone
   leaves them dangling (`config.json` missing) — and the same applies to the
   Engram source. Fix: flat directories of **hardlinks** (`ln -L`), zero extra
   space: `/mnt/nas/ai/tf4/model` (288 GB) and `/mnt/nas/ai/tf4/engram`
   (shards 47/48, 190 GB).
2. **The image's entrypoint must be cleared.** `scripts/node_run.sh` does
   `docker run … IMAGE bash -c "…"`; our image's entrypoint is
   `python -m tensorfold`, so it ran `python -m tensorfold bash -c …` and died
   instantly. `--entrypoint ""` (not `--entrypoint bash`, which makes the extra
   `bash` a script filename) fixes it.
3. **`node_inputs.sh`'s token map silently failed**: the host python has no
   `tokenizers`, and the in-image fallback used the entrypoint-less invocation
   above; it also needs `HOME=/tmp` with `--user`. Run it by hand:
   `docker run --rm --network none -e HOME=/tmp -v MODEL:/model:ro -v CACHE:/cache --entrypoint python3 IMAGE /tools/make_token_map.py /model /cache/dsv41_token_map.json`
4. **All four nodes need the image** — the rank-0-only copy left dgx6 without it
   (`node_run.sh` runs it on every node).

## Where the artifacts live now

| what | where |
|---|---|
| recipe + engine sources | dgx5 `~/tf-tp4`, `~/tf-tp4/engine` |
| engine binaries (17) | every node `~/tf4/engine` |
| kit (verified, + RoPE/vision) | every node `~/tf4/kit` |
| model (hardlink flat) | NAS `/mnt/nas/ai/tf4/model` |
| Engram shards 47/48 | NAS `/mnt/nas/ai/tf4/engram` |
| rank-cache + token map | every node `~/tf4/cache` |
| site config | dgx5 `~/tf-tp4/config.env` (4.0bpw TP4, RoCE 2 rails) |
| logs | `~/tf4/log/{tp4-lane,serve-start*}.log` |

## Next attempt (if taken up)

1. Ask upstream (or bisect) the `MissingEntry`: run `tf-dsv41-load` with
   `--digests out.jsonl` to see *which* entry is missing, or try the 2.9bpw
   MiaAI checkpoint (the family's older, widely used weights) on the same
   engine.
2. Alternatively build from branch `deepseek-v41-zig-tp2` (the 2x family,
   `MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks`) — 196 GB weights, more
   users, and `sehoon123`'s wide-copy-window patches on top. Two nodes only,
   which also matches what we have left.
3. Keep the measurement plan ready: pp2048, tg c1/c2, kept-prompt reuse
   (second question over the same prefix), needle at 128K/512K, then
   tool-eval-bench v2.8.0 for the quality number.