launcher/specs/tech-architecture/tensorfold-1.0.12-bringup.md

TensorFold 1.0.12 (Zig) — 4-node bring-up, 2026-10-21/22

Goal: test the operator's thesis that our TensorFold numbers (89/100, 6.7 s turn before the session cache) were an artifact of our pinned engine, not of TF. Our recipe pins TF 0.6.0 (x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark); the public family has since moved to 1.0.12 / 1.2.4 with a native Zig engine.

Target chosen: bertholomus/deepseek-v4.1-tensorfold-tp4-4xgb10 — an exact 4-node TP4 engine (deepseek-v41-zig-tp4 @ 51726210), 1M context, 16 streams over a shared 4.19M-position window, kept-prompt reuse, exact DSpark, vision, grammar/tool calls, EXL3 4.0bpw weights (288 GB).

What worked

• Engine build — zig build install native -Dtarget=aarch64-linux-gnu.2.28 -Doptimize=ReleaseFast -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121 inside our dsv41-tf-1017:v1 image (it is nvcr.io/nvidia/pytorch:26.07-py3 + our TF layers) with Zig 0.17.0. Clean build, ~2 min.
• Kit — the git kit is complete (357 files) but the four RoPE tables and vision/gate_bias_vl.safetensors are shipped only in the HF repo (bertholomus/DeepSeek-V4.1-Flash-TensorFold-TP4-4xGB10, kit/…). kit/verify_kit.sh then reports 362 of 362 files match MANIFEST. The HF kit is byte-identical to git's for aot/aot.json + MANIFEST.
• Weights — bertholomus/DeepSeek-V4.1-Flash-1017-EXL3-4.0bpw@a2f91e27 (288 GB) on the NAS, plus tools/serving_1017.py (adds serving-1017.safetensors with the MTP markov head and the F32 engram q/k, plus chat_template.jinja).
• Per-rank cache — node_inputs.sh → ~75.6 GiB rank<g>of4-<hash>.bin per node (~20 min each, four in parallel) and dsv41_token_map.json (129280 → 99092 ids). tf-dsv41-check reports 189,610 entries, 0 mismatches, 0 layout mismatches.

The blocker

serve.sh start gets as far as reading the rank cache (67.3 GiB in 13 s) and then dies:

tensorfold: the native CUDA engine cannot load /model with /kit/aot glue kernels (FileNotFound)

• With TENSORFOLD_CUDA_KERNELS=native the same error says "with its own glue kernels" — so the missing file is on the model side, not the kit.
• tf-dsv41-load /model 0 4 --cache /cache/rank-cache → error: MissingEntry.
• The model directory is complete: 41/41 shards from the index, config.json, tokenizer.json, quantization_config.json, serving-1017.safetensors.

So the 4.0bpw checkpoint + engine at that commit disagree on some entry the loader requires (not a missing shard, not a kit file). Worth reporting upstream with the exact inputs above.

Deviations we had to make (keep if the lane is retried)

1. HF symlink trees break bind mounts. hf download (hub ≥ 0.34) makes snapshot files symlinks into ../../blobs/…. Mounting the snapshot alone leaves them dangling (config.json missing) — and the same applies to the Engram source. Fix: flat directories of hardlinks (ln -L), zero extra space: /mnt/nas/ai/tf4/model (288 GB) and /mnt/nas/ai/tf4/engram (shards 47/48, 190 GB).
2. The image's entrypoint must be cleared. scripts/node_run.sh does docker run … IMAGE bash -c "…"; our image's entrypoint is python -m tensorfold, so it ran python -m tensorfold bash -c … and died instantly. --entrypoint "" (not --entrypoint bash, which makes the extra bash a script filename) fixes it.
3. node_inputs.sh's token map silently failed: the host python has no tokenizers, and the in-image fallback used the entrypoint-less invocation above; it also needs HOME=/tmp with --user. Run it by hand: docker run --rm --network none -e HOME=/tmp -v MODEL:/model:ro -v CACHE:/cache --entrypoint python3 IMAGE /tools/make_token_map.py /model /cache/dsv41_token_map.json
4. All four nodes need the image — the rank-0-only copy left dgx6 without it (node_run.sh runs it on every node).

Where the artifacts live now

what	where
recipe + engine sources	dgx5 ~/tf-tp4, ~/tf-tp4/engine
engine binaries (17)	every node ~/tf4/engine
kit (verified, + RoPE/vision)	every node ~/tf4/kit
model (hardlink flat)	NAS /mnt/nas/ai/tf4/model
Engram shards 47/48	NAS /mnt/nas/ai/tf4/engram
rank-cache + token map	every node ~/tf4/cache
site config	dgx5 ~/tf-tp4/config.env (4.0bpw TP4, RoCE 2 rails)
logs	~/tf4/log/{tp4-lane,serve-start*}.log

Next attempt (if taken up)

1. Ask upstream (or bisect) the MissingEntry: run tf-dsv41-load with --digests out.jsonl to see which entry is missing, or try the 2.9bpw MiaAI checkpoint (the family's older, widely used weights) on the same engine.
2. Alternatively build from branch deepseek-v41-zig-tp2 (the 2x family, MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks) — 196 GB weights, more users, and sehoon123's wide-copy-window patches on top. Two nodes only, which also matches what we have left.
3. Keep the measurement plan ready: pp2048, tg c1/c2, kept-prompt reuse (second question over the same prefix), needle at 128K/512K, then tool-eval-bench v2.8.0 for the quality number.