dsv41-flash-1017-2x-tensorfold: measure the upstream TensorFold dual-Spark lane — 89/100 quality (best 1017 lane) but responsiveness 23 (6.7s turn) vs the 4x SGLang lane's 87/75; decode +20-30%

Brought up on 2nodes-dac (dgx1 head + dgx2 worker) with LibertAIDAI/...-EXL3-2.9bpw
on node-local disk, Engram packed per rank (94.4 GiB/node), TensorFold 0.6.0 +
Jay Leaton patches baked into a 24.5 GB image. NOTE.md carries the full bring-up
recipe (incl. the hfrs/python xet failure on the NAS mount and the chat-template
requirement). Not an sm121 recipe yet.

recipes/dsv41-flash-1017-2x-tensorfold/NOTE.md

@@ -0,0 +1,67 @@
+# DeepSeek-V4.1-Flash-1017 — TensorFold 2x (dual Spark) experimental lane
+
+Status: **measured 2026-10-19, not yet an sm121 recipe** (no `recipe.toml`; the
+lane was brought up with the upstream launcher — see below). Reports live here so
+the numbers sit next to the other 1017 lanes.
+
+Stack (upstream `x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark`, forum thread
+386121): TensorFold 0.6.0 (Jay Leaton's DeepSeek-family patches + the repo's
+`tensorfold-1017.patch`) on `nvcr.io/nvidia/pytorch:26.07-py3`, TP2 over the
+direct DAC link (cluster `2nodes-dac`: dgx1 head + dgx2 worker), weights
+`LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw` (210.8 GB, node-local on both
+nodes), Engram tables packed per rank from the official 1017 shards 47/48
+(94.4 GiB a node), `CONTEXT=262144`, `PARALLEL=8`, thinking on / effort 75
+(upstream default).
+
+## Measured (2026-10-18/19)
+
+| metric | TensorFold 2x | 4x SGLang (EP2/1M, our lane) |
+|---|---|---|
+| quality (tool-eval-bench v2.8.0, seed 42, hardmode) | **89/100** (172/194) | 87/100 (168/194) |
+| responsiveness / median turn | **23 / 6.7 s** | **75 / 1.4 s** |
+| deployability | 69 | 83 |
+| decode c1 (probe, median of 5 cold 2K passes) | **50.4 tok/s** | 38.8-43.0 tok/s |
+| prefill (probe TTFT for the same ~2K prompt) | 5.4 s (~380 tok/s) | 0.75 s (~2,700 tok/s) |
+| nodes / GPUs | 2 | 4 |
+
+## Reading
+
+- **Quality is at least as good** (89 vs 87 — one scenario apart at single-trial
+  noise) and it does it on half the cluster.
+- **Decode per GPU is much better** (+20-30% against a 4-node SGLang lane that
+  already uses twice the hardware).
+- **Interactive latency is 4-5x worse**: prefill is ~7x slower, so an agent turn
+  that re-prefills a working context pays seconds. The harness scores that as
+  responsiveness 23 vs 75.
+
+Verdict for our fleet: TensorFold 2x is the right shape for **long-generation /
+batch work on a small cluster**; the 4x SGLang lane stays the interactive agent
+profile. Note the two are not purity-matched: TensorFold runs thinking-on at
+effort 75 (upstream default) and EXL3 2.9bpw weights, SGLang runs fp8 with
+effort 100.
+
+## How it was brought up (for the eventual recipe)
+
+1. Weights: `hf download LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw`
+   (xet **disabled** — hfrs' xet batch path and the Python client's xet path both
+   fail writing into the NAS HF cache with `I/O error: File exists`; plain HTTP
+   with `HF_HUB_DISABLE_XET=1` runs at ~270 MB/s) into the NAS hub cache, then a
+   node-local copy on each launch node (EXL3 streams weights every decode step —
+   NFS/NAS streaming would cost most of the decode).
+2. Engram: `build/jayleaton/scripts/pack_engram.py --src <1017 snapshot> --config
+   <checkpoint>/config.json --out <dst> --rank {0,1}` (~95 GiB a node, ~30 min,
+   reads the two 189 GiB source shards).
+3. `chat_template.jinja` must be copied into the checkpoint dir (the EXL3
+   checkpoint has none; the TF HTTP app requires one at init) — it ships in the
+   upstream repo at `build/mia-runtime/files/chat_template.jinja`.
+4. Image: `scripts/fetch-sources.sh` (pins MiaAI-Lab @6f7d1590 + jayleaton
+   @2dcce030 + vendor/TensorFold @c4646171) then `docker build -t
+   tensorfold-1017:<ver> .` (24.5 GB).
+5. Prebuild the CUDA extensions CPU-only on **both** nodes before first start
+   (18/18 builders, ~3 min; `prebuild_ext.py` + `prebuild-offline.py`).
+6. `scripts/cluster.py start` refuses to run without `models/verified.json`
+   (produced by their `download.py`); we started the two `docker run` lines from
+   `cluster.py args` directly after verifying the checkpoint file-for-file
+   against the Hub API (66/66 files, sizes match).
+
+Node cost on dgx1/dgx2: ~305 GB per node (210 GB weights + 94 GB engram + image).