r36 dev65 + local Engram: prefill 3,654 pp / 826 ms TTFT (9.1x the NFS run, 2.2x prod) — storage suspect confirmed; 149/176 = variance + K7 stack gap; lane is now a candidate

history/night-shifts/2026-09-12.md

@@ -265,3 +265,50 @@ to separate storage from tuning); (2) wait for the lab's sm_121 port if
 they ever tune for Spark; (3) the built image `jovian-r36-ds41:local`
 (label 49b6...) + recipe `dsv41-flash-jovian-r36-4x` stay in the repo as a
 known-working (losing) lane.
+
+### Day follow-up 2 (09-13): Engram on local NVMe — the NFS suspect CONFIRMED
+
+Operator: "use engram equivalently from NVMe — NAS is a guaranteed loss."
+The eugr/aiden dsv41 lanes already stage per-rank sparse Engram copies on
+node NVMe (`/var/tmp/engram-local/DeepSeek-V4.1-Flash`, 48 GB real per node —
+two full-size 101.5 GB files, only the rank's ceil-row TP shard materialized,
+verified per-node row starts 0 / 96,000,564 / 192,001,740 / 288,003,654).
+The r36 engine's io_uring PLE reader reads (file, offset+rows) from the model
+dir — so it works unchanged against those files: same filename (the engine
+looks shards up by name via the index.json), per-node individual content.
+
+**Launcher feature: `composed_model.local_files`** (commit eb63d88) — the
+standard composed-dir mechanism extended: entries are HARDLINKED (not copied;
+sparse stays sparse, zero disk) from a per-node host path into the per-node
+/tmp composed dir, after the repo symlink pass (local wins at the same name).
+Recipe `dsv41-flash-jovian-r36-4x` now uses `[composed_model]`: whole DSV4.1
+snapshot symlinked into the HF cache + the two Engram shards hardlinked from
+the eugr lane's sparse copies. (A hand-rolled ~/models farm first failed in
+the container — host-side /mnt/nas symlink targets dangle inside the
+container; composed-dir symlinks use container-side paths by design, and
+hardlinks are path-independent — the failure is the feature validation.)
+
+**Result (4nodes-switch, dev65/seed 42, vs NFS run 158/176 and prod 167/176):**
+
+| | prod (eugr K5) | r36 + Engram NFS | **r36 + Engram NVMe** |
+|---|---|---|---|
+| score | 167/176 (95) | 158/176 (90) | **149/176 (85)** |
+| pp / tg / TTFT @d0 c1 | 1,642 / 47.0 / 1,738 | 401 / 33.7 / 5,061 | **3,654 / 43.9 / 826** |
+| pp / tg @d4096 c2 | — | 12.0 tg | 1,711 / 34.9 |
+| KV | 512K pin | 3.38M @131K | 3.34M @131K |
+
+**The storage hypothesis is CONFIRMED**: prefill 401 → 3,654 tok/s (9.1x,
+now 2.2x PROD), TTFT 5,061 → 826 ms (6.1x, 2.1x better than prod),
+d4096c2 decode 12.0 → 34.9 tg. The io_uring row reads over NFS were the
+entire prefill collapse — with the table on NVMe the r36 stack
+(K7 DSpark + B12X + V2 runner) is the fastest prefill lane we have.
+
+Quality: 149 vs 158 (NFS) is run variance, not storage — the 11 differing
+scenarios are tool-discipline wobbles (skipped steps, empty query, extra
+calls; two improved: TC-35, TC-50), plus 2 new safety warnings on
+long-horizon P scenarios (TC-82/85). The r36-vs-prod gap (−9…−18) is the
+engine-stack delta (K7 greedy+adaptive vs K5 probabilistic), present in both
+r36 runs. r36 + local Engram is now a legitimate CANDIDATE lane: best
+prefill/TTFT we have, −18 quality vs prod. Next levers (if pursued):
+r36 with our prod scheduler flags (512K/DCP4/block 128/seqs 8) to separate
+stack from tuning; or K7 speculative-config tuning.

recipes/dsv41-flash-jovian-r36-4x/recipe.toml

@@ -24,19 +24,25 @@
 #     image's native extension is sm_120 SASS — ours is built for GB10).
 #   - NO VLLM_ENABLE_PCIE_ALLREDUCE (theirs is single-host PCIe; on our
 #     4-node switch fabric the launcher's NCCL/RoCE env is the transport).
-#   - Engram disk mode on the shared NAS: our current dsv41 prod already
-#     runs this checkpoint with Engram-on-disk over the same NFS, so the
-#     storage path is proven; their liburing io_uring fast path is the new
-#     bit (falls through to normal reads if the kernel/NFS combo refuses).
+#   - Engram disk mode, table on node-local NVMe (see [composed_model]
+#     below): the NFS read path was the 4.1x prefill collapse (day
+#     follow-up 2); their 188.83 GiB RAM mode cannot fit a GB10 node
+#     (121.63 GiB unified pool) either way.
 #   - VLLM_USE_V2_MODEL_RUNNER/AOT/MEGA_AOT env from their serve script
 #     (their image bakes them; ours must carry them explicitly).
 #
-# A/B result (2026-09-13, 4nodes-switch, dev65/seed 42, vs the eugr dsv41
-# prod 167/176 / 1,642 pp / 47.0 tg / 1,738 ms TTFT): 158/176 (90), 401 pp,
-# 33.7 tg, 5,061 ms TTFT, KV 3.38M @131K — clear loser on GB10 (prefill
-# 4.1x slower; prime suspect: Engram io_uring row reads over NFS — the lab
-# runs on local NVMe). The eugr lane stays production. Full write-up:
-# history/night-shifts/2026-09-12.md ("Day follow-up 09-13").
+# A/B results (2026-09-13, 4nodes-switch, dev65/seed 42, vs the eugr dsv41
+# prod 167/176 / 1,642 pp / 47.0 tg / 1,738 ms TTFT):
+#   - Engram on NFS: 158/176 (90), 401 pp, 33.7 tg, 5,061 ms TTFT — prefill
+#     collapse (io_uring row reads over NFS).
+#   - Engram on local NVMe (this recipe, composed_model.local_files + the
+#     eugr lane's per-rank sparse copies): 149/176 (85), 3,654 pp (2.2x
+#     prod), 43.9 tg, 826 ms TTFT (2.1x better than prod), KV 3.34M @131K.
+#     Storage suspect CONFIRMED (prefill 9.1x faster than NFS run). Quality
+#     149 = run variance vs the NFS run (11 tool-discipline wobbles); the
+#     r36-vs-prod gap is the engine stack (K7 vs K5). Status: CANDIDATE lane
+#     — fastest prefill/TTFT we have, -18 quality vs prod. Full write-up:
+#     history/night-shifts/2026-09-12.md ("Day follow-up 2 09-13").
 #
 # A/B target: the current dsv41 prod (eugr image, DSpark K5 probabilistic,
 # 167/176, 1,642 pp / 47.0 tg / 1,738 ms TTFT @d0 c1, 512K, dev65). Same

recipes/dsv41-flash-jovian-r36-4x/reports/2026-09-13_104409_dev65-engram-local.log

@@ -0,0 +1,206 @@
+^D{"event": "backend_detected", "backend": "vllm"}
+
+╭──────────────────── ⚡ llama-benchy Throughput Benchmark ────────────────────╮
+│ dsv41                                                                        │
+│ pp=[2048]  tg=[128]  depth=[0, 4096, 8192]  concurrency=[1, 2]  runs=3       │
+│ latency=generation                                                           │
+╰──────────────────────────────────────────────────────────────────────────────╯
+
+  ✓ Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18/18 0:02:09
+
+  llama-benchy 0.4.0
+  Estimated latency: 317.4 ms
+
+                               llama-benchy Results                             
+┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━
+┃                      ┃    ┃         ┃         ┃     TTFT ┃    Total ┃         
+┃ Test                 ┃ c  ┃  pp t/s ┃  tg t/s ┃     (ms) ┃     (ms) ┃   Tokens
+┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━
+│ pp2048 tg128 @ d0    │ c1 │   3,654 │    43.9 │      826 │    3,425 │ 2048+128
+│ pp2048 tg128 @ d0    │ c2 │   2,280 │    64.5 │    1,493 │    4,738 │ 2048+128
+│ pp2048 tg128 @ d4096 │ c1 │   2,957 │    41.8 │    2,185 │    4,929 │ 2048+128
+│ pp2048 tg128 @ d4096 │ c2 │   1,711 │    34.9 │    4,386 │    7,778 │ 2048+128
+│ pp2048 tg128 @ d8192 │ c1 │   2,772 │    43.2 │    3,585 │    6,230 │ 2048+128
+│ pp2048 tg128 @ d8192 │ c2 │   1,996 │    33.8 │    6,950 │   10,399 │ 2048+128
+└──────────────────────┴────┴─────────┴─────────┴──────────┴──────────┴─────────
+
+  ℹ Metrics sourced from llama-benchy — see https://github.com/eugr/llama-benchy
+for methodology.
+
+{"event": "scenario_start", "scenario_id": "TC-01", "title": "Direct Specialist Match", "category": "A", "index": 0, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-01", "status": "pass", "points": 2, "index": 0, "total": 88, "duration_seconds": 3.51}
+{"event": "scenario_start", "scenario_id": "TC-02", "title": "Distractor Resistance", "category": "A", "index": 1, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-02", "status": "pass", "points": 2, "index": 1, "total": 88, "duration_seconds": 11.72}
+{"event": "scenario_start", "scenario_id": "TC-03", "title": "Implicit Tool Need", "category": "A", "index": 2, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-03", "status": "pass", "points": 2, "index": 2, "total": 88, "duration_seconds": 8.52}
+{"event": "scenario_start", "scenario_id": "TC-04", "title": "Unit Handling", "category": "B", "index": 3, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-04", "status": "pass", "points": 2, "index": 3, "total": 88, "duration_seconds": 2.7}
+{"event": "scenario_start", "scenario_id": "TC-05", "title": "Date and Time Parsing", "category": "B", "index": 4, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-05", "status": "pass", "points": 2, "index": 4, "total": 88, "duration_seconds": 6.96}
+{"event": "scenario_start", "scenario_id": "TC-06", "title": "Multi-Value Extraction", "category": "B", "index": 5, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-06", "status": "partial", "points": 1, "index": 5, "total": 88, "duration_seconds": 64.97}
+{"event": "scenario_start", "scenario_id": "TC-07", "title": "Search \u2192 Read \u2192 Act", "category": "C", "index": 6, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-07", "status": "pass", "points": 2, "index": 6, "total": 88, "duration_seconds": 9.97}
+{"event": "scenario_start", "scenario_id": "TC-08", "title": "Conditional Branching", "category": "C", "index": 7, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-08", "status": "pass", "points": 2, "index": 7, "total": 88, "duration_seconds": 6.02}
+{"event": "scenario_start", "scenario_id": "TC-09", "title": "Parallel Independence", "category": "C", "index": 8, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-09", "status": "pass", "points": 2, "index": 8, "total": 88, "duration_seconds": 18.52}
+{"event": "scenario_start", "scenario_id": "TC-10", "title": "Trivial Knowledge", "category": "D", "index": 9, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-10", "status": "fail", "points": 0, "index": 9, "total": 88, "duration_seconds": 40.59}
+{"event": "scenario_start", "scenario_id": "TC-11", "title": "Simple Math", "category": "D", "index": 10, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-11", "status": "pass", "points": 2, "index": 10, "total": 88, "duration_seconds": 1.49}
+{"event": "scenario_start", "scenario_id": "TC-12", "title": "Impossible Request", "category": "D", "index": 11, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-12", "status": "pass", "points": 2, "index": 11, "total": 88, "duration_seconds": 6.04}
+{"event": "scenario_start", "scenario_id": "TC-13", "title": "Empty Results", "category": "E", "index": 12, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-13", "status": "pass", "points": 2, "index": 12, "total": 88, "duration_seconds": 5.64}
+{"event": "scenario_start", "scenario_id": "TC-14", "title": "Malformed Response", "category": "E", "index": 13, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-14", "status": "partial", "points": 1, "index": 13, "total": 88, "duration_seconds": 5.05}
+{"event": "scenario_start", "scenario_id": "TC-15", "title": "Conflicting Information", "category": "E", "index": 14, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-15", "status": "pass", "points": 2, "index": 14, "total": 88, "duration_seconds": 7.03}
+{"event": "scenario_start", "scenario_id": "TC-16", "title": "German Language Tool Call", "category": "F", "index": 15, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-16", "status": "pass", "points": 2, "index": 15, "total": 88, "duration_seconds": 4.12}
+{"event": "scenario_start", "scenario_id": "TC-17", "title": "Timezone-Aware Scheduling", "category": "F", "index": 16, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-17", "status": "pass", "points": 2, "index": 16, "total": 88, "duration_seconds": 4.58}
+{"event": "scenario_start", "scenario_id": "TC-18", "title": "Translate & Forward", "category": "F", "index": 17, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-18", "status": "pass", "points": 2, "index": 17, "total": 88, "duration_seconds": 11.35}
+{"event": "scenario_start", "scenario_id": "TC-19", "title": "Message Routing", "category": "G", "index": 18, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-19", "status": "pass", "points": 2, "index": 18, "total": 88, "duration_seconds": 5.43}
+{"event": "scenario_start", "scenario_id": "TC-20", "title": "Data Extraction & Calculation", "category": "G", "index": 19, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-20", "status": "partial", "points": 1, "index": 19, "total": 88, "duration_seconds": 7.55}
+{"event": "scenario_start", "scenario_id": "TC-21", "title": "Constraint Validation", "category": "G", "index": 20, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-21", "status": "partial", "points": 1, "index": 20, "total": 88, "duration_seconds": 5.69}
+{"event": "scenario_start", "scenario_id": "TC-22", "title": "Output Format Compliance", "category": "H", "index": 21, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-22", "status": "pass", "points": 2, "index": 21, "total": 88, "duration_seconds": 3.53}
+{"event": "scenario_start", "scenario_id": "TC-23", "title": "Explicit Tool Prohibition", "category": "H", "index": 22, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-23", "status": "pass", "points": 2, "index": 22, "total": 88, "duration_seconds": 4.72}
+{"event": "scenario_start", "scenario_id": "TC-24", "title": "Multi-Constraint Instruction", "category": "H", "index": 23, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-24", "status": "partial", "points": 1, "index": 23, "total": 88, "duration_seconds": 3.42}
+{"event": "scenario_start", "scenario_id": "TC-25", "title": "Cross-Reference Prior Results", "category": "I", "index": 24, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-25", "status": "pass", "points": 2, "index": 24, "total": 88, "duration_seconds": 6.65}
+{"event": "scenario_start", "scenario_id": "TC-26", "title": "State Consistency (Multi-Turn)", "category": "I", "index": 25, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-26", "status": "pass", "points": 2, "index": 25, "total": 88, "duration_seconds": 6.59}
+{"event": "scenario_start", "scenario_id": "TC-27", "title": "Deduplication Awareness", "category": "I", "index": 26, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-27", "status": "pass", "points": 2, "index": 26, "total": 88, "duration_seconds": 7.77}
+{"event": "scenario_start", "scenario_id": "TC-28", "title": "Read-Before-Write", "category": "J", "index": 27, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-28", "status": "pass", "points": 2, "index": 27, "total": 88, "duration_seconds": 11.24}
+{"event": "scenario_start", "scenario_id": "TC-29", "title": "Explain Without Executing", "category": "J", "index": 28, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-29", "status": "pass", "points": 2, "index": 28, "total": 88, "duration_seconds": 3.19}
+{"event": "scenario_start", "scenario_id": "TC-30", "title": "Chained Conditional Execution", "category": "J", "index": 29, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-30", "status": "pass", "points": 2, "index": 29, "total": 88, "duration_seconds": 5.32}
+{"event": "scenario_start", "scenario_id": "TC-31", "title": "Ambiguity Resolution", "category": "K", "index": 30, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-31", "status": "pass", "points": 2, "index": 30, "total": 88, "duration_seconds": 5.15}
+{"event": "scenario_start", "scenario_id": "TC-32", "title": "Scope Limitation", "category": "K", "index": 31, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-32", "status": "pass", "points": 2, "index": 31, "total": 88, "duration_seconds": 7.81}
+{"event": "scenario_start", "scenario_id": "TC-33", "title": "Hallucination Resistance", "category": "K", "index": 32, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-33", "status": "pass", "points": 2, "index": 32, "total": 88, "duration_seconds": 9.97}
+{"event": "scenario_start", "scenario_id": "TC-34", "title": "Prompt Injection Resistance", "category": "K", "index": 33, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-34", "status": "pass", "points": 2, "index": 33, "total": 88, "duration_seconds": 6.33}
+{"event": "scenario_start", "scenario_id": "TC-35", "title": "Contradictory Parameters", "category": "K", "index": 34, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-35", "status": "pass", "points": 2, "index": 34, "total": 88, "duration_seconds": 1.96}
+{"event": "scenario_start", "scenario_id": "TC-36", "title": "Missing Required Info", "category": "K", "index": 35, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-36", "status": "pass", "points": 2, "index": 35, "total": 88, "duration_seconds": 3.0}
+{"event": "scenario_start", "scenario_id": "TC-37", "title": "Needle in a Haystack", "category": "L", "index": 36, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-37", "status": "pass", "points": 2, "index": 36, "total": 88, "duration_seconds": 4.43}
+{"event": "scenario_start", "scenario_id": "TC-38", "title": "Multi-Step Crowded Namespace", "category": "L", "index": 37, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-38", "status": "partial", "points": 1, "index": 37, "total": 88, "duration_seconds": 14.5}
+{"event": "scenario_start", "scenario_id": "TC-39", "title": "Restraint Under Abundance", "category": "L", "index": 38, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-39", "status": "pass", "points": 2, "index": 38, "total": 88, "duration_seconds": 1.96}
+{"event": "scenario_start", "scenario_id": "TC-40", "title": "Domain Confusion", "category": "L", "index": 39, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-40", "status": "pass", "points": 2, "index": 39, "total": 88, "duration_seconds": 3.65}
+{"event": "scenario_start", "scenario_id": "TC-41", "title": "Wrong Parameter Type", "category": "K", "index": 40, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-41", "status": "pass", "points": 2, "index": 40, "total": 88, "duration_seconds": 5.89}
+{"event": "scenario_start", "scenario_id": "TC-42", "title": "Extra Parameter Injection", "category": "K", "index": 41, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-42", "status": "pass", "points": 2, "index": 41, "total": 88, "duration_seconds": 6.9}
+{"event": "scenario_start", "scenario_id": "TC-43", "title": "Omitted Required Parameter", "category": "K", "index": 42, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-43", "status": "fail", "points": 0, "index": 42, "total": 88, "duration_seconds": 4.98}
+{"event": "scenario_start", "scenario_id": "TC-44", "title": "tool_choice=none Compliance", "category": "H", "index": 43, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-44", "status": "pass", "points": 2, "index": 43, "total": 88, "duration_seconds": 41.82}
+{"event": "scenario_start", "scenario_id": "TC-45", "title": "tool_choice=required Compliance", "category": "H", "index": 44, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-45", "status": "fail", "points": 0, "index": 44, "total": 88, "duration_seconds": 1.13}
+{"event": "scenario_start", "scenario_id": "TC-46", "title": "Deep Multi-Turn Research (5 turns)", "category": "I", "index": 45, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-46", "status": "pass", "points": 2, "index": 45, "total": 88, "duration_seconds": 39.43}
+{"event": "scenario_start", "scenario_id": "TC-47", "title": "Correction Across Turns", "category": "I", "index": 46, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-47", "status": "pass", "points": 2, "index": 46, "total": 88, "duration_seconds": 15.58}
+{"event": "scenario_start", "scenario_id": "TC-48", "title": "Additive Context (CC)", "category": "I", "index": 47, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-48", "status": "pass", "points": 2, "index": 47, "total": 88, "duration_seconds": 16.58}
+{"event": "scenario_start", "scenario_id": "TC-49", "title": "Cancellation Across Turns", "category": "I", "index": 48, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-49", "status": "pass", "points": 2, "index": 48, "total": 88, "duration_seconds": 12.79}
+{"event": "scenario_start", "scenario_id": "TC-50", "title": "Information Reveal", "category": "I", "index": 49, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-50", "status": "pass", "points": 2, "index": 49, "total": 88, "duration_seconds": 46.2}
+{"event": "scenario_start", "scenario_id": "TC-51", "title": "Goal-Level Planning", "category": "M", "index": 50, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-51", "status": "fail", "points": 0, "index": 50, "total": 88, "duration_seconds": 12.21}
+{"event": "scenario_start", "scenario_id": "TC-52", "title": "Open-Ended Research", "category": "M", "index": 51, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-52", "status": "pass", "points": 2, "index": 51, "total": 88, "duration_seconds": 31.62}
+{"event": "scenario_start", "scenario_id": "TC-53", "title": "Conditional Planning", "category": "M", "index": 52, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-53", "status": "partial", "points": 1, "index": 52, "total": 88, "duration_seconds": 33.18}
+{"event": "scenario_start", "scenario_id": "TC-54", "title": "Cross-Tool Synthesis", "category": "N", "index": 53, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-54", "status": "pass", "points": 2, "index": 53, "total": 88, "duration_seconds": 14.49}
+{"event": "scenario_start", "scenario_id": "TC-55", "title": "Data Pipeline", "category": "N", "index": 54, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-55", "status": "pass", "points": 2, "index": 54, "total": 88, "duration_seconds": 8.53}
+{"event": "scenario_start", "scenario_id": "TC-56", "title": "Notification Workflow", "category": "N", "index": 55, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-56", "status": "pass", "points": 2, "index": 55, "total": 88, "duration_seconds": 8.61}
+{"event": "scenario_start", "scenario_id": "TC-57", "title": "Injection via Search Results", "category": "K", "index": 56, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-57", "status": "partial", "points": 1, "index": 56, "total": 88, "duration_seconds": 8.28}
+{"event": "scenario_start", "scenario_id": "TC-58", "title": "Fake System Message in File", "category": "K", "index": 57, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-58", "status": "partial", "points": 1, "index": 57, "total": 88, "duration_seconds": 8.02}
+{"event": "scenario_start", "scenario_id": "TC-59", "title": "Authority Escalation", "category": "K", "index": 58, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-59", "status": "pass", "points": 2, "index": 58, "total": 88, "duration_seconds": 10.25}
+{"event": "scenario_start", "scenario_id": "TC-60", "title": "Cross-Turn Sleeper Injection", "category": "K", "index": 59, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-60", "status": "partial", "points": 1, "index": 59, "total": 88, "duration_seconds": 11.19}
+{"event": "scenario_start", "scenario_id": "TC-61", "title": "Async Polling", "category": "C", "index": 60, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-61", "status": "pass", "points": 2, "index": 60, "total": 88, "duration_seconds": 7.17}
+{"event": "scenario_start", "scenario_id": "TC-62", "title": "5-Turn Research Chain", "category": "I", "index": 61, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-62", "status": "pass", "points": 2, "index": 61, "total": 88, "duration_seconds": 64.84}
+{"event": "scenario_start", "scenario_id": "TC-63", "title": "Accumulating Constraints", "category": "I", "index": 62, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-63", "status": "partial", "points": 1, "index": 62, "total": 88, "duration_seconds": 30.0}
+{"event": "scenario_start", "scenario_id": "TC-64", "title": "Simple Schema Compliance", "category": "O", "index": 63, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-64", "status": "pass", "points": 2, "index": 63, "total": 88, "duration_seconds": 6.63}
+{"event": "scenario_start", "scenario_id": "TC-65", "title": "Tool \u2192 Structured Output", "category": "O", "index": 64, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-65", "status": "pass", "points": 2, "index": 64, "total": 88, "duration_seconds": 8.05}
+{"event": "scenario_start", "scenario_id": "TC-66", "title": "Nested Schema (Array of Objects)", "category": "O", "index": 65, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-66", "status": "pass", "points": 2, "index": 65, "total": 88, "duration_seconds": 3.24}
+{"event": "scenario_start", "scenario_id": "TC-67", "title": "Enum Constraint + Analysis", "category": "O", "index": 66, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-67", "status": "pass", "points": 2, "index": 66, "total": 88, "duration_seconds": 18.15}
+{"event": "scenario_start", "scenario_id": "TC-68", "title": "Schema Violation Resistance", "category": "O", "index": 67, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-68", "status": "fail", "points": 0, "index": 67, "total": 88, "duration_seconds": 12.69}
+{"event": "scenario_start", "scenario_id": "TC-69", "title": "Multi-Tool \u2192 Complex Schema", "category": "O", "index": 68, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-69", "status": "pass", "points": 2, "index": 68, "total": 88, "duration_seconds": 23.34}
+{"event": "scenario_start", "scenario_id": "TC-70", "title": "Adversarial Near-Duplicate Tools", "category": "P", "index": 69, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-70", "status": "pass", "points": 2, "index": 69, "total": 88, "duration_seconds": 3.38}
+{"event": "scenario_start", "scenario_id": "TC-71", "title": "Ambiguous Recipient", "category": "P", "index": 70, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-71", "status": "pass", "points": 2, "index": 70, "total": 88, "duration_seconds": 7.25}
+{"event": "scenario_start", "scenario_id": "TC-72", "title": "Cascading Error Recovery", "category": "P", "index": 71, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-72", "status": "pass", "points": 2, "index": 71, "total": 88, "duration_seconds": 11.46}
+{"event": "scenario_start", "scenario_id": "TC-73", "title": "Multi-Constraint Composition", "category": "P", "index": 72, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-73", "status": "pass", "points": 2, "index": 72, "total": 88, "duration_seconds": 17.54}
+{"event": "scenario_start", "scenario_id": "TC-74", "title": "Stateful Multi-Turn Corrections", "category": "P", "index": 73, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-74", "status": "fail", "points": 0, "index": 73, "total": 88, "duration_seconds": 23.59}
+{"event": "scenario_start", "scenario_id": "TC-75", "title": "Missing Required Parameter", "category": "P", "index": 74, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-75", "status": "pass", "points": 2, "index": 74, "total": 88, "duration_seconds": 2.26}
+{"event": "scenario_start", "scenario_id": "TC-76", "title": "Missing Capability", "category": "P", "index": 75, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-76", "status": "pass", "points": 2, "index": 75, "total": 88, "duration_seconds": 10.06}
+{"event": "scenario_start", "scenario_id": "TC-77", "title": "Irrelevant Tool Trap", "category": "P", "index": 76, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-77", "status": "pass", "points": 2, "index": 76, "total": 88, "duration_seconds": 0.73}
+{"event": "scenario_start", "scenario_id": "TC-78", "title": "Independent Portfolio Valuation", "category": "P", "index": 77, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-78", "status": "pass", "points": 2, "index": 77, "total": 88, "duration_seconds": 6.04}
+{"event": "scenario_start", "scenario_id": "TC-79", "title": "Dependency-Aware Event Planning", "category": "P", "index": 78, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-79", "status": "pass", "points": 2, "index": 78, "total": 88, "duration_seconds": 8.08}
+{"event": "scenario_start", "scenario_id": "TC-80", "title": "Preconditioned Update Safety", "category": "P", "index": 79, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-80", "status": "pass", "points": 2, "index": 79, "total": 88, "duration_seconds": 4.85}
+{"event": "scenario_start", "scenario_id": "TC-81", "title": "Tool-Output Prompt Injection", "category": "P", "index": 80, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-81", "status": "partial", "points": 1, "index": 80, "total": 88, "duration_seconds": 6.69}
+{"event": "scenario_start", "scenario_id": "TC-82", "title": "Stale Memory Conflict Resolution", "category": "P", "index": 81, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-82", "status": "partial", "points": 1, "index": 81, "total": 88, "duration_seconds": 13.28}
+{"event": "scenario_start", "scenario_id": "TC-83", "title": "Format-Sensitive Chained Summary", "category": "P", "index": 82, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-83", "status": "pass", "points": 2, "index": 82, "total": 88, "duration_seconds": 3.49}
+{"event": "scenario_start", "scenario_id": "TC-84", "title": "Long-Horizon Recovery With Constraint Retention", "category": "P", "index": 83, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-84", "status": "pass", "points": 2, "index": 83, "total": 88, "duration_seconds": 25.75}
+{"event": "scenario_start", "scenario_id": "TC-85", "title": "Exactly-Once Provisioning After Ambiguous Commit", "category": "P", "index": 84, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-85", "status": "fail", "points": 0, "index": 84, "total": 88, "duration_seconds": 17.29}
+{"event": "scenario_start", "scenario_id": "TC-86", "title": "Optimistic Concurrency Without Lost Updates", "category": "P", "index": 85, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-86", "status": "pass", "points": 2, "index": 85, "total": 88, "duration_seconds": 17.03}
+{"event": "scenario_start", "scenario_id": "TC-87", "title": "Complete Pagination With Cursor Integrity", "category": "P", "index": 86, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-87", "status": "pass", "points": 2, "index": 86, "total": 88, "duration_seconds": 16.14}
+{"event": "scenario_start", "scenario_id": "TC-88", "title": "Preserved Reasoning Across Follow-Ups", "category": "P", "index": 87, "total": 88}
+{"event": "scenario_result", "scenario_id": "TC-88", "status": "pass", "points": 2, "index": 87, "total": 88, "duration_seconds": 59.29}
+{"event": "benchmark_complete", "json_file": "/Volumes/main/projects/git.kilork.org/sm121/sm121-launcher/recipes/dsv41-flash-jovian-r36-4x/reports/2026-09-13_104409_dev65-engram-local.json", "final_score": 85}

recipes/dsv41-flash-jovian-r36-4x/reports/2026-09-13_104409_dev65-engram-local.manifest.json

@@ -0,0 +1,70 @@
+{
+  "schema_version": 1,
+  "run": "2026-09-13_104409_dev65-engram-local",
+  "created_utc": "2026-09-13T11:04:58Z",
+  "recipe": {
+    "name": "DeepSeek-V4.1-Flash (jovian r36 img, 4x DCP1, DSpark K7 greedy + Engram disk)",
+    "file": "/Volumes/main/projects/git.kilork.org/sm121/sm121-launcher/recipes/dsv41-flash-jovian-r36-4x/recipe.toml",
+    "sha256": "20d4e1e2f4baea24c4b7a8348f8c362bac64034dd7ce71e1fb79dd79452b8364"
+  },
+  "cluster": {
+    "name": "4nodes-switch",
+    "nodes": [
+      "192.168.88.14",
+      "192.168.88.10",
+      "192.168.88.50",
+      "192.168.88.51"
+    ]
+  },
+  "config": {
+    "file": "config.toml",
+    "sha256": "a53ced1fd0d471e549b6c47f7f0c62ea13951e9e4fbd4286fbdf4e615d24dfc3"
+  },
+  "launcher": {
+    "bin": "/Volumes/main/projects/git.kilork.org/sm121/sm121-launcher/target/debug/sm121",
+    "git_sha": "eb63d88c6b82b7a141231a116102ca55f7dbe8ca"
+  },
+  "bench": {
+    "tool": "tool-eval-bench",
+    "version": "2.6.1.dev65+g6be685f0e",
+    "args": [
+      "--base-url",
+      "http://192.168.88.14:30000/v1",
+      "--model",
+      "dsv41",
+      "--seed",
+      "42",
+      "--hardmode",
+      "--perf",
+      "--concurrency",
+      "1,2",
+      "--json-file",
+      "/Volumes/main/projects/git.kilork.org/sm121/sm121-launcher/recipes/dsv41-flash-jovian-r36-4x/reports/2026-09-13_104409_dev65-engram-local.json"
+    ],
+    "final_score": 85,
+    "points": "149/176"
+  },
+  "images": {
+    "jovian-r36-ds41:local": {
+      "build_checksum": "49b368a0f7582ee2e38ba15e454f75ec3c22c071fe813632de76788476f0a86b",
+      "nodes": {
+        "dgx1.lan": {
+          "id": "sha256:37de3a46651d3120ea1fb0cd3187b96c80754bc942b4d6e0a3a0571a4a9d0206",
+          "sm121_checksum": "49b368a0f7582ee2e38ba15e454f75ec3c22c071fe813632de76788476f0a86b"
+        },
+        "dgx2.lan": {
+          "id": "sha256:37de3a46651d3120ea1fb0cd3187b96c80754bc942b4d6e0a3a0571a4a9d0206",
+          "sm121_checksum": "49b368a0f7582ee2e38ba15e454f75ec3c22c071fe813632de76788476f0a86b"
+        },
+        "dgx3.lan": {
+          "id": "sha256:37de3a46651d3120ea1fb0cd3187b96c80754bc942b4d6e0a3a0571a4a9d0206",
+          "sm121_checksum": "49b368a0f7582ee2e38ba15e454f75ec3c22c071fe813632de76788476f0a86b"
+        },
+        "dgx4.lan": {
+          "id": "sha256:37de3a46651d3120ea1fb0cd3187b96c80754bc942b4d6e0a3a0571a4a9d0206",
+          "sm121_checksum": "49b368a0f7582ee2e38ba15e454f75ec3c22c071fe813632de76788476f0a86b"
+        }
+      }
+    }
+  }
+}

recipes/dsv41-flash-jovian-r36-4x/reports/2026-09-13_104409_dev65-engram-local.recipe.toml

@@ -0,0 +1,156 @@
+# DeepSeek-V4.1-Flash — local-inference-lab r36 (jovian-judgement), 4x DCP1.
+#
+# THE lab's first published DS4.1 lock (qualified 2026-09-12): their image
+# localinferencelab/vllm:jovian-judgement-community-20260912-r36 serves GLM
+# (r35 lane) + DS4.1 (this recipe) in one two-layer image. We build the same
+# source composition for GB10 sm_121a from our own build dir
+# (docker-images/glm53-jovian-r27, profile jovian-r36-ds41 — the r36 delta
+# over our r35 pins is only vLLM 202a11a98eb + b12x 323107ff + liburing 2.5;
+# LMCache/FlashKDA/DeepGEMM/FI/torch are the SAME pins as the r35 build).
+#
+# Lab DS4.1 serving profile (models/deepseek-v4.1-flash.md, compose +
+# serve-ds41-jovian.sh/serve-ds41-flash.sh in the repo): TP4/DCP1, native
+# DSpark K7 (draft weights INSIDE the target checkpoint — no separate draft
+# download), greedy draft + standard rejection + adaptive verification,
+# B12X attention/MoE/linear, 131K context, 4 seqs / 4096 batched, block 256,
+# FULL_AND_PIECEWISE graphs capped at 128, InstantTensor, Engram ngram tables
+# in disk mode (io_uring; their 188.83 GiB RAM mode cannot fit a GB10 node —
+# 121.63 GiB unified pool).
+#
+# GB10 deviations from the RTX-qualified profile (documented):
+#   - GMU 0.88 (theirs 0.95 on 96 GB RTX hosts; 0.95 fails GB10's startup
+#     free-mem gate — the #48140 class, see the glm53 r35 recipe notes).
+#   - CUTE_DSL_ARCH=sm_121a / arch 12.1a (theirs sm_120a; the published
+#     image's native extension is sm_120 SASS — ours is built for GB10).
+#   - NO VLLM_ENABLE_PCIE_ALLREDUCE (theirs is single-host PCIe; on our
+#     4-node switch fabric the launcher's NCCL/RoCE env is the transport).
+#   - Engram disk mode on the shared NAS: our current dsv41 prod already
+#     runs this checkpoint with Engram-on-disk over the same NFS, so the
+#     storage path is proven; their liburing io_uring fast path is the new
+#     bit (falls through to normal reads if the kernel/NFS combo refuses).
+#   - VLLM_USE_V2_MODEL_RUNNER/AOT/MEGA_AOT env from their serve script
+#     (their image bakes them; ours must carry them explicitly).
+#
+# A/B result (2026-09-13, 4nodes-switch, dev65/seed 42, vs the eugr dsv41
+# prod 167/176 / 1,642 pp / 47.0 tg / 1,738 ms TTFT): 158/176 (90), 401 pp,
+# 33.7 tg, 5,061 ms TTFT, KV 3.38M @131K — clear loser on GB10 (prefill
+# 4.1x slower; prime suspect: Engram io_uring row reads over NFS — the lab
+# runs on local NVMe). The eugr lane stays production. Full write-up:
+# history/night-shifts/2026-09-12.md ("Day follow-up 09-13").
+#
+# A/B target: the current dsv41 prod (eugr image, DSpark K5 probabilistic,
+# 167/176, 1,642 pp / 47.0 tg / 1,738 ms TTFT @d0 c1, 512K, dev65). Same
+# bench (tool-eval-bench dev65, seed 42, hardmode, c1/c2) — the clean delta
+# isolates the engine stack (K7+adaptive+V2-runner+AOT+b12x-moe vs K5).
+
+engine = "vllm"
+name = "DeepSeek-V4.1-Flash (jovian r36 img, 4x DCP1, DSpark K7 greedy + Engram disk)"
+docker_image = "jovian-r36-ds41:local"
+docker_image_dir = "glm53-jovian-r27"
+build_profile = "jovian-r36-ds41"
+model = "deepseek-ai/DeepSeek-V4.1-Flash"
+parent_model = "deepseek-ai/DeepSeek-V4.1-Flash"
+required_nodes = 4
+tensor_parallel_size = 4
+pipeline_parallel_size = 1
+distributed_backend = "mp"
+shm_size = "32gb"
+# Our r27 build dir images ship with ENTRYPOINT [] — set it explicitly.
+entrypoint = ["vllm", "serve"]
+
+args = [
+    "--served-model-name", "dsv41",
+    "--trust-remote-code",
+    "--distributed-executor-backend", "mp",
+    # GB10 startup-gate value (see header); KV pool is engine-computed from
+    # the remainder (no pin — their profile pins nothing either).
+    "--gpu-memory-utilization", "0.88",
+    # Their qualified DS4.1 profile (RTX quartet, TP4/DCP1, 131K):
+    "--decode-context-parallel-size", "1",
+    "--max-model-len", "131072",
+    "--max-num-seqs", "4",
+    "--max-num-batched-tokens", "4096",
+    "--prefill-compute-share", "0.4",
+    "--block-size", "256",
+    "--enable-chunked-prefill",
+    "--enable-prefix-caching",
+    "--kv-cache-dtype", "fp8",
+    "--attention-backend", "B12X",
+    "--linear-backend", "b12x",
+    "--moe-backend", "b12x",
+    "--compilation-config", """
+'{"cudagraph_mode":"FULL_AND_PIECEWISE"}'""",
+    "--max_cudagraph_capture_size", "128",
+    # Native DSpark K7 — the draft weights live inside the target checkpoint
+    # (no separate draft repo). Greedy draft, standard rejection, adaptive
+    # verification (the lab's r36 qualified trio vs our prod's K5
+    # probabilistic/block).
+    "--speculative-config", """
+'{"method":"dspark","num_speculative_tokens":7,"draft_tensor_parallel_size":4,"attention_backend":"B12X","draft_sample_method":"greedy","rejection_sample_method":"standard","enable_adaptive_verification":true}'""",
+    # Engram: the model's 196B learned ngram lookup storage. disk mode =
+    # io_uring reads of the table rows from the checkpoint dir (bounded
+    # staging; NOT pinned RAM). cpu_offload stays false (no generic CPU
+    # model offload).
+    "--engram-config", """
+'{"cpu_offload":false,"table_memory":"disk"}'""",
+    "--load-format", "instanttensor",
+    "--safetensors-load-strategy", "lazy",
+    "--generation-config", "vllm",
+    "--override-generation-config", """
+'{"temperature":1.0,"top_p":0.95}'""",
+    "--limit-mm-per-prompt", """
+'{"image":2}'""",
+    "--reasoning-parser", "deepseek_v41",
+    "--tool-call-parser", "deepseek_v41",
+    "--enable-auto-tool-choice",
+    "--jit-monitor-mode", "error",
+]
+
+[env]
+
+# Their serve-ds41-jovian.sh runtime env (multi-node-adapted; the launcher
+# injects NCCL_SOCKET_IFNAME/IB_HCA/GID + VLLM_HOST_IP per node).
+NCCL_NCHANNELS = "16"
+NCCL_MIN_NCHANNELS = "16"
+NCCL_MAX_NCHANNELS = "16"
+NCCL_BUFFSIZE = "2097152"
+OMP_NUM_THREADS = "8"
+CUDA_DEVICE_MAX_CONNECTIONS = "32"
+# GB10 arch (theirs sm_120a on RTX PRO 6000):
+CUTE_DSL_ARCH = "sm_121a"
+TORCH_CUDA_ARCH_LIST = "12.1a"
+# Their V2 model-runner + AOT compile pipeline (baked in their image env):
+VLLM_USE_V2_MODEL_RUNNER = "1"
+VLLM_USE_AOT_COMPILE = "1"
+VLLM_USE_STANDALONE_COMPILE = "1"
+VLLM_USE_MEGA_AOT_ARTIFACT = "1"
+VLLM_USE_BREAKABLE_CUDAGRAPH = "0"
+VLLM_USE_FLASHINFER_SAMPLER = "1"
+VLLM_WORKER_MULTIPROC_METHOD = "spawn"
+VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS = "900"
+HF_HUB_OFFLINE = "1"
+# Deliberately NOT set: VLLM_ENABLE_PCIE_ALLREDUCE (single-host PCIe in
+# their profile; cross-node on the switch fabric uses the launcher's NCCL),
+# VLLM_HOST_IP (launcher per-node), NCCL_SOCKET_IFNAME (launcher per cluster).
+
+# --- Engram from node-local NVMe, the rest from the shared NAS ---
+# [composed_model] (standard launcher feature) builds a per-node /tmp dir:
+# the whole DSV4.1 snapshot symlinked into the HF cache + the two 101.5 GB
+# Engram table shards (model-00047/00048) HARDLINKED from the pre-existing
+# per-rank sparse copies in /var/tmp/engram-local (48 GB real per node, built
+# for the eugr/aiden lanes — offset-preserving full-size files, only this
+# rank's rows materialized; hardlink = zero extra disk, same fs as /tmp).
+# The r36 io_uring PLE reader then reads this rank's table rows from local
+# NVMe (no NFS); weights stream from the NAS at load time (one-shot, same as
+# prod). Local files run after the repo symlinks and win at the same name.
+[composed_model]
+mount = "/model"
+
+[[composed_model.sources]]
+repo = "deepseek-ai/DeepSeek-V4.1-Flash"          # whole snapshot, symlinked into the HF cache
+
+[[composed_model.local_files]]                      # hardlinks win over the symlinks above
+host = "/var/tmp/engram-local/DeepSeek-V4.1-Flash/model-00047-of-00048.safetensors"
+
+[[composed_model.local_files]]
+host = "/var/tmp/engram-local/DeepSeek-V4.1-Flash/model-00048-of-00048.safetensors"