docs: dgx4 NVMe investigation - DRAM-less FTL thrash, not a fault

Measured the wedge node's drive with a 4 KiB O_DIRECT random-read harness
against the package's 98 GB Engram shard, reference arm on idle dgx5:

- dgx4 (KingSpec XG7000-2TB 2242): 68.8K IOPS / p50 69us on a hot 2 GB slice,
  38.8K / 169us at 20 GB, 12.9K / 1183us at 98 GB, and 24.8K with p99.9 10.5ms
  at 64 threads - throughput collapses as the working set grows, then
  congestion-collapses with queue depth.
- dgx5 (ESL01TBTLCZ): 113K / 109us, 112K, 80K - flat, and 204K at 64 threads.
- sequential is fine (1.9 vs 3.0 GB/s), which is why package copies never
  flagged it.

Identity: DRAM-less, Maxio MAP1602A + 232L YMTC TLC, 40 MB HMB allocated
(nvme dmesg), no firmware tool; SMART is clean (0 media errors, 0 error log,
no thermal throttle, 96 h, 42 C) - it answers everything, just slowly.

Arithmetic: 41 Engram reads/token => a 4096-token chunk is ~170K reads /
680 MB => 7-13 s on that drive vs 1-2 s on a healthy one, i.e. the ~8-20 s
collective spin window. That is why the reader patch helps warm paths and
cannot save cold/sustained ones.

Also records the purchase requirement (DRAM'd enterprise M.2 2242:
PM9A3/PM9E1, MP700 Micro) and the reproducible measurement recipe.

AGENTS.md

@@ -445,7 +445,9 @@ Engram random-read duty — the TP transport poisons on a cold-boot first
 chunk (`seq 93`, prefill=1) *and* after sustained load (`seq 2833389` after a
 full bench), with or without the upstream reader patch. `4nodes-switch`
 (`prod4b`, dgx1–4) stays configured for after that drive is replaced. Full
-evidence: `specs/tech-architecture/flare-roce-poison.md`.
+evidence: `specs/tech-architecture/flare-roce-poison.md`; the drive
+itself (latency percentiles vs working-set size, DRAM-less/HMB thrash):
+`specs/tech-architecture/dgx4-nvme-investigation.md`.
 
 **Uniform binary rule.** The patched runtime (upstream `main` + PR #184) is
 installed as `~/.local/lib/flare-agent/flare` **and** `/usr/bin/flare` on

specs/tech-architecture/dgx4-nvme-investigation.md

@@ -0,0 +1,131 @@
+# dgx4's NVMe: why the Spark wedge is one bad drive (2026-11-04)
+
+The FLARE lane on dgx1–4 died repeatedly with `RoCE collective … timed out …
+the transport is poisoned` (see [`flare-roce-poison.md`](flare-roce-poison.md)).
+The trigger is not the fabric, not the rank order, not the engine: it is
+**dgx4's boot SSD**, and it is not a fault — the drive is healthy, it is simply
+the wrong class for the workload.
+
+## The drive
+
+| | dgx4 | every other node |
+|---|---|---|
+| model | **KingSpec `XG7000-2TB 2242`** (`/sys/class/nvme/nvme0/model`) | `ESL01TBTLCZ-27J2-TYN` 1 TB |
+| firmware | `GT0d9d63` (vid `0x1e4b`, not in the public firmware DB) | `ERFM12.0` |
+| PCIe | Gen4 x4, 16 GT/s, ASPM disabled, no AER errors | Gen4 x4 |
+| HMB | **`hmpre`/`hmmin` 10240 = 40 MB, and it is allocated** (`nvme nvme0: allocated 40 MiB host memory buffer (1 segment)`) | declares 64 MB |
+| power states | 5 (`npss 4`), APST on, idles in **PS3** | 5, APST on |
+| health | 96 h powered, 54.9 TB read / 13.4 TB written, `media_errors 0`, 0 error-log entries, 0 thermal-throttle events, 42 °C | healthy |
+
+Vendor identity (community teardowns/press): KingSpec XG7000 = **DRAM-less**
+design, **Maxio MAP1602A** controller, **232-layer YMTC TLC**, graphene
+heat-spacer instead of a heatsink, no first-party management or firmware tool.
+It is a cheap "fast sequential" gaming drive; "DRAM-less" is the operative word
+— the flash-translation mapping table lives in the 40 MB host buffer plus the
+controller's small SRAM, not in on-drive DRAM.
+
+## What it does under our workload
+
+Measured with a 4 KiB `O_DIRECT` random-read harness on the package's Engram
+shard (98 GB), reporting per-read completion latency:
+
+| working set (random 4 K, 16 threads, 15 s) | dgx4 XG7000 | dgx5 ESL01 |
+|---|---|---|
+| 2 GB | 68.8 K IOPS, p50 **69 µs** | 113 K IOPS, p50 109 µs |
+| 20 GB | 38.8 K IOPS, p50 169 µs | 112 K IOPS, p50 110 µs |
+| 98 GB | **12.9 K IOPS, p50 1,183 µs**, p99.9 4.6 ms | 80 K IOPS, p50 200 µs |
+| 98 GB, 64 threads | **24.8 K IOPS, p50 2.45 ms**, p99.9 10.5 ms | **204 K IOPS**, p50 283 µs |
+
+Two things stand out:
+
+1. **Throughput collapses as the working set grows** (69 K → 13 K IOPS) while
+   the healthy drive stays flat (113 K → 80 K). That is the mapping cache:
+   98 GB / 4 KiB = 24 M mappings; even at 8 B per entry that is ~190 MB of
+   table against a 40 MB HMB, so a random read over the whole table is a
+   mapping miss — an extra NAND fetch — most of the time. Latency p50 goes
+   from 69 µs (hot 2 GB slice, table fits) to 1.2–2.5 ms (whole table).
+2. **It degrades with queue depth instead of scaling**: 16 → 64 threads moves
+   it 12.9 K → 24.8 K IOPS but p50 1.18 → 2.45 ms (congestion collapse). The
+   healthy drive goes 80 K → 204 K IOPS and holds sub-millisecond p50.
+
+Sequential is fine, which is why nothing noticed before: streaming 4 GiB with
+`iflag=direct` reads **1.9 GB/s** on dgx4 vs 3.0 GB/s on dgx5 (~63 %). Package
+copies, image pulls and `flare download` are all sequential, and the weight
+load is read once into GPU memory.
+
+## Why that kills the engine (the arithmetic)
+
+FLARE serves this package with disk-backed Engram tables and reads rows **at
+token time**. From the production lane's own metrics: a 157 K-token prefill
+did **6.49 M Engram reads** → **≈41 reads/token**. A 4096-token prefill chunk
+(the package's baked geometry) therefore needs **≈170 K random reads ≈ 680 MB**.
+
+| node | sustained random read at 98 GB | 680 MB chunk |
+|---|---|---|
+| dgx5-class drive | ~330–800 MB/s | **1–2 s** |
+| dgx4 XG7000 | ~52–99 MB/s | **7–13 s** |
+
+The collective's spin limit is ~8–20 s (20 M polls in the package's kernel),
+and it covers a whole chunk. A device that needs 7–13 s per chunk with a tail
+that spikes to 10–18 ms p99.9 sits right on that boundary — which is exactly
+what we see: healthy on a warm stand, dead on a cold boot (`seq 93`,
+prefill = 1 token) and dead again after sustained load (`seq 2833389` after a
+full bench). The upstream reader patch (PR #184, io_uring lanes) raises the
+reader's parallelism but cannot raise the device's ceiling, which is why it
+helps on the warm/cached path and does not save the cold or sustained cases.
+
+## What this is not
+
+- **Not a failing drive**: `media_errors 0`, zero error-log entries, no
+  thermal-throttle transitions, 42 °C, and 96 power-on hours. It answers every
+  I/O correctly, just slowly under random load. SMART will never flag it.
+- **Not thermal**: no T1/T2 management transitions; the graphene pad is enough
+  for a 6 W-class drive.
+- **Not ASPM/APST flakiness**: the link stays Gen4 x4 with ASPM disabled;
+  idle wake is single-digit milliseconds, not seconds.
+- **Not the PCIe slot**: the same slot class reaches 3.0 GB/s sequential on the
+  other nodes' drives; the /usr/bin and root filesystem are unaffected.
+
+## Decisions taken (2026-11-04)
+
+- dgx4 does **not** serve. It keeps the 2 TB disk as the orbit controller and
+  package store (CLI store 488 GB + agent cache 488 GB — still the only node
+  that fits both) and the production lane runs on dgx1+dgx2+dgx3+dgx8.
+- The `4nodes-switch` / `prod4b` configuration stays in the config for after a
+  drive swap, and the A/B in `flare-roce-poison.md` is the acceptance test:
+  157 K first-touch prompt **and** a full bench on a cold stand.
+
+## What to buy instead (if dgx4 must serve)
+
+The requirement is sustained random-read at a large working set, i.e. a
+**DRAM'd controller with a big FTL map cache**, not peak sequential speed:
+
+1. **Samsung PM9A3 M.2 2242** (enterprise, Elpis 8-channel, DRAM) or
+   **PM9E1 2242** if reachable — designed for exactly this duty.
+2. **Corsair MP700 Micro 2 TB** (Gen5 2242, Phison E28, 8-channel with DRAM).
+3. **Corsair MP600 Micro 2 TB** (2242, Gen4) — consumer DRAM-equipped class,
+   the floor of what is acceptable.
+4. **Avoid** anything DRAM-less/HMB in this role (XG7000, MP600 Mini 2230,
+   WD SN770M): they are the same architecture as the problem drive.
+
+Before any reinstall: back up `~/.config/flare/orbit/controller.key` — restoring
+it after a DGX OS reimage avoids re-pairing all eight agents.
+
+## Reproducing the measurement
+
+```bash
+# /tmp/lat2.c: N threads, 4 KiB O_DIRECT random reads in a centered working
+# set, per-read latency percentiles + >200 ms / >1 s / >5 s counters
+cc -O2 -o /tmp/lat2 /tmp/lat2.c -lpthread
+F=~/.cache/flare/packages/<pkg>/weights/engram-0-weights.safetensors
+/tmp/lat2 "$F" 16 15 2      # hot 2 GB slice
+/tmp/lat2 "$F" 16 15 98     # whole 98 GB table
+/tmp/lat2 "$F" 64 20 98     # queue-depth collapse
+# identity/health
+sudo nvme id-ctrl /dev/nvme0 | grep -E '^(mn|fr|hmpre|npss)'
+sudo nvme smart-log /dev/nvme0 | grep -E 'media_errors|Temperature|Thermal'
+journalctl -k | grep 'host memory buffer'
+```
+
+Run the reference arm on an idle non-serving node (dgx5) — never on a node that
+is currently serving.

specs/tech-architecture/flare-roce-poison.md

@@ -35,6 +35,9 @@ measured, on the production package and flags.
   Sustained random-read ceiling measured on the wedge node: **≈29 K IOPS**
   (4 threads, O_DIRECT 4 K, 10 s) vs ≈33 K on a healthy dgx5 — while a
   first-touch prefill demanded **≈24 K reads/s**, i.e. 80–90 % of the drive.
+  Full drive investigation (latency percentiles vs working-set size,
+  DRAM-less/HMB mapping-cache thrash, chunk arithmetic):
+  [`dgx4-nvme-investigation.md`](dgx4-nvme-investigation.md).
 - **Mitigation that works today**: the unmerged upstream io_uring reader
   (PR #184) on top of current `main` — **partially**. Same hardware, same
   package, same prompt: 3/3 wedges before, clean run after **on a warm stand**