# dgx4's NVMe: why the Spark wedge is one bad drive (2026-11-04)

The FLARE lane on dgx1–4 died repeatedly with `RoCE collective … timed out …
the transport is poisoned` (see [`flare-roce-poison.md`](flare-roce-poison.md)).
The trigger is not the fabric, not the rank order, not the engine: it is
**dgx4's boot SSD**, and it is not a fault — the drive is healthy, it is simply
the wrong class for the workload.

## The drive

| | dgx4 | every other node |
|---|---|---|
| model | **KingSpec `XG7000-2TB 2242`** (`/sys/class/nvme/nvme0/model`) | `ESL01TBTLCZ-27J2-TYN` 1 TB |
| firmware | `GT0d9d63` (vid `0x1e4b`, not in the public firmware DB) | `ERFM12.0` |
| PCIe | Gen4 x4, 16 GT/s, ASPM disabled, no AER errors | Gen4 x4 |
| HMB | **`hmpre`/`hmmin` 10240 = 40 MB, and it is allocated** (`nvme nvme0: allocated 40 MiB host memory buffer (1 segment)`) | declares 64 MB |
| power states | 5 (`npss 4`), APST on, idles in **PS3** | 5, APST on |
| health | 96 h powered, 54.9 TB read / 13.4 TB written, `media_errors 0`, 0 error-log entries, 0 thermal-throttle events, 42 °C | healthy |

Vendor identity (community teardowns/press): KingSpec XG7000 = **DRAM-less**
design, **Maxio MAP1602A** controller, **232-layer YMTC TLC**, graphene
heat-spacer instead of a heatsink, no first-party management or firmware tool.
It is a cheap "fast sequential" gaming drive; "DRAM-less" is the operative word
— the flash-translation mapping table lives in the 40 MB host buffer plus the
controller's small SRAM, not in on-drive DRAM.

## What it does under our workload

Measured with a 4 KiB `O_DIRECT` random-read harness on the package's Engram
shard (98 GB), reporting per-read completion latency:

| working set (random 4 K, 16 threads, 15 s) | dgx4 XG7000 | dgx5 ESL01 |
|---|---|---|
| 2 GB | 68.8 K IOPS, p50 **69 µs** | 113 K IOPS, p50 109 µs |
| 20 GB | 38.8 K IOPS, p50 169 µs | 112 K IOPS, p50 110 µs |
| 98 GB | **12.9 K IOPS, p50 1,183 µs**, p99.9 4.6 ms | 80 K IOPS, p50 200 µs |
| 98 GB, 64 threads | **24.8 K IOPS, p50 2.45 ms**, p99.9 10.5 ms | **204 K IOPS**, p50 283 µs |

Two things stand out:

1. **Throughput collapses as the working set grows** (69 K → 13 K IOPS) while
   the healthy drive stays flat (113 K → 80 K). That is the mapping cache:
   98 GB / 4 KiB = 24 M mappings; even at 8 B per entry that is ~190 MB of
   table against a 40 MB HMB, so a random read over the whole table is a
   mapping miss — an extra NAND fetch — most of the time. Latency p50 goes
   from 69 µs (hot 2 GB slice, table fits) to 1.2–2.5 ms (whole table).
2. **It degrades with queue depth instead of scaling**: 16 → 64 threads moves
   it 12.9 K → 24.8 K IOPS but p50 1.18 → 2.45 ms (congestion collapse). The
   healthy drive goes 80 K → 204 K IOPS and holds sub-millisecond p50.

Sequential is fine, which is why nothing noticed before: streaming 4 GiB with
`iflag=direct` reads **1.9 GB/s** on dgx4 vs 3.0 GB/s on dgx5 (~63 %). Package
copies, image pulls and `flare download` are all sequential, and the weight
load is read once into GPU memory.

## Why that kills the engine (the arithmetic)

FLARE serves this package with disk-backed Engram tables and reads rows **at
token time**. From the production lane's own metrics: a 157 K-token prefill
did **6.49 M Engram reads** → **≈41 reads/token**. A 4096-token prefill chunk
(the package's baked geometry) therefore needs **≈170 K random reads ≈ 680 MB**.

| node | sustained random read at 98 GB | 680 MB chunk |
|---|---|---|
| dgx5-class drive | ~330–800 MB/s | **1–2 s** |
| dgx4 XG7000 | ~52–99 MB/s | **7–13 s** |

The collective's spin limit is ~8–20 s (20 M polls in the package's kernel),
and it covers a whole chunk. A device that needs 7–13 s per chunk with a tail
that spikes to 10–18 ms p99.9 sits right on that boundary — which is exactly
what we see: healthy on a warm stand, dead on a cold boot (`seq 93`,
prefill = 1 token) and dead again after sustained load (`seq 2833389` after a
full bench). The upstream reader patch (PR #184, io_uring lanes) raises the
reader's parallelism but cannot raise the device's ceiling, which is why it
helps on the warm/cached path and does not save the cold or sustained cases.

## What this is not

- **Not a failing drive**: `media_errors 0`, zero error-log entries, no
  thermal-throttle transitions, 42 °C, and 96 power-on hours. It answers every
  I/O correctly, just slowly under random load. SMART will never flag it.
- **Not thermal**: no T1/T2 management transitions; the graphene pad is enough
  for a 6 W-class drive.
- **Not ASPM/APST flakiness**: the link stays Gen4 x4 with ASPM disabled;
  idle wake is single-digit milliseconds, not seconds.
- **Not the PCIe slot**: the same slot class reaches 3.0 GB/s sequential on the
  other nodes' drives; the /usr/bin and root filesystem are unaffected.

## Decisions taken (2026-11-04)

- dgx4 does **not** serve. It keeps the 2 TB disk as the orbit controller and
  package store (CLI store 488 GB + agent cache 488 GB — still the only node
  that fits both) and the production lane runs on dgx1+dgx2+dgx3+dgx8.
- The `4nodes-switch` / `prod4b` configuration stays in the config for after a
  drive swap, and the A/B in `flare-roce-poison.md` is the acceptance test:
  157 K first-touch prompt **and** a full bench on a cold stand.

## What to buy instead (if dgx4 must serve)

The requirement is sustained random-read at a large working set, i.e. a
**DRAM'd controller with a big FTL map cache**, not peak sequential speed:

1. **Samsung PM9A3 M.2 2242** (enterprise, Elpis 8-channel, DRAM) or
   **PM9E1 2242** if reachable — designed for exactly this duty.
2. **Corsair MP700 Micro 2 TB** (Gen5 2242, Phison E28, 8-channel with DRAM).
3. **Corsair MP600 Micro 2 TB** (2242, Gen4) — consumer DRAM-equipped class,
   the floor of what is acceptable.
4. **Avoid** anything DRAM-less/HMB in this role (XG7000, MP600 Mini 2230,
   WD SN770M): they are the same architecture as the problem drive.

Before any reinstall: back up `~/.config/flare/orbit/controller.key` — restoring
it after a DGX OS reimage avoids re-pairing all eight agents.

## Reproducing the measurement

```bash
# /tmp/lat2.c: N threads, 4 KiB O_DIRECT random reads in a centered working
# set, per-read latency percentiles + >200 ms / >1 s / >5 s counters
cc -O2 -o /tmp/lat2 /tmp/lat2.c -lpthread
F=~/.cache/flare/packages/<pkg>/weights/engram-0-weights.safetensors
/tmp/lat2 "$F" 16 15 2      # hot 2 GB slice
/tmp/lat2 "$F" 16 15 98     # whole 98 GB table
/tmp/lat2 "$F" 64 20 98     # queue-depth collapse
# identity/health
sudo nvme id-ctrl /dev/nvme0 | grep -E '^(mn|fr|hmpre|npss)'
sudo nvme smart-log /dev/nvme0 | grep -E 'media_errors|Temperature|Thermal'
journalctl -k | grep 'host memory buffer'
```

Run the reference arm on an idle non-serving node (dgx5) — never on a node that
is currently serving.