launcher/specs/tech-architecture/dgx4-nvme-investigation.md

dgx4's NVMe: why the Spark wedge is one bad drive (2026-11-04)

The FLARE lane on dgx1–4 died repeatedly with RoCE collective … timed out … the transport is poisoned (see flare-roce-poison.md). The trigger is not the fabric, not the rank order, not the engine: it is dgx4's boot SSD, and it is not a fault — the drive is healthy, it is simply the wrong class for the workload.

The drive

	dgx4	every other node
model	KingSpec XG7000-2TB 2242 (/sys/class/nvme/nvme0/model)	ESL01TBTLCZ-27J2-TYN 1 TB
firmware	GT0d9d63 (vid 0x1e4b, not in the public firmware DB)	ERFM12.0
PCIe	Gen4 x4, 16 GT/s, ASPM disabled, no AER errors	Gen4 x4
HMB	hmpre/hmmin 10240 = 40 MB, and it is allocated (nvme nvme0: allocated 40 MiB host memory buffer (1 segment))	declares 64 MB
power states	5 (npss 4), APST on, idles in PS3	5, APST on
health	96 h powered, 54.9 TB read / 13.4 TB written, media_errors 0, 0 error-log entries, 0 thermal-throttle events, 42 °C	healthy

Vendor identity (community teardowns/press): KingSpec XG7000 = DRAM-less design, Maxio MAP1602A controller, 232-layer YMTC TLC, graphene heat-spacer instead of a heatsink, no first-party management or firmware tool. It is a cheap "fast sequential" gaming drive; "DRAM-less" is the operative word — the flash-translation mapping table lives in the 40 MB host buffer plus the controller's small SRAM, not in on-drive DRAM.

What it does under our workload

Measured with a 4 KiB O_DIRECT random-read harness on the package's Engram shard (98 GB), reporting per-read completion latency:

working set (random 4 K, 16 threads, 15 s)	dgx4 XG7000	dgx5 ESL01
2 GB	68.8 K IOPS, p50 69 µs	113 K IOPS, p50 109 µs
20 GB	38.8 K IOPS, p50 169 µs	112 K IOPS, p50 110 µs
98 GB	12.9 K IOPS, p50 1,183 µs, p99.9 4.6 ms	80 K IOPS, p50 200 µs
98 GB, 64 threads	24.8 K IOPS, p50 2.45 ms, p99.9 10.5 ms	204 K IOPS, p50 283 µs

Two things stand out:

1. Throughput collapses as the working set grows (69 K → 13 K IOPS) while the healthy drive stays flat (113 K → 80 K). That is the mapping cache: 98 GB / 4 KiB = 24 M mappings; even at 8 B per entry that is ~190 MB of table against a 40 MB HMB, so a random read over the whole table is a mapping miss — an extra NAND fetch — most of the time. Latency p50 goes from 69 µs (hot 2 GB slice, table fits) to 1.2–2.5 ms (whole table).
2. It degrades with queue depth instead of scaling: 16 → 64 threads moves it 12.9 K → 24.8 K IOPS but p50 1.18 → 2.45 ms (congestion collapse). The healthy drive goes 80 K → 204 K IOPS and holds sub-millisecond p50.

Sequential is fine, which is why nothing noticed before: streaming 4 GiB with iflag=direct reads 1.9 GB/s on dgx4 vs 3.0 GB/s on dgx5 (~63 %). Package copies, image pulls and flare download are all sequential, and the weight load is read once into GPU memory.

Why that kills the engine (the arithmetic)

FLARE serves this package with disk-backed Engram tables and reads rows at token time. From the production lane's own metrics: a 157 K-token prefill did 6.49 M Engram reads → ≈41 reads/token. A 4096-token prefill chunk (the package's baked geometry) therefore needs ≈170 K random reads ≈ 680 MB.

node	sustained random read at 98 GB	680 MB chunk
dgx5-class drive	~330–800 MB/s	1–2 s
dgx4 XG7000	~52–99 MB/s	7–13 s

The collective's spin limit is ~8–20 s (20 M polls in the package's kernel), and it covers a whole chunk. A device that needs 7–13 s per chunk with a tail that spikes to 10–18 ms p99.9 sits right on that boundary — which is exactly what we see: healthy on a warm stand, dead on a cold boot (seq 93, prefill = 1 token) and dead again after sustained load (seq 2833389 after a full bench). The upstream reader patch (PR #184, io_uring lanes) raises the reader's parallelism but cannot raise the device's ceiling, which is why it helps on the warm/cached path and does not save the cold or sustained cases.

What this is not

• Not a failing drive: media_errors 0, zero error-log entries, no thermal-throttle transitions, 42 °C, and 96 power-on hours. It answers every I/O correctly, just slowly under random load. SMART will never flag it.
• Not thermal: no T1/T2 management transitions; the graphene pad is enough for a 6 W-class drive.
• Not ASPM/APST flakiness: the link stays Gen4 x4 with ASPM disabled; idle wake is single-digit milliseconds, not seconds.
• Not the PCIe slot: the same slot class reaches 3.0 GB/s sequential on the other nodes' drives; the /usr/bin and root filesystem are unaffected.

Decisions taken (2026-11-04)

• dgx4 does not serve. It keeps the 2 TB disk as the orbit controller and package store (CLI store 488 GB + agent cache 488 GB — still the only node that fits both) and the production lane runs on dgx1+dgx2+dgx3+dgx8.
• The 4nodes-switch / prod4b configuration stays in the config for after a drive swap, and the A/B in flare-roce-poison.md is the acceptance test: 157 K first-touch prompt and a full bench on a cold stand.

What to buy instead (if dgx4 must serve)

The requirement is sustained random-read at a large working set, i.e. a DRAM'd controller with a big FTL map cache, not peak sequential speed:

1. Samsung PM9A3 M.2 2242 (enterprise, Elpis 8-channel, DRAM) or PM9E1 2242 if reachable — designed for exactly this duty.
2. Corsair MP700 Micro 2 TB (Gen5 2242, Phison E28, 8-channel with DRAM).
3. Corsair MP600 Micro 2 TB (2242, Gen4) — consumer DRAM-equipped class, the floor of what is acceptable.
4. Avoid anything DRAM-less/HMB in this role (XG7000, MP600 Mini 2230, WD SN770M): they are the same architecture as the problem drive.

Before any reinstall: back up ~/.config/flare/orbit/controller.key — restoring it after a DGX OS reimage avoids re-pairing all eight agents.

Reproducing the measurement

# /tmp/lat2.c: N threads, 4 KiB O_DIRECT random reads in a centered working
# set, per-read latency percentiles + >200 ms / >1 s / >5 s counters
cc -O2 -o /tmp/lat2 /tmp/lat2.c -lpthread
F=~/.cache/flare/packages/<pkg>/weights/engram-0-weights.safetensors
/tmp/lat2 "$F" 16 15 2      # hot 2 GB slice
/tmp/lat2 "$F" 16 15 98     # whole 98 GB table
/tmp/lat2 "$F" 64 20 98     # queue-depth collapse
# identity/health
sudo nvme id-ctrl /dev/nvme0 | grep -E '^(mn|fr|hmpre|npss)'
sudo nvme smart-log /dev/nvme0 | grep -E 'media_errors|Temperature|Thermal'
journalctl -k | grep 'host memory buffer'

Run the reference arm on an idle non-serving node (dgx5) — never on a node that is currently serving.