dgx4's NVMe: why the Spark wedge is one bad drive (2026-11-04)

The FLARE lane on dgx1–4 died repeatedly with RoCE collective … timed out … the transport is poisoned (see flare-roce-poison.md). The trigger is not the fabric, not the rank order, not the engine: it is dgx4's boot SSD, and it is not a fault — the drive is healthy, it is simply the wrong class for the workload.

The drive

	dgx4	every other node
model	KingSpec `XG7000-2TB 2242` (`/sys/class/nvme/nvme0/model`)	`ESL01TBTLCZ-27J2-TYN` 1 TB
firmware	`GT0d9d63` (vid `0x1e4b`, not in the public firmware DB)	`ERFM12.0`
PCIe	Gen4 x4, 16 GT/s, ASPM disabled, no AER errors	Gen4 x4
HMB	`hmpre`/`hmmin` 10240 = 40 MB, and it is allocated (`nvme nvme0: allocated 40 MiB host memory buffer (1 segment)`)	declares 64 MB
power states	5 (`npss 4`), APST on, idles in PS3	5, APST on
health	96 h powered, 54.9 TB read / 13.4 TB written, `media_errors 0`, 0 error-log entries, 0 thermal-throttle events, 42 °C	healthy

Vendor identity (community teardowns/press): KingSpec XG7000 = DRAM-less design, Maxio MAP1602A controller, 232-layer YMTC TLC, graphene heat-spacer instead of a heatsink, no first-party management or firmware tool. It is a cheap "fast sequential" gaming drive; "DRAM-less" is the operative word — the flash-translation mapping table lives in the 40 MB host buffer plus the controller's small SRAM, not in on-drive DRAM.

What it does under our workload

Measured with a 4 KiB O_DIRECT random-read harness on the package's Engram shard (98 GB), reporting per-read completion latency:

working set (random 4 K, 16 threads, 15 s)	dgx4 XG7000	dgx5 ESL01
2 GB	68.8 K IOPS, p50 69 µs	113 K IOPS, p50 109 µs
20 GB	38.8 K IOPS, p50 169 µs	112 K IOPS, p50 110 µs
98 GB	12.9 K IOPS, p50 1,183 µs, p99.9 4.6 ms	80 K IOPS, p50 200 µs
98 GB, 64 threads	24.8 K IOPS, p50 2.45 ms, p99.9 10.5 ms	204 K IOPS, p50 283 µs

Two things stand out:

Throughput collapses as the working set grows (69 K → 13 K IOPS) while the healthy drive stays flat (113 K → 80 K). That is the mapping cache: 98 GB / 4 KiB = 24 M mappings; even at 8 B per entry that is ~190 MB of table against a 40 MB HMB, so a random read over the whole table is a mapping miss — an extra NAND fetch — most of the time. Latency p50 goes from 69 µs (hot 2 GB slice, table fits) to 1.2–2.5 ms (whole table).
It degrades with queue depth instead of scaling: 16 → 64 threads moves it 12.9 K → 24.8 K IOPS but p50 1.18 → 2.45 ms (congestion collapse). The healthy drive goes 80 K → 204 K IOPS and holds sub-millisecond p50.

Sequential is fine, which is why nothing noticed before: streaming 4 GiB with iflag=direct reads 1.9 GB/s on dgx4 vs 3.0 GB/s on dgx5 (~63 %). Package copies, image pulls and flare download are all sequential, and the weight load is read once into GPU memory.

Why that kills the engine (the arithmetic)

FLARE serves this package with disk-backed Engram tables and reads rows at token time. From the production lane's own metrics: a 157 K-token prefill did 6.49 M Engram reads → ≈41 reads/token. A 4096-token prefill chunk (the package's baked geometry) therefore needs ≈170 K random reads ≈ 680 MB.

node	sustained random read at 98 GB	680 MB chunk
dgx5-class drive	~330–800 MB/s	1–2 s
dgx4 XG7000	~52–99 MB/s	7–13 s

The collective's spin limit is ~8–20 s (20 M polls in the package's kernel), and it covers a whole chunk. A device that needs 7–13 s per chunk with a tail that spikes to 10–18 ms p99.9 sits right on that boundary — which is exactly what we see: healthy on a warm stand, dead on a cold boot (seq 93, prefill = 1 token) and dead again after sustained load (seq 2833389 after a full bench). The upstream reader patch (PR #184, io_uring lanes) raises the reader's parallelism but cannot raise the device's ceiling, which is why it helps on the warm/cached path and does not save the cold or sustained cases.

What this is not

Not a failing drive: media_errors 0, zero error-log entries, no thermal-throttle transitions, 42 °C, and 96 power-on hours. It answers every I/O correctly, just slowly under random load. SMART will never flag it.
Not thermal: no T1/T2 management transitions; the graphene pad is enough for a 6 W-class drive.
Not ASPM/APST flakiness: the link stays Gen4 x4 with ASPM disabled; idle wake is single-digit milliseconds, not seconds.
Not the PCIe slot: the same slot class reaches 3.0 GB/s sequential on the other nodes' drives; the /usr/bin and root filesystem are unaffected.

Decisions taken (2026-11-04)

dgx4 does not serve. It keeps the 2 TB disk as the orbit controller and package store (CLI store 488 GB + agent cache 488 GB — still the only node that fits both) and the production lane runs on dgx1+dgx2+dgx3+dgx8.
The 4nodes-switch / prod4b configuration stays in the config for after a drive swap, and the A/B in flare-roce-poison.md is the acceptance test: 157 K first-touch prompt and a full bench on a cold stand.

What to buy instead (if dgx4 must serve)

The requirement is sustained random-read at a large working set.

DRAM-less alone is not the verdict. The healthy reference drive in this fleet (ESL01TBTLCZ) also declares a host memory buffer (64 MB) and stays flat across working sets — what separates it from the KingSpec is the FTL's map-cache behaviour and HMB implementation, not the mere absence of DRAM.

Ordered 2026-11-04: Corsair MP600 Mini 2 TB (M.2 2230). Class per TechPowerUp/KitGuru: Phison PS5027-E27T (4-channel, DRAM-less), 64 MB HMB, 162-layer Kioxia BiCS6 TLC, 1,200 TBW, 5-year warranty — the same architecture family as the problem drive, but a newer and much better regarded controller with a larger HMB. Worth measuring, not assuming.

Naming trap: the MP600 Micro (2242) is the older Phison E21T + Micron 176L design — a controller generation behind the Mini's E27T. The shorter model name is not the better drive.

Alternatives if the Mini fails the acceptance test:

Samsung PM9A3 M.2 2242 (enterprise, Elpis 8-channel, DRAM) or PM9E1 2242 if reachable — designed for exactly this duty.
Corsair MP700 Micro 2 TB (Gen5 2242, Phison E28, 8-channel with DRAM).
Avoid a small HMB plus a budget FTL in this role (XG7000, WD SN770M).

Acceptance test (run before trusting the drive with a lane)

Same harness as below, then the engine. A candidate must clear both:

gate	threshold (healthy class)
4 K random in the 98 GB shard, 16 threads	>= 60 K IOPS and p50 <= 250 us
same, 64 threads	>= 120 K IOPS, p99.9 <= 2 ms (no congestion collapse)
sequential (`dd bs=1M iflag=direct`)	>= 2.5 GB/s (sanity, not the point)
engine: cold-boot 157 K first-touch prompt on `prod4b`	completes, no poison
engine: full 97-scenario bench on the same boot	completes, no poison

The two engine rows are the real test — the harness only predicts them.

Before any reinstall: back up ~/.config/flare/orbit/controller.key — restoring it after a DGX OS reimage avoids re-pairing all eight agents. The package can be re-sourced from dgx1's agent cache (NODE:/...) or re-downloaded (~50 min) rather than copied through the Mac.

Mounting: the GX10 slot is M.2 2242, and a 2230 module mounts on the same single screw (an ASUS Ascent GX10 owner confirmed 2230 support); a 2230->2242 adapter bracket is the fallback, and a thermal pad is worth adding since a 2230 has less surface area than the stock 2242.

Swapping the drive on dgx4 (size trap first)

Measured on 2026-11-04:

device	bytes	TiB
stock `XG7000-2TB` namespace	2,048,408,248,320	1.863
Corsair MP600 Mini 2 TB	2,000,398,934,016	1.819

The replacement is ~48 GB smaller, so a raw sector clone cannot fit: a duplicator either refuses or truncates, and a partial copy leaves a 2.048 TB GPT plus a 1.907 TiB ext4 on a 1.819 TiB device (backup GPT header out of range, filesystem larger than the disk). dgx4's used space is 1.33 TiB, of which 1,215 GB is re-fetchable package store (~/flare-pkgs 477 GB + ~/.cache/flare 738 GB) — only ~10 KB of config is irreplaceable.

Two paths that work:

Fresh DGX OS install + restore (recommended — no offline shrink, no boot-media gymnastics on the controller):
1. Back up ~/.config/flare/ (controller.key + agent identity — without it all eight agents must be re-paired), ~/.ssh, /etc/netplan, /etc/fstab, /etc/hosts. A ready-made tarball of exactly these: ~/dgx4-swap-backup-20261104/dgx4-config.tar.gz (9.6 KB).
2. Install DGX OS on the new module, restore the tarball, reinstall the flare agent and the patched runtime (the same build the other ranks run — a mixed set fails at rendezvous).
3. Re-create the package store with flare download … --revision gb10 --tensor-parallel 4 (~50 min, network). Do not rsync it from dgx1 while dgx1 serves: that node's disk is on the serving critical path.
4. Run the acceptance gates below, then the two engine tests.
Filesystem-aware clone with resize (Clonezilla device-device, which copies used blocks and resizes): keeps OS, keys and both stores unchanged, but needs a USB boot environment and a sgdisk -e/resize2fs pass afterwards. Fine if a duplicator with smart/resizing mode is at hand; a plain raw duplicator is not usable here.

A 4 TB module (instead of 2 TB) sidesteps the size trap entirely and makes a raw clone valid — relevant if the replacement is still returnable.

Reproducing the measurement

# /tmp/lat2.c: N threads, 4 KiB O_DIRECT random reads in a centered working
# set, per-read latency percentiles + >200 ms / >1 s / >5 s counters
cc -O2 -o /tmp/lat2 /tmp/lat2.c -lpthread
F=~/.cache/flare/packages/<pkg>/weights/engram-0-weights.safetensors
/tmp/lat2 "$F" 16 15 2      # hot 2 GB slice
/tmp/lat2 "$F" 16 15 98     # whole 98 GB table
/tmp/lat2 "$F" 64 20 98     # queue-depth collapse
# identity/health
sudo nvme id-ctrl /dev/nvme0 | grep -E '^(mn|fr|hmpre|npss)'
sudo nvme smart-log /dev/nvme0 | grep -E 'media_errors|Temperature|Thermal'
journalctl -k | grep 'host memory buffer'

Run the reference arm on an idle non-serving node (dgx5) — never on a node that is currently serving.

dgx4-nvme-investigation.md 10 KB Historie Surový