# dgx4's NVMe: why the Spark wedge is one bad drive (2026-11-04)

The FLARE lane on dgx1–4 died repeatedly with `RoCE collective … timed out …
the transport is poisoned` (see [`flare-roce-poison.md`](flare-roce-poison.md)).
The trigger is not the fabric, not the rank order, not the engine: it is
**dgx4's boot SSD**, and it is not a fault — the drive is healthy, it is simply
the wrong class for the workload.

## The drive

| | dgx4 | every other node |
|---|---|---|
| model | **KingSpec `XG7000-2TB 2242`** (`/sys/class/nvme/nvme0/model`) | `ESL01TBTLCZ-27J2-TYN` 1 TB |
| firmware | `GT0d9d63` (vid `0x1e4b`, not in the public firmware DB) | `ERFM12.0` |
| PCIe | Gen4 x4, 16 GT/s, ASPM disabled, no AER errors | Gen4 x4 |
| HMB | **`hmpre`/`hmmin` 10240 = 40 MB, and it is allocated** (`nvme nvme0: allocated 40 MiB host memory buffer (1 segment)`) | declares 64 MB |
| power states | 5 (`npss 4`), APST on, idles in **PS3** | 5, APST on |
| health | 96 h powered, 54.9 TB read / 13.4 TB written, `media_errors 0`, 0 error-log entries, 0 thermal-throttle events, 42 °C | healthy |

Vendor identity (community teardowns/press): KingSpec XG7000 = **DRAM-less**
design, **Maxio MAP1602A** controller, **232-layer YMTC TLC**, graphene
heat-spacer instead of a heatsink, no first-party management or firmware tool.
It is a cheap "fast sequential" gaming drive; "DRAM-less" is the operative word
— the flash-translation mapping table lives in the 40 MB host buffer plus the
controller's small SRAM, not in on-drive DRAM.

## What it does under our workload

Measured with a 4 KiB `O_DIRECT` random-read harness on the package's Engram
shard (98 GB), reporting per-read completion latency:

| working set (random 4 K, 16 threads, 15 s) | dgx4 XG7000 | dgx5 ESL01 |
|---|---|---|
| 2 GB | 68.8 K IOPS, p50 **69 µs** | 113 K IOPS, p50 109 µs |
| 20 GB | 38.8 K IOPS, p50 169 µs | 112 K IOPS, p50 110 µs |
| 98 GB | **12.9 K IOPS, p50 1,183 µs**, p99.9 4.6 ms | 80 K IOPS, p50 200 µs |
| 98 GB, 64 threads | **24.8 K IOPS, p50 2.45 ms**, p99.9 10.5 ms | **204 K IOPS**, p50 283 µs |

Two things stand out:

1. **Throughput collapses as the working set grows** (69 K → 13 K IOPS) while
   the healthy drive stays flat (113 K → 80 K). That is the mapping cache:
   98 GB / 4 KiB = 24 M mappings; even at 8 B per entry that is ~190 MB of
   table against a 40 MB HMB, so a random read over the whole table is a
   mapping miss — an extra NAND fetch — most of the time. Latency p50 goes
   from 69 µs (hot 2 GB slice, table fits) to 1.2–2.5 ms (whole table).
2. **It degrades with queue depth instead of scaling**: 16 → 64 threads moves
   it 12.9 K → 24.8 K IOPS but p50 1.18 → 2.45 ms (congestion collapse). The
   healthy drive goes 80 K → 204 K IOPS and holds sub-millisecond p50.

Sequential is fine, which is why nothing noticed before: streaming 4 GiB with
`iflag=direct` reads **1.9 GB/s** on dgx4 vs 3.0 GB/s on dgx5 (~63 %). Package
copies, image pulls and `flare download` are all sequential, and the weight
load is read once into GPU memory.

## Why that kills the engine (the arithmetic)

FLARE serves this package with disk-backed Engram tables and reads rows **at
token time**. From the production lane's own metrics: a 157 K-token prefill
did **6.49 M Engram reads** → **≈41 reads/token**. A 4096-token prefill chunk
(the package's baked geometry) therefore needs **≈170 K random reads ≈ 680 MB**.

| node | sustained random read at 98 GB | 680 MB chunk |
|---|---|---|
| dgx5-class drive | ~330–800 MB/s | **1–2 s** |
| dgx4 XG7000 | ~52–99 MB/s | **7–13 s** |

The collective's spin limit is ~8–20 s (20 M polls in the package's kernel),
and it covers a whole chunk. A device that needs 7–13 s per chunk with a tail
that spikes to 10–18 ms p99.9 sits right on that boundary — which is exactly
what we see: healthy on a warm stand, dead on a cold boot (`seq 93`,
prefill = 1 token) and dead again after sustained load (`seq 2833389` after a
full bench). The upstream reader patch (PR #184, io_uring lanes) raises the
reader's parallelism but cannot raise the device's ceiling, which is why it
helps on the warm/cached path and does not save the cold or sustained cases.

## What this is not

- **Not a failing drive**: `media_errors 0`, zero error-log entries, no
  thermal-throttle transitions, 42 °C, and 96 power-on hours. It answers every
  I/O correctly, just slowly under random load. SMART will never flag it.
- **Not thermal**: no T1/T2 management transitions; the graphene pad is enough
  for a 6 W-class drive.
- **Not ASPM/APST flakiness**: the link stays Gen4 x4 with ASPM disabled;
  idle wake is single-digit milliseconds, not seconds.
- **Not the PCIe slot**: the same slot class reaches 3.0 GB/s sequential on the
  other nodes' drives; the /usr/bin and root filesystem are unaffected.

## Decisions taken (2026-11-04)

- dgx4 does **not** serve. It keeps the 2 TB disk as the orbit controller and
  package store (CLI store 488 GB + agent cache 488 GB — still the only node
  that fits both) and the production lane runs on dgx1+dgx2+dgx3+dgx8.
- The `4nodes-switch` / `prod4b` configuration stays in the config for after a
  drive swap, and the A/B in `flare-roce-poison.md` is the acceptance test:
  157 K first-touch prompt **and** a full bench on a cold stand.

## What to buy instead (if dgx4 must serve)

The requirement is sustained random-read at a large working set.

**DRAM-less alone is not the verdict.** The healthy reference drive in this
fleet (`ESL01TBTLCZ`) *also* declares a host memory buffer (64 MB) and stays
flat across working sets — what separates it from the KingSpec is the FTL's
map-cache behaviour and HMB implementation, not the mere absence of DRAM.

Ordered 2026-11-04: **Corsair MP600 Mini 2 TB** (M.2 2230). Class per
TechPowerUp/KitGuru: Phison **PS5027-E27T** (4-channel, DRAM-less), **64 MB
HMB**, 162-layer Kioxia BiCS6 **TLC**, 1,200 TBW, 5-year warranty — the same
architecture family as the problem drive, but a newer and much better regarded
controller with a larger HMB. Worth measuring, not assuming.

Naming trap: the **MP600 Micro** (2242) is the *older* Phison **E21T** +
Micron 176L design — a controller generation behind the Mini's E27T. The
shorter model name is not the better drive.

Alternatives if the Mini fails the acceptance test:

1. **Samsung PM9A3 M.2 2242** (enterprise, Elpis 8-channel, DRAM) or
   **PM9E1 2242** if reachable — designed for exactly this duty.
2. **Corsair MP700 Micro 2 TB** (Gen5 2242, Phison E28, 8-channel with DRAM).
3. Avoid a small HMB plus a budget FTL in this role (XG7000, WD SN770M).

### Acceptance test (run before trusting the drive with a lane)

Same harness as below, then the engine. A candidate must clear both:

| gate | threshold (healthy class) |
|---|---|
| 4 K random in the 98 GB shard, 16 threads | **>= 60 K IOPS and p50 <= 250 us** |
| same, 64 threads | **>= 120 K IOPS, p99.9 <= 2 ms** (no congestion collapse) |
| sequential (`dd bs=1M iflag=direct`) | >= 2.5 GB/s (sanity, not the point) |
| **engine**: cold-boot 157 K first-touch prompt on `prod4b` | completes, no poison |
| **engine**: full 97-scenario bench on the same boot | completes, no poison |

The two engine rows are the real test — the harness only predicts them.

Before any reinstall: back up `~/.config/flare/orbit/controller.key` — restoring
it after a DGX OS reimage avoids re-pairing all eight agents. The package can
be re-sourced from dgx1's agent cache (`NODE:/...`) or re-downloaded (~50 min)
rather than copied through the Mac.

Mounting: the GX10 slot is M.2 2242, and a 2230 module mounts on the same
single screw (an ASUS Ascent GX10 owner confirmed 2230 support); a 2230->2242
adapter bracket is the fallback, and a thermal pad is worth adding since a 2230
has less surface area than the stock 2242.

## Swapping the drive on dgx4 (size trap first)

Measured on 2026-11-04:

| device | bytes | TiB |
|---|---|---|
| stock `XG7000-2TB` namespace | 2,048,408,248,320 | 1.863 |
| Corsair MP600 Mini 2 TB | 2,000,398,934,016 | 1.819 |

The replacement is **~48 GB smaller**, so a **raw sector clone cannot fit**:
a duplicator either refuses or truncates, and a partial copy leaves a 2.048 TB
GPT plus a 1.907 TiB ext4 on a 1.819 TiB device (backup GPT header out of
range, filesystem larger than the disk). dgx4's used space is 1.33 TiB, of
which **1,215 GB is re-fetchable** package store (`~/flare-pkgs` 477 GB +
`~/.cache/flare` 738 GB) — only ~10 KB of config is irreplaceable.

Two paths that work:

1. **Fresh DGX OS install + restore** (recommended — no offline shrink, no
   boot-media gymnastics on the controller):
   1. Back up `~/.config/flare/` (**controller.key + agent identity — without
      it all eight agents must be re-paired**), `~/.ssh`, `/etc/netplan`,
      `/etc/fstab`, `/etc/hosts`. A ready-made tarball of exactly these:
      `~/dgx4-swap-backup-20261104/dgx4-config.tar.gz` (9.6 KB).
   2. Install DGX OS on the new module, restore the tarball, reinstall the
      `flare` agent and the patched runtime (the same build the other ranks
      run — a mixed set fails at rendezvous).
   3. Re-create the package store with `flare download … --revision gb10
      --tensor-parallel 4` (~50 min, network). **Do not rsync it from dgx1
      while dgx1 serves**: that node's disk is on the serving critical path.
   4. Run the acceptance gates below, then the two engine tests.
2. **Filesystem-aware clone with resize** (Clonezilla `device-device`, which
   copies used blocks and resizes): keeps OS, keys and both stores unchanged,
   but needs a USB boot environment and a `sgdisk -e`/`resize2fs` pass
   afterwards. Fine if a duplicator with smart/resizing mode is at hand; a
   plain raw duplicator is not usable here.

A 4 TB module (instead of 2 TB) sidesteps the size trap entirely and makes a
raw clone valid — relevant if the replacement is still returnable.

## Reproducing the measurement

```bash
# /tmp/lat2.c: N threads, 4 KiB O_DIRECT random reads in a centered working
# set, per-read latency percentiles + >200 ms / >1 s / >5 s counters
cc -O2 -o /tmp/lat2 /tmp/lat2.c -lpthread
F=~/.cache/flare/packages/<pkg>/weights/engram-0-weights.safetensors
/tmp/lat2 "$F" 16 15 2      # hot 2 GB slice
/tmp/lat2 "$F" 16 15 98     # whole 98 GB table
/tmp/lat2 "$F" 64 20 98     # queue-depth collapse
# identity/health
sudo nvme id-ctrl /dev/nvme0 | grep -E '^(mn|fr|hmpre|npss)'
sudo nvme smart-log /dev/nvme0 | grep -E 'media_errors|Temperature|Thermal'
journalctl -k | grep 'host memory buffer'
```

Run the reference arm on an idle non-serving node (dgx5) — never on a node that
is currently serving.