# Upstream issue descriptions — 2026-09-09

Draft for the b12x / vLLM fork owner (voipmonitor / local-inference-lab family).
Context: we run the b12x fork stack on 2x/4x DGX Spark (DGX GB10), pinned
builds (r27: vLLM `63a82f8d` / b12x `e8ad299b` / LMCache `822330db`;
r28: vLLM `62f6385c` / b12x `7216f6b9` / LMCache `934c287a`), RoCEnante ON,
GLM-5.3-Flash-NVFP4-Spark (184 GB) as the production model, seed-42 hardmode
tool-eval bench (176-pt scale).

Framing (per operator): the fork side regularly reports the stack "alive and
breathing" — and it is: the **current** sparkring production path (native
MTP3, dual-domain NCCL / mesh, instanttensor loader) does not touch either
blocker below. Both blockers only appear when we combine **b12x O_DIRECT
loader** + **DFlash2 drafter** on the r27/r28 base. Both are therefore
integration/feature-port questions, not "your stack is broken" reports.

---

## 1. b12x O_DIRECT loader: `eh_proj.weight` not in the pool allocation plan

**Symptom** (r28, `--load-format b12x`, GLM-5.3-Flash-NVFP4-Spark, 4x TP4/DCP4):

The O_DIRECT loader itself works — 44 shards streamed, 92.6 GB physical per
node at ~1.5 GB/s (404 s), 40.07 GiB into write-combined shared storage. It
dies at the very last routing step:

```
ValueError: checkpoint destination was not allocated as a weight:
  model.language_model.layers.45.eh_proj.weight
  (b12x/loader/_checkpoint.py:370)
```

`eh_proj` is the GLM-5.3 (Step-3.5-MTP family) extra-head projection linear —
present on every GLM-5.3 layer, absent from the pool allocation plan that
`_checkpoint.py` builds. Everything else (attention, MoE, MLP, embeddings,
the MTP head) routes fine; this one name is missing from the plan.

**Repro**: any GLM-5.3-Flash recipe with `--load-format b12x` (we keep a
pinned test recipe: `glm53-flash-r28-4x-dcp4-b12xload`).

**Impact / our mitigation**: zero for us today — we serve GLM-5.3 with
`--load-format instanttensor` (the default path, works, same model quality).
The b12x loader matters only if the fork wants GLM-5.3 on the O_DIRECT path
(lower host RAM pressure, faster shard streaming). If the allocation plan is
data-driven from the checkpoint config, adding the `eh_proj` linear to the
plan is likely a small fix; if it is a hardcoded per-architecture list, the
GLM-5.3 `GlmMoeDsaForCausalLM` entry needs that extra projection.

---

## 2. DFlash2 drafter: bf16-KV wiring exists in the JJ R8 build, not in r27/r28 base

**Symptom** (r27 and r28, 2x or 4x, DFlash2 drafter enabled):

```
ValueError: No valid attention backend found for
  kv_cache_dtype=fp8_ds_mla, use_mla=False, block_size=256
```

Root cause (verified in the r28 source, `vllm` `62f6385c`):

- The DFlash drafter (`qwen3_dflash.py:200`) inherits its KV dtype from the
  engine-wide cache config (`get_kv_quant_mode(self.kv_cache_dtype)`); the
  GLM-5.3 engine runs `fp8_ds_mla`.
- The DFlash attention backend
  (`vllm/v1/attention/backends/dflash_attn/__init__.py:219,227`) hard-requires
  bfloat16: `q and out must be bfloat16`, `K and V caches must be bfloat16`.
- r28's `fa4720ea` ("restore GLM boundaries across DCP and DFlash groups")
  fixed the prefix-cache/boundary logic across DCP/DFlash groups but not the
  dtype constraint.

**The sparkring reference that proves it is solved elsewhere**: the (now
retired) `JJ R8` GLM-5.3 TP4/DCP4 profile ran exactly this combination —
FP8 KV (24 GiB/rank) **plus** the BF16 DFlash2 draft
(`incoai/GLM-5.3-Flash-DFlash2`, depth 7) — on `local-inference-lab/vllm`
`dev/jovian-judgement` (public tag `sparkring-glm53-flash-gb10-e02b1746`,
commit `e02b1746`) + `voipmonitor/b12x` @ `9ae41c5c`. So a per-drafter
bf16-KV path (or an fp8-capable DFlashAttention) **exists in the JJ R8/R10
line** and simply was not carried into the r27/r28 base.

**Framing question for the owner** (this decides whether there is any work):

sparkring's current production GLM-5.3 profiles are **native MTP3** (the
in-checkpoint MTP head — which is what our r27/r28 production already runs,
MTP k=5), and the DFlash2 profile was **retired** in the README in favor of
it. If native MTP is the long-term path, the DFlash2 dtype wiring is a
non-issue for r27/r28 (we keep DFlash2 recipes as reference only and stop
chasing it). If DFlash2 is still wanted (e.g. for the 386B full model where
MTP may not be in-checkpoint), then the request is a feature port: bring the
JJ R8 per-drafter bf16-KV wiring (or the fp8-capable DFlashAttention) into
the r27/r28 base — a port from a known-good sibling build, not a new
design.

---

## 3. Reference: 386B full model (GLM-5.3-NVFP4-Spark, 406 GiB) — capacity, not a bug

Not an issue report — context for the owner if they are asked about the
full model on Spark clusters: on 4x DGX Spark (121.63 GiB HBM/node) the
weights alone are 101.5 GiB/node; vLLM's restart-on-OOM doubles the load
pass and the nodes go into an unrecoverable OOM-thrash (verified; HA power
cycle required). 5-6+ nodes is the physical requirement. DFlash2 would make
it *worse* (a drafter adds memory, it does not save it), and no DFlash2
variant of the 386B model exists on HF — DFlash2 checkpoints are 1.3 GB
draft models for the 184 GB Flash model only.