launcher/specs/tech-architecture/upstream-issues-2026-09-09.md

Upstream issue descriptions — 2026-09-09

Draft for the b12x / vLLM fork owner (voipmonitor / local-inference-lab family). Context: we run the b12x fork stack on 2x/4x DGX Spark (DGX GB10), pinned builds (r27: vLLM 63a82f8d / b12x e8ad299b / LMCache 822330db; r28: vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a), RoCEnante ON, GLM-5.3-Flash-NVFP4-Spark (184 GB) as the production model, seed-42 hardmode tool-eval bench (176-pt scale).

Framing (per operator): the fork side regularly reports the stack "alive and breathing" — and it is: the current sparkring production path (native MTP3, dual-domain NCCL / mesh, instanttensor loader) does not touch either blocker below. Both blockers only appear when we combine b12x O_DIRECT loader + DFlash2 drafter on the r27/r28 base. Both are therefore integration/feature-port questions, not "your stack is broken" reports.

---

1. b12x O_DIRECT loader: eh_proj.weight not in the pool allocation plan

Symptom (r28, --load-format b12x, GLM-5.3-Flash-NVFP4-Spark, 4x TP4/DCP4):

The O_DIRECT loader itself works — 44 shards streamed, 92.6 GB physical per node at ~1.5 GB/s (404 s), 40.07 GiB into write-combined shared storage. It dies at the very last routing step:

ValueError: checkpoint destination was not allocated as a weight:
  model.language_model.layers.45.eh_proj.weight
  (b12x/loader/_checkpoint.py:370)

eh_proj is the GLM-5.3 (Step-3.5-MTP family) extra-head projection linear — present on every GLM-5.3 layer, absent from the pool allocation plan that _checkpoint.py builds. Everything else (attention, MoE, MLP, embeddings, the MTP head) routes fine; this one name is missing from the plan.

Repro: any GLM-5.3-Flash recipe with --load-format b12x (we keep a pinned test recipe: glm53-flash-r28-4x-dcp4-b12xload).

Impact / our mitigation: zero for us today — we serve GLM-5.3 with --load-format instanttensor (the default path, works, same model quality). The b12x loader matters only if the fork wants GLM-5.3 on the O_DIRECT path (lower host RAM pressure, faster shard streaming). If the allocation plan is data-driven from the checkpoint config, adding the eh_proj linear to the plan is likely a small fix; if it is a hardcoded per-architecture list, the GLM-5.3 GlmMoeDsaForCausalLM entry needs that extra projection.

---

2. DFlash2 drafter: bf16-KV wiring exists in the JJ R8 build, not in r27/r28 base

Symptom (r27 and r28, 2x or 4x, DFlash2 drafter enabled):

ValueError: No valid attention backend found for
  kv_cache_dtype=fp8_ds_mla, use_mla=False, block_size=256

Root cause (verified in the r28 source, vllm 62f6385c):

• The DFlash drafter (qwen3_dflash.py:200) inherits its KV dtype from the engine-wide cache config (get_kv_quant_mode(self.kv_cache_dtype)); the GLM-5.3 engine runs fp8_ds_mla.
• The DFlash attention backend (vllm/v1/attention/backends/dflash_attn/__init__.py:219,227) hard-requires bfloat16: q and out must be bfloat16, K and V caches must be bfloat16.
• r28's fa4720ea ("restore GLM boundaries across DCP and DFlash groups") fixed the prefix-cache/boundary logic across DCP/DFlash groups but not the dtype constraint.

The sparkring reference that proves it is solved elsewhere: the (now retired) JJ R8 GLM-5.3 TP4/DCP4 profile ran exactly this combination — FP8 KV (24 GiB/rank) plus the BF16 DFlash2 draft (incoai/GLM-5.3-Flash-DFlash2, depth 7) — on local-inference-lab/vllm dev/jovian-judgement (public tag sparkring-glm53-flash-gb10-e02b1746, commit e02b1746) + voipmonitor/b12x @ 9ae41c5c. So a per-drafter bf16-KV path (or an fp8-capable DFlashAttention) exists in the JJ R8/R10 line and simply was not carried into the r27/r28 base.

Framing question for the owner (this decides whether there is any work):

sparkring's current production GLM-5.3 profiles are native MTP3 (the in-checkpoint MTP head — which is what our r27/r28 production already runs, MTP k=5), and the DFlash2 profile was retired in the README in favor of it. If native MTP is the long-term path, the DFlash2 dtype wiring is a non-issue for r27/r28 (we keep DFlash2 recipes as reference only and stop chasing it). If DFlash2 is still wanted (e.g. for the 386B full model where MTP may not be in-checkpoint), then the request is a feature port: bring the JJ R8 per-drafter bf16-KV wiring (or the fp8-capable DFlashAttention) into the r27/r28 base — a port from a known-good sibling build, not a new design.

---

3. Reference: 386B full model (GLM-5.3-NVFP4-Spark, 406 GiB) — capacity, not a bug

Not an issue report — context for the owner if they are asked about the full model on Spark clusters: on 4x DGX Spark (121.63 GiB HBM/node) the weights alone are 101.5 GiB/node; vLLM's restart-on-OOM doubles the load pass and the nodes go into an unrecoverable OOM-thrash (verified; HA power cycle required). 5-6+ nodes is the physical requirement. DFlash2 would make it worse (a drafter adds memory, it does not save it), and no DFlash2 variant of the 386B model exists on HF — DFlash2 checkpoints are 1.3 GB draft models for the 184 GB Flash model only.