|
|
@@ -0,0 +1,105 @@
|
|
|
+# Upstream issue descriptions — 2026-09-09
|
|
|
+
|
|
|
+Draft for the b12x / vLLM fork owner (voipmonitor / local-inference-lab family).
|
|
|
+Context: we run the b12x fork stack on 2x/4x DGX Spark (DGX GB10), pinned
|
|
|
+builds (r27: vLLM `63a82f8d` / b12x `e8ad299b` / LMCache `822330db`;
|
|
|
+r28: vLLM `62f6385c` / b12x `7216f6b9` / LMCache `934c287a`), RoCEnante ON,
|
|
|
+GLM-5.3-Flash-NVFP4-Spark (184 GB) as the production model, seed-42 hardmode
|
|
|
+tool-eval bench (176-pt scale).
|
|
|
+
|
|
|
+Framing (per operator): the fork side regularly reports the stack "alive and
|
|
|
+breathing" — and it is: the **current** sparkring production path (native
|
|
|
+MTP3, dual-domain NCCL / mesh, instanttensor loader) does not touch either
|
|
|
+blocker below. Both blockers only appear when we combine **b12x O_DIRECT
|
|
|
+loader** + **DFlash2 drafter** on the r27/r28 base. Both are therefore
|
|
|
+integration/feature-port questions, not "your stack is broken" reports.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 1. b12x O_DIRECT loader: `eh_proj.weight` not in the pool allocation plan
|
|
|
+
|
|
|
+**Symptom** (r28, `--load-format b12x`, GLM-5.3-Flash-NVFP4-Spark, 4x TP4/DCP4):
|
|
|
+
|
|
|
+The O_DIRECT loader itself works — 44 shards streamed, 92.6 GB physical per
|
|
|
+node at ~1.5 GB/s (404 s), 40.07 GiB into write-combined shared storage. It
|
|
|
+dies at the very last routing step:
|
|
|
+
|
|
|
+```
|
|
|
+ValueError: checkpoint destination was not allocated as a weight:
|
|
|
+ model.language_model.layers.45.eh_proj.weight
|
|
|
+ (b12x/loader/_checkpoint.py:370)
|
|
|
+```
|
|
|
+
|
|
|
+`eh_proj` is the GLM-5.3 (Step-3.5-MTP family) extra-head projection linear —
|
|
|
+present on every GLM-5.3 layer, absent from the pool allocation plan that
|
|
|
+`_checkpoint.py` builds. Everything else (attention, MoE, MLP, embeddings,
|
|
|
+the MTP head) routes fine; this one name is missing from the plan.
|
|
|
+
|
|
|
+**Repro**: any GLM-5.3-Flash recipe with `--load-format b12x` (we keep a
|
|
|
+pinned test recipe: `glm53-flash-r28-4x-dcp4-b12xload`).
|
|
|
+
|
|
|
+**Impact / our mitigation**: zero for us today — we serve GLM-5.3 with
|
|
|
+`--load-format instanttensor` (the default path, works, same model quality).
|
|
|
+The b12x loader matters only if the fork wants GLM-5.3 on the O_DIRECT path
|
|
|
+(lower host RAM pressure, faster shard streaming). If the allocation plan is
|
|
|
+data-driven from the checkpoint config, adding the `eh_proj` linear to the
|
|
|
+plan is likely a small fix; if it is a hardcoded per-architecture list, the
|
|
|
+GLM-5.3 `GlmMoeDsaForCausalLM` entry needs that extra projection.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 2. DFlash2 drafter: bf16-KV wiring exists in the JJ R8 build, not in r27/r28 base
|
|
|
+
|
|
|
+**Symptom** (r27 and r28, 2x or 4x, DFlash2 drafter enabled):
|
|
|
+
|
|
|
+```
|
|
|
+ValueError: No valid attention backend found for
|
|
|
+ kv_cache_dtype=fp8_ds_mla, use_mla=False, block_size=256
|
|
|
+```
|
|
|
+
|
|
|
+Root cause (verified in the r28 source, `vllm` `62f6385c`):
|
|
|
+
|
|
|
+- The DFlash drafter (`qwen3_dflash.py:200`) inherits its KV dtype from the
|
|
|
+ engine-wide cache config (`get_kv_quant_mode(self.kv_cache_dtype)`); the
|
|
|
+ GLM-5.3 engine runs `fp8_ds_mla`.
|
|
|
+- The DFlash attention backend
|
|
|
+ (`vllm/v1/attention/backends/dflash_attn/__init__.py:219,227`) hard-requires
|
|
|
+ bfloat16: `q and out must be bfloat16`, `K and V caches must be bfloat16`.
|
|
|
+- r28's `fa4720ea` ("restore GLM boundaries across DCP and DFlash groups")
|
|
|
+ fixed the prefix-cache/boundary logic across DCP/DFlash groups but not the
|
|
|
+ dtype constraint.
|
|
|
+
|
|
|
+**The sparkring reference that proves it is solved elsewhere**: the (now
|
|
|
+retired) `JJ R8` GLM-5.3 TP4/DCP4 profile ran exactly this combination —
|
|
|
+FP8 KV (24 GiB/rank) **plus** the BF16 DFlash2 draft
|
|
|
+(`incoai/GLM-5.3-Flash-DFlash2`, depth 7) — on `local-inference-lab/vllm`
|
|
|
+`dev/jovian-judgement` (public tag `sparkring-glm53-flash-gb10-e02b1746`,
|
|
|
+commit `e02b1746`) + `voipmonitor/b12x` @ `9ae41c5c`. So a per-drafter
|
|
|
+bf16-KV path (or an fp8-capable DFlashAttention) **exists in the JJ R8/R10
|
|
|
+line** and simply was not carried into the r27/r28 base.
|
|
|
+
|
|
|
+**Framing question for the owner** (this decides whether there is any work):
|
|
|
+
|
|
|
+sparkring's current production GLM-5.3 profiles are **native MTP3** (the
|
|
|
+in-checkpoint MTP head — which is what our r27/r28 production already runs,
|
|
|
+MTP k=5), and the DFlash2 profile was **retired** in the README in favor of
|
|
|
+it. If native MTP is the long-term path, the DFlash2 dtype wiring is a
|
|
|
+non-issue for r27/r28 (we keep DFlash2 recipes as reference only and stop
|
|
|
+chasing it). If DFlash2 is still wanted (e.g. for the 386B full model where
|
|
|
+MTP may not be in-checkpoint), then the request is a feature port: bring the
|
|
|
+JJ R8 per-drafter bf16-KV wiring (or the fp8-capable DFlashAttention) into
|
|
|
+the r27/r28 base — a port from a known-good sibling build, not a new
|
|
|
+design.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 3. Reference: 386B full model (GLM-5.3-NVFP4-Spark, 406 GiB) — capacity, not a bug
|
|
|
+
|
|
|
+Not an issue report — context for the owner if they are asked about the
|
|
|
+full model on Spark clusters: on 4x DGX Spark (121.63 GiB HBM/node) the
|
|
|
+weights alone are 101.5 GiB/node; vLLM's restart-on-OOM doubles the load
|
|
|
+pass and the nodes go into an unrecoverable OOM-thrash (verified; HA power
|
|
|
+cycle required). 5-6+ nodes is the physical requirement. DFlash2 would make
|
|
|
+it *worse* (a drafter adds memory, it does not save it), and no DFlash2
|
|
|
+variant of the 386B model exists on HF — DFlash2 checkpoints are 1.3 GB
|
|
|
+draft models for the 184 GB Flash model only.
|
