Upstream watch — 2026-10-16 (night shift)

Autonomous pass during the 10-15→16 night shift. Focus: the two queued models (Ling-3.1-flash, Step-5-Preview) + repo/forum deltas. Launcher-side only; no new node deploys (cluster powers off at end of shift).

The two queued models — feasibility (the headline)

Ling-3.1-flash (inclusionAI) — arch SUPPORTED on our base, but no small weights yet

Architecture: BailingMoeV3ForCausalLM (model_type: bailing_hybrid, custom_code, fp8). Ant Group's Ling/Bailing MoE-V3 line.
vLLM v0.31.0 (our kindling base) already registers it: BailingMoeV3ForCausalLM → bailing_moe_v3 AND BailingMoeV3MTPModel → bailing_moe_v3_mtp (MTP head supported). So no nightly vLLM needed — the earlier "requires nightly" note is stale for the arch. A Ling lane can ride a v0.31.0-based image (kindling or stock).
BUT the day-0 NVFP4 cut cbfight/Ling-3.1-flash-NVFP4-lossless8 is an EMPTY repo (only .gitattributes, 0 bytes) — reserved, not populated. So the cheap 4x-lane (NVFP4 ≈ GLM-5.3-Flash class) is not available yet.
The only vLLM-loadable Ling today is the original inclusionAI/Ling-3.1-flash (fp8, 572B total, ~689 GB) → TP8+EP8 = all 8 Sparks. Not a 4x lane.
Other cuts are GGUF/EXL3 (llama.cpp / exllamav3), not our vLLM stack: AtomicChat/Ling-3.1-flash-GGUF, AMAImedia/...-BF16-GGUF, vcruz305/...-EXL3-SAGE-*.
Action: watch cbfight (and inclusionAI) for a real NVFP4/FP8-small cut → then a 4x lane on a v0.31.0 base is straightforward (arch already works). A 572B-fp8 8x lane is possible but heavy (689 GB download + all 8 Sparks); not attempted overnight before power-off. Recipe authored as prep: recipes/ling-31-flash-nvfp4/ (targets the future NVFP4 cut; 4x).

Step-5-Preview-NVFP4 (stepfun-ai) — VL model, needs the dedicated image

Architecture: Step5ForConditionalGeneration (model_type: step5_vl) — a vision-language model (8-bit, modelopt/NVFP4), NOT text-only.
vLLM v0.31.0 registers StepVLForConditionalGeneration (step_vl) but not Step5ForConditionalGeneration (step5_vl) — a different, newer arch. So stock v0.31.0 won't load it; needs the dedicated image vllm/vllm-openai:step-5-preview (or a vLLM new enough to carry step5_vl).
604B, ~473 GB → 8 Sparks (TP8). GB10/sm_121a unverified (kernels, UMA memory gate vLLM #48140, GMU). This is a research probe, not a bench target.
Action: recipe authored as prep recipes/step5-preview-nvfp4/ (dedicated image, 8x). A live GB10 probe needs the 8x cluster up + a 473 GB download — deferred to a supervised window (wasteful to pull overnight before power-off).

Repo deltas (GitHub / Docker)

vLLM: latest stable tag still v0.31.0; v0.32.0 is rc0/rc1/rc2 only (no stable). Our kindling base (v0.31.0) is current. Do NOT re-pin overnight — the v0.32.0 stable re-pin + 38-overlay rebase is a supervised, bench-gated task.
HawkBearPig/dgpp: HEAD moved to d9311e72… (our recipe pin is 3ad63d6b…). DGPP is active daily. Consider bumping the pin after reading the changelog (supervised) — the provisioning marker keys on rev, so a bump triggers a clean rebuild.
kindlingai/gr8: HEAD 3df902dc…, active.
SeraphimSerapis/tool-eval-bench: latest tag v2.8.0 = our pin. No newer release; harness stays pinned (do not upgrade casually — re-baseline anchors).
vllm/vllm-openai (Docker Hub): step-5-preview tag is the Step-5 base (dedicated image, per the model's requirement).

DGPP w2 live-run findings (from the provisioning test, 2nodes-dac)

Provisioning (clone-at-pin → release.sh → install → up) now works end-to-end (4 launcher bugs fixed — see engine-drivers.md). DGPP's own preflight then gates on deployment prereqs the launcher does NOT own:

Checkpoint missing — HawkBearPig/GLM-5.3-Flash-NVFP4-FP8 not in the node HF cache. DGPP: python3 scripts/download_model.py --config <file> on rank 0 (or pre-stage the snapshot). ~GLM-5.3-Flash class size.
RoCE lanes — "the transport supports at most two lanes; select DGPP_ROCE_DEVICES explicitly" on multi-fabric hosts. The site .env needs DGPP_ROCE_DEVICES set to the two fabric devices (per the DGPP thread 383406). Not yet encoded in the launcher's generated .env — a candidate follow-up (derive from config nccl.ib_hca, or a recipe [dgpp].roce_devices).
Preflight is strict: any FAIL → "no services started". So w2 needs BOTH the model and the RoCE selection before it serves.

Forum mining (DGPP thread 383406, 107 posts)

Post #106 (DGPP author, the newest release notes = the HEAD d9311e72 that moved past our 3ad63d6b pin) adds: dense Qwen3.8-27B support, three quantization options across Qwen and full GLM-5.3, DGPP_NO_SWAP=1 (launch ranks in a systemd scope that disallows swap — directly relevant to the GB10 unified-memory swap problem), a 1-second decode-gap target (worst mixed- workload pause 1.8s → 0.5s), and faster long-context GLM prefill. This is a strong reason to bump the DGPP pin (supervised — read the full changelog + re-bench; the marker keys on rev, so a bump cleanly rebuilds).
Post #73 (community 2x DGPP): 2× ASUS GX10, driver 580.173.02, CUDA 13.0.88, one ConnectX-7 RoCE lane per node (200G, GID 3) via a switch — matches our fabric. Confirms DGPP wants an explicit single-lane selection.
The thread does NOT document the exact DGPP_ROCE_DEVICES value syntax (only DGPP_NO_SWAP). The device names come from the preflight output (roceP2p1s0f0/f1, rocep1s0f0/f1); the DGPP repo docs/config are the source of truth for the exact env format. Candidate launcher follow-up: a [dgpp].roce_devices recipe field → emit DGPP_ROCE_DEVICES in the .env.

Live 8x probe results (ran overnight 2026-10-16, cluster powered back on)

Both models were already on the NAS HF cache (Ling fp8 534 GB, Step-5 NVFP4 368 GB) — the download was never the blocker (corrected). Served on 8nodes-switch:

Ling-3.1-flash fp8 — FEASIBLE on GB10. Stock vllm/vllm-openai:v0.31.0 (arch built-in), 8 ranks TP8, 534 GB loaded over NFS in ~40 min, reached readiness on :30000, generated correctly. vLLM auto-selected FP8 MoE (DeepGEMM/TRITON) + TRITON_MLA + FLASH_ATTN prefill; no UMA boot gate at max-model-len 65536 / GMU 0.78. A valid bench score needs serving config: --enable-auto-tool-choice --tool-call-parser <bailing> (tool scenarios otherwise 400 on tool_choice=auto) + thinking control (the model's default long reasoning blows the 120 s/turn timeout → 94/97 scenarios excluded, so the first 67/100 is not a real quality number). Follow-up: find the bailing tool-call parser, re-serve with tool config, bench with --no-think.
Step-5-Preview-NVFP4 — NOT runnable on GB10 (hard blocker). The dedicated image resolves Step5ForConditionalGeneration + modelopt_fp4, but the engine aborts at model init: NotImplementedError: Step5 DSA requires SM90 or SM103; got DeviceCapability(major=12, minor=1). GB10 is SM121 (CC 12.1); Step-5's DeepSeek Sparse Attention kernel is built only for SM90 (Hopper) / SM103 (B-series Blackwell). Not a config issue — ruled out on Spark until upstream ships an SM121 DSA path.

Not done overnight (deferred, supervised)

Ling 572B-fp8 8x lane (689 GB + all 8 Sparks) and Step-5 8x GB10 probe (473 GB + dedicated image) — both heavy downloads that a power-off would not waste (NVMe caches persist) but are too large/risky to run unsupervised, and the arch/image prerequisites need a supervised base decision.
DGPP w2 full serve — needs model download + DGPP_ROCE_DEVICES; the launcher side is validated.

Ling tool-calling + making 3.1-flash smaller (fp4) — research

Tool-call / reasoning parsers (for a valid Ling bench)

vLLM v0.31.0 ships a dedicated ling3 tool parser (vllm/tool_parsers/ling3_tool_parser.py → Ling3ToolParser, registered as ling3). The Ling chat template supports enable_thinking (on/off, default on), so the harness --no-think (enable_thinking=false) is honored — that is the fix for the 94/97 timeout failures (the model's default long reasoning blows the 120 s/turn budget). The tool-scenario 400s are fixed by --enable-auto-tool-choice --tool-call-parser ling3. The cookbook also uses --reasoning-parser ling3 (present in the newer cu130 nightly wheel; not in v0.31.0 stable) + --speculative-config '{"method":"mtp",...}'. Added --enable-auto-tool-choice --tool-call-parser ling3 to the Ling recipe (recipes 531013f). Launcher bench does not yet pass --no-think — either run tool-eval-bench directly with --no-think, or add a passthrough flag (follow-up).

FP8 is a first-class format — for serving AND as a quant source

Verified by reading safetensors headers (not the config torch_dtype field):

Qwen/Qwen3.8-27B base = all BF16 (392 tensors); Qwen3.8-27B-FP8 is derived from it. So Qwen HAS a full-precision source.
inclusionAI/Ling-3.1-flash = mixed: 321 BF16 + 393 F8_E4M3 + 507 F32. It is a bf16 model whose heavy MoE linears are stored fp8 (block 128×128, DeepSeek-style); the non-quant parts (attention, shared experts, router, embeddings, lm_head) are already BF16 in the same checkpoint; the F32 tensors are the fp8 block scales (weight_scale_inv) + norms. No separate bf16 repo for 3.1-flash — but bf16 is not required: fp8 is a valid PTQ source (inclusionAI's own Ling-3.0-flash-fp4 records routed_experts_source_quant_method = fp8_block_128x128).

The fp4 format (fully specified by `Ling-3.0-flash-fp4` config)

routed experts → MXFP4 (routed_experts_quant_method = mxfp4, scale_dtype = e8m0, group_size = 32).
dense / shared experts → fp8 block 128×128 (fmt e4m3, activation dynamic).
17 modules kept BF16 (modules_to_not_convert): q_a_proj, q_b_proj, fused_qkv_a_proj_with_mqa, kv_a/kv_b_proj*, b_proj, mlp.gate (router), lm_head, word_embeddings, *.attention.g_proj.
Size: 3.0-flash fp4 = 70.4 GB (vs fp8 128.5 / bf16 255). Note it is MXFP4 (OCP: group 32, e8m0 scale), NOT NVIDIA NVFP4 (group 16, e4m3 scale, as in GLM-5.3-Flash-NVFP4 / Step-5). Both 4-bit, both serve on vLLM, both run on GB10.

Serving fp4 on DGX Spark — solved (inclusionAI `ling-cookbook`)

guide/local-deploy/ling-3.0-flash/dgx-spark-vllm-ling-3-flash-fp4.ipynb: stock vLLM ARM64 cu130 nightly (--torch-backend=cu130, --extra-index-url https://wheels.vllm.ai/nightly/cu130, sm_121) serves the fp4 at TP=1 on ONE Spark (62.4 GB). Flags: --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --gpu-memory-utilization 0.90 --speculative-config '{"method":"mtp", "num_speculative_tokens":3}'. Ling-3.0 arch + ling3 parser are merged into the official wheel (no fork needed). Extrapolated: 3.1-flash fp4 ≈ ~110 GB → one Spark (vs 534 GB fp8 → 8 Sparks). That is the whole "поменьше" payoff: half the expert bytes → fits one box + frees KV budget (the GB10 UMA gate).

The gap — how to CREATE the fp4 (fp8 → mxfp4 experts)

inclusionAI's offline quant tool is NOT public — not in Ling, vllm-ling-v3, or ling-cookbook (only CUDA kernels csrc/.../quantization/fp4/mxfp4_experts_quant.cu + tests). Likely internal (Ant/Quark).
vLLM online mxfp4 (tests/quantization/test_online_mxfp4.py, vllm/model_executor/layers/quantization/online/mxfp4.py) quantizes bf16/fp16 → mxfp4, NOT fp8. Our 3.1-flash experts are F8_E4M3, so --quantization mxfp4 does not apply directly (it expects bf16 weights).
So "just flag our fp8 to mxfp4" does NOT work — the source dtype is wrong.

Realistic path to a smaller 3.1-flash (fp4)

Write our own fp8-block-experts → mxfp4-experts converter (exactly what inclusionAI did for 3.0). The transform is fully determined:

From the NAS checkpoint: expert tensors F8_E4M3 + their fp8 block scales → dequant to bf16 (using weight_scale_inv).
Requant to MXFP4 (group 32, e8m0 scale). Reference impl in vLLM: tests/quantization/reference_mxfp4.py + kernel mxfp4_experts_quant.cu.
Keep non-expert tensors as-is (dense fp8, the 17 bf16 modules); write the config with the routed_experts_* keys (copy 3.0-flash-fp4's, adjust for 72 layers).
Bench the quality delta (MXFP4 experts → small cost). Well-defined but a real engineering task — the natural project #2 after the quantization warm-up.

Alternatives for "smaller" now

GGUF Q4 via llama.cpp — AtomicChat/Ling-3.1-flash-GGUF already on HF + cookbook dgx-spark-llamacpp-ling-3-flash-q4-gguf.ipynb (for 3.0). Smallest cut, but a different stack (llama.cpp, not our vLLM serve line).
Wait for an official 3.1-flash fp4 — inclusionAI has fp4 only for 3.0 so far; 3.1 may follow.

Learning plan (operator wants to learn NVFP4/fp4 PTQ — no prior experience)

Warm-up on Qwen4-27B (when it lands) — clean bf16 source + an official ModelOpt recipe exists for the Qwen line → run the whole install → calibrate → recipe YAML → export → vllm serve --quantization modelopt_fp4 → bench-delta loop on ONE Spark. Low risk, covers 100% of the pipeline. (Deferred until Qwen4-27B is out.)
Ling-3.1-flash fp4 converter (project #2) — the fp8→mxfp4-experts path above; the format + reference impl are the spec.

Key reference files to study: inclusionAI/Ling-3.0-flash-fp4 config.json (the format), ling-cookbook dgx-spark-vllm-ling-3-flash-fp4 (serving), vllm tests/quantization/reference_mxfp4.py + vllm/model_executor/layers/quantization/online/mxfp4.py + csrc/.../fp4/mxfp4_experts_quant.cu (the mxfp4 math + kernel).

upstream-watch-2026-10-16.md 14 KB 永久連結 文件歷史 原始文件