# Upstream watch — 2026-10-16 (night shift)

Autonomous pass during the 10-15→16 night shift. Focus: the two queued models
(Ling-3.1-flash, Step-5-Preview) + repo/forum deltas. Launcher-side only; no
new node deploys (cluster powers off at end of shift).

## The two queued models — feasibility (the headline)

### Ling-3.1-flash (inclusionAI) — arch SUPPORTED on our base, but no small weights yet
- Architecture: **`BailingMoeV3ForCausalLM`** (`model_type: bailing_hybrid`,
  `custom_code`, fp8). Ant Group's Ling/Bailing MoE-V3 line.
- **vLLM v0.31.0 (our kindling base) already registers it**:
  `BailingMoeV3ForCausalLM → bailing_moe_v3` AND `BailingMoeV3MTPModel →
  bailing_moe_v3_mtp` (MTP head supported). So **no nightly vLLM needed** — the
  earlier "requires nightly" note is stale for the arch. A Ling lane can ride a
  v0.31.0-based image (kindling or stock).
- BUT the day-0 **NVFP4 cut `cbfight/Ling-3.1-flash-NVFP4-lossless8` is an
  EMPTY repo** (only `.gitattributes`, 0 bytes) — reserved, not populated. So
  the cheap 4x-lane (NVFP4 ≈ GLM-5.3-Flash class) is **not available yet**.
- The only vLLM-loadable Ling today is the **original `inclusionAI/Ling-3.1-flash`**
  (fp8, 572B total, ~689 GB) → **TP8+EP8 = all 8 Sparks**. Not a 4x lane.
- Other cuts are GGUF/EXL3 (llama.cpp / exllamav3), not our vLLM stack:
  `AtomicChat/Ling-3.1-flash-GGUF`, `AMAImedia/...-BF16-GGUF`,
  `vcruz305/...-EXL3-SAGE-*`.
- **Action:** watch `cbfight` (and inclusionAI) for a real NVFP4/FP8-small cut →
  then a 4x lane on a v0.31.0 base is straightforward (arch already works).
  A 572B-fp8 8x lane is possible but heavy (689 GB download + all 8 Sparks);
  not attempted overnight before power-off. Recipe authored as prep:
  `recipes/ling-31-flash-nvfp4/` (targets the future NVFP4 cut; 4x).

### Step-5-Preview-NVFP4 (stepfun-ai) — VL model, needs the dedicated image
- Architecture: **`Step5ForConditionalGeneration`** (`model_type: step5_vl`) —
  a **vision-language** model (8-bit, modelopt/NVFP4), NOT text-only.
- vLLM v0.31.0 registers `StepVLForConditionalGeneration` (`step_vl`) but **not**
  `Step5ForConditionalGeneration` (`step5_vl`) — a different, newer arch. So
  stock v0.31.0 won't load it; needs the **dedicated image
  `vllm/vllm-openai:step-5-preview`** (or a vLLM new enough to carry step5_vl).
- 604B, ~473 GB → 8 Sparks (TP8). **GB10/sm_121a unverified** (kernels, UMA
  memory gate vLLM #48140, GMU). This is a research probe, not a bench target.
- **Action:** recipe authored as prep `recipes/step5-preview-nvfp4/` (dedicated
  image, 8x). A live GB10 probe needs the 8x cluster up + a 473 GB download —
  deferred to a supervised window (wasteful to pull overnight before power-off).

## Repo deltas (GitHub / Docker)

- **vLLM**: latest stable tag still **v0.31.0**; v0.32.0 is **rc0/rc1/rc2** only
  (no stable). Our kindling base (v0.31.0) is current. Do NOT re-pin overnight —
  the v0.32.0 stable re-pin + 38-overlay rebase is a supervised, bench-gated task.
- **HawkBearPig/dgpp**: HEAD moved to **`d9311e72…`** (our recipe pin is
  `3ad63d6b…`). DGPP is active daily. Consider bumping the pin after reading the
  changelog (supervised) — the provisioning marker keys on `rev`, so a bump
  triggers a clean rebuild.
- **kindlingai/gr8**: HEAD `3df902dc…`, active.
- **SeraphimSerapis/tool-eval-bench**: latest tag **v2.8.0** = our pin. No newer
  release; harness stays pinned (do not upgrade casually — re-baseline anchors).
- **vllm/vllm-openai** (Docker Hub): `step-5-preview` tag is the Step-5 base
  (dedicated image, per the model's requirement).

## DGPP w2 live-run findings (from the provisioning test, 2nodes-dac)

Provisioning (clone-at-pin → release.sh → install → up) now works end-to-end
(4 launcher bugs fixed — see `engine-drivers.md`). DGPP's own preflight then
gates on deployment prereqs the launcher does NOT own:
1. **Checkpoint missing** — `HawkBearPig/GLM-5.3-Flash-NVFP4-FP8` not in the
   node HF cache. DGPP: `python3 scripts/download_model.py --config <file>` on
   rank 0 (or pre-stage the snapshot). ~GLM-5.3-Flash class size.
2. **RoCE lanes** — "the transport supports at most two lanes; select
   `DGPP_ROCE_DEVICES` explicitly" on multi-fabric hosts. The site `.env` needs
   `DGPP_ROCE_DEVICES` set to the two fabric devices (per the DGPP thread
   383406). Not yet encoded in the launcher's generated `.env` — a candidate
   follow-up (derive from config `nccl.ib_hca`, or a recipe `[dgpp].roce_devices`).
3. Preflight is strict: any FAIL → "no services started". So w2 needs BOTH the
   model and the RoCE selection before it serves.

## Forum mining (DGPP thread 383406, 107 posts)

- **Post #106 (DGPP author, the newest release notes = the HEAD `d9311e72` that
  moved past our `3ad63d6b` pin)** adds: **dense Qwen3.8-27B** support, **three
  quantization options across Qwen and full GLM-5.3**, `DGPP_NO_SWAP=1` (launch
  ranks in a systemd scope that disallows swap — directly relevant to the GB10
  unified-memory swap problem), a 1-second decode-gap target (worst mixed-
  workload pause 1.8s → 0.5s), and faster long-context GLM prefill. **This is a
  strong reason to bump the DGPP pin** (supervised — read the full changelog +
  re-bench; the marker keys on `rev`, so a bump cleanly rebuilds).
- **Post #73 (community 2x DGPP)**: 2× ASUS GX10, driver 580.173.02, CUDA
  13.0.88, **one ConnectX-7 RoCE lane per node (200G, GID 3) via a switch** —
  matches our fabric. Confirms DGPP wants an explicit single-lane selection.
- The thread does NOT document the exact `DGPP_ROCE_DEVICES` value syntax (only
  `DGPP_NO_SWAP`). The device names come from the preflight output
  (`roceP2p1s0f0/f1`, `rocep1s0f0/f1`); the DGPP repo docs/config are the source
  of truth for the exact env format. Candidate launcher follow-up: a
  `[dgpp].roce_devices` recipe field → emit `DGPP_ROCE_DEVICES` in the `.env`.

## Live 8x probe results (ran overnight 2026-10-16, cluster powered back on)

Both models were already on the NAS HF cache (Ling fp8 534 GB, Step-5 NVFP4
368 GB) — the download was never the blocker (corrected). Served on 8nodes-switch:

- **Ling-3.1-flash fp8 — FEASIBLE on GB10.** Stock `vllm/vllm-openai:v0.31.0`
  (arch built-in), 8 ranks TP8, 534 GB loaded over NFS in ~40 min, reached
  readiness on :30000, generated correctly. vLLM auto-selected FP8 MoE
  (DeepGEMM/TRITON) + TRITON_MLA + FLASH_ATTN prefill; no UMA boot gate at
  max-model-len 65536 / GMU 0.78. **A valid bench score needs serving config**:
  `--enable-auto-tool-choice --tool-call-parser <bailing>` (tool scenarios
  otherwise 400 on `tool_choice=auto`) + thinking control (the model's default
  long reasoning blows the 120 s/turn timeout → 94/97 scenarios excluded, so the
  first 67/100 is not a real quality number). Follow-up: find the bailing
  tool-call parser, re-serve with tool config, bench with `--no-think`.
- **Step-5-Preview-NVFP4 — NOT runnable on GB10 (hard blocker).** The dedicated
  image resolves `Step5ForConditionalGeneration` + `modelopt_fp4`, but the engine
  aborts at model init: `NotImplementedError: Step5 DSA requires SM90 or SM103;
  got DeviceCapability(major=12, minor=1)`. GB10 is SM121 (CC 12.1); Step-5's
  DeepSeek Sparse Attention kernel is built only for SM90 (Hopper) / SM103
  (B-series Blackwell). Not a config issue — ruled out on Spark until upstream
  ships an SM121 DSA path.

## Not done overnight (deferred, supervised)

- Ling 572B-fp8 8x lane (689 GB + all 8 Sparks) and Step-5 8x GB10 probe
  (473 GB + dedicated image) — both heavy downloads that a power-off would not
  waste (NVMe caches persist) but are too large/risky to run unsupervised, and
  the arch/image prerequisites need a supervised base decision.
- DGPP w2 full serve — needs model download + `DGPP_ROCE_DEVICES`; the launcher
  side is validated.

## Ling tool-calling + making 3.1-flash smaller (fp4) — research

### Tool-call / reasoning parsers (for a valid Ling bench)
vLLM v0.31.0 ships a dedicated **`ling3`** tool parser
(`vllm/tool_parsers/ling3_tool_parser.py` → `Ling3ToolParser`, registered as
`ling3`). The Ling chat template supports **`enable_thinking`** (on/off, default
on), so the harness `--no-think` (enable_thinking=false) is honored — that is the
fix for the 94/97 timeout failures (the model's default long reasoning blows the
120 s/turn budget). The tool-scenario 400s are fixed by
`--enable-auto-tool-choice --tool-call-parser ling3`. The cookbook also uses
`--reasoning-parser ling3` (present in the newer cu130 nightly wheel; not in
v0.31.0 stable) + `--speculative-config '{"method":"mtp",...}'`. Added
`--enable-auto-tool-choice --tool-call-parser ling3` to the Ling recipe
(recipes `531013f`). Launcher `bench` does not yet pass `--no-think` — either run
tool-eval-bench directly with `--no-think`, or add a passthrough flag (follow-up).

### FP8 is a first-class format — for serving AND as a quant source
Verified by reading safetensors headers (not the config `torch_dtype` field):
- `Qwen/Qwen3.8-27B` base = **all BF16** (392 tensors); `Qwen3.8-27B-FP8` is
  derived from it. So Qwen HAS a full-precision source.
- `inclusionAI/Ling-3.1-flash` = **mixed**: 321 BF16 + 393 F8_E4M3 + 507 F32.
  It is a bf16 model whose heavy MoE linears are stored fp8 (block 128×128,
  DeepSeek-style); the non-quant parts (attention, shared experts, router,
  embeddings, lm_head) are already BF16 in the same checkpoint; the F32 tensors
  are the fp8 block scales (`weight_scale_inv`) + norms. **No separate bf16 repo
  for 3.1-flash** — but bf16 is not required: fp8 is a valid PTQ source
  (inclusionAI's own `Ling-3.0-flash-fp4` records
  `routed_experts_source_quant_method = fp8_block_128x128`).

### The fp4 format (fully specified by `Ling-3.0-flash-fp4` config)
- **routed experts → MXFP4** (`routed_experts_quant_method = mxfp4`,
  `scale_dtype = e8m0`, `group_size = 32`).
- **dense / shared experts → fp8 block 128×128** (fmt e4m3, activation dynamic).
- **17 modules kept BF16** (`modules_to_not_convert`): `q_a_proj`, `q_b_proj`,
  `fused_qkv_a_proj_with_mqa`, `kv_a/kv_b_proj*`, `b_proj`, `mlp.gate` (router),
  `lm_head`, `word_embeddings`, `*.attention.g_proj`.
- Size: 3.0-flash fp4 = **70.4 GB** (vs fp8 128.5 / bf16 255). Note it is
  **MXFP4** (OCP: group 32, e8m0 scale), NOT NVIDIA NVFP4 (group 16, e4m3 scale,
  as in GLM-5.3-Flash-NVFP4 / Step-5). Both 4-bit, both serve on vLLM, both run
  on GB10.

### Serving fp4 on DGX Spark — solved (inclusionAI `ling-cookbook`)
`guide/local-deploy/ling-3.0-flash/dgx-spark-vllm-ling-3-flash-fp4.ipynb`:
**stock vLLM ARM64 cu130 nightly** (`--torch-backend=cu130`,
`--extra-index-url https://wheels.vllm.ai/nightly/cu130`, sm_121) serves the fp4
at **TP=1 on ONE Spark** (62.4 GB). Flags:
`--reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3
--gpu-memory-utilization 0.90 --speculative-config '{"method":"mtp",
"num_speculative_tokens":3}'`. Ling-3.0 arch + `ling3` parser are merged into the
official wheel (no fork needed). Extrapolated: **3.1-flash fp4 ≈ ~110 GB → one
Spark** (vs 534 GB fp8 → 8 Sparks). That is the whole "поменьше" payoff: half the
expert bytes → fits one box + frees KV budget (the GB10 UMA gate).

### The gap — how to CREATE the fp4 (fp8 → mxfp4 experts)
- inclusionAI's **offline quant tool is NOT public** — not in `Ling`,
  `vllm-ling-v3`, or `ling-cookbook` (only CUDA kernels
  `csrc/.../quantization/fp4/mxfp4_experts_quant.cu` + tests). Likely internal
  (Ant/Quark).
- vLLM **online mxfp4** (`tests/quantization/test_online_mxfp4.py`,
  `vllm/model_executor/layers/quantization/online/mxfp4.py`) quantizes
  **bf16/fp16 → mxfp4**, NOT fp8. Our 3.1-flash experts are F8_E4M3, so
  `--quantization mxfp4` does not apply directly (it expects bf16 weights).
- So "just flag our fp8 to mxfp4" does NOT work — the source dtype is wrong.

### Realistic path to a smaller 3.1-flash (fp4)
Write our own **fp8-block-experts → mxfp4-experts** converter (exactly what
inclusionAI did for 3.0). The transform is fully determined:
1. From the NAS checkpoint: expert tensors `F8_E4M3` + their fp8 block scales →
   **dequant to bf16** (using `weight_scale_inv`).
2. **Requant to MXFP4** (group 32, e8m0 scale). Reference impl in vLLM:
   `tests/quantization/reference_mxfp4.py` + kernel `mxfp4_experts_quant.cu`.
3. Keep non-expert tensors as-is (dense fp8, the 17 bf16 modules); write the
   config with the `routed_experts_*` keys (copy 3.0-flash-fp4's, adjust for 72
   layers).
4. Bench the quality delta (MXFP4 experts → small cost).
Well-defined but a real engineering task — the natural **project #2** after the
quantization warm-up.

### Alternatives for "smaller" now
- **GGUF Q4 via llama.cpp** — `AtomicChat/Ling-3.1-flash-GGUF` already on HF +
  cookbook `dgx-spark-llamacpp-ling-3-flash-q4-gguf.ipynb` (for 3.0). Smallest
  cut, but a different stack (llama.cpp, not our vLLM serve line).
- **Wait for an official 3.1-flash fp4** — inclusionAI has fp4 only for 3.0 so
  far; 3.1 may follow.

### Learning plan (operator wants to learn NVFP4/fp4 PTQ — no prior experience)
1. **Warm-up on Qwen4-27B** (when it lands) — clean bf16 source + an official
   ModelOpt recipe exists for the Qwen line → run the whole
   install → calibrate → recipe YAML → export → `vllm serve --quantization
   modelopt_fp4` → bench-delta loop on ONE Spark. Low risk, covers 100% of the
   pipeline. (Deferred until Qwen4-27B is out.)
2. **Ling-3.1-flash fp4 converter** (project #2) — the fp8→mxfp4-experts path
   above; the format + reference impl are the spec.

Key reference files to study: `inclusionAI/Ling-3.0-flash-fp4` config.json (the
format), `ling-cookbook` `dgx-spark-vllm-ling-3-flash-fp4` (serving),
`vllm` `tests/quantization/reference_mxfp4.py` +
`vllm/model_executor/layers/quantization/online/mxfp4.py` +
`csrc/.../fp4/mxfp4_experts_quant.cu` (the mxfp4 math + kernel).