upstream-watch 10-16: Ling fp4 research — ling3 parser, fp8-as-source (Ling is mixed bf16+fp8 ckpt), fp4 format spec (mxfp4 experts+fp8 dense), single-Spark serving via cookbook, fp8->mxfp4 converter path + reference files, Qwen4-27B warm-up plan

specs/tech-architecture/upstream-watch-2026-10-16.md

@@ -122,3 +122,103 @@ Both models were already on the NAS HF cache (Ling fp8 534 GB, Step-5 NVFP4
   the arch/image prerequisites need a supervised base decision.
 - DGPP w2 full serve — needs model download + `DGPP_ROCE_DEVICES`; the launcher
   side is validated.
+
+## Ling tool-calling + making 3.1-flash smaller (fp4) — research
+
+### Tool-call / reasoning parsers (for a valid Ling bench)
+vLLM v0.31.0 ships a dedicated **`ling3`** tool parser
+(`vllm/tool_parsers/ling3_tool_parser.py` → `Ling3ToolParser`, registered as
+`ling3`). The Ling chat template supports **`enable_thinking`** (on/off, default
+on), so the harness `--no-think` (enable_thinking=false) is honored — that is the
+fix for the 94/97 timeout failures (the model's default long reasoning blows the
+120 s/turn budget). The tool-scenario 400s are fixed by
+`--enable-auto-tool-choice --tool-call-parser ling3`. The cookbook also uses
+`--reasoning-parser ling3` (present in the newer cu130 nightly wheel; not in
+v0.31.0 stable) + `--speculative-config '{"method":"mtp",...}'`. Added
+`--enable-auto-tool-choice --tool-call-parser ling3` to the Ling recipe
+(recipes `531013f`). Launcher `bench` does not yet pass `--no-think` — either run
+tool-eval-bench directly with `--no-think`, or add a passthrough flag (follow-up).
+
+### FP8 is a first-class format — for serving AND as a quant source
+Verified by reading safetensors headers (not the config `torch_dtype` field):
+- `Qwen/Qwen3.8-27B` base = **all BF16** (392 tensors); `Qwen3.8-27B-FP8` is
+  derived from it. So Qwen HAS a full-precision source.
+- `inclusionAI/Ling-3.1-flash` = **mixed**: 321 BF16 + 393 F8_E4M3 + 507 F32.
+  It is a bf16 model whose heavy MoE linears are stored fp8 (block 128×128,
+  DeepSeek-style); the non-quant parts (attention, shared experts, router,
+  embeddings, lm_head) are already BF16 in the same checkpoint; the F32 tensors
+  are the fp8 block scales (`weight_scale_inv`) + norms. **No separate bf16 repo
+  for 3.1-flash** — but bf16 is not required: fp8 is a valid PTQ source
+  (inclusionAI's own `Ling-3.0-flash-fp4` records
+  `routed_experts_source_quant_method = fp8_block_128x128`).
+
+### The fp4 format (fully specified by `Ling-3.0-flash-fp4` config)
+- **routed experts → MXFP4** (`routed_experts_quant_method = mxfp4`,
+  `scale_dtype = e8m0`, `group_size = 32`).
+- **dense / shared experts → fp8 block 128×128** (fmt e4m3, activation dynamic).
+- **17 modules kept BF16** (`modules_to_not_convert`): `q_a_proj`, `q_b_proj`,
+  `fused_qkv_a_proj_with_mqa`, `kv_a/kv_b_proj*`, `b_proj`, `mlp.gate` (router),
+  `lm_head`, `word_embeddings`, `*.attention.g_proj`.
+- Size: 3.0-flash fp4 = **70.4 GB** (vs fp8 128.5 / bf16 255). Note it is
+  **MXFP4** (OCP: group 32, e8m0 scale), NOT NVIDIA NVFP4 (group 16, e4m3 scale,
+  as in GLM-5.3-Flash-NVFP4 / Step-5). Both 4-bit, both serve on vLLM, both run
+  on GB10.
+
+### Serving fp4 on DGX Spark — solved (inclusionAI `ling-cookbook`)
+`guide/local-deploy/ling-3.0-flash/dgx-spark-vllm-ling-3-flash-fp4.ipynb`:
+**stock vLLM ARM64 cu130 nightly** (`--torch-backend=cu130`,
+`--extra-index-url https://wheels.vllm.ai/nightly/cu130`, sm_121) serves the fp4
+at **TP=1 on ONE Spark** (62.4 GB). Flags:
+`--reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3
+--gpu-memory-utilization 0.90 --speculative-config '{"method":"mtp",
+"num_speculative_tokens":3}'`. Ling-3.0 arch + `ling3` parser are merged into the
+official wheel (no fork needed). Extrapolated: **3.1-flash fp4 ≈ ~110 GB → one
+Spark** (vs 534 GB fp8 → 8 Sparks). That is the whole "поменьше" payoff: half the
+expert bytes → fits one box + frees KV budget (the GB10 UMA gate).
+
+### The gap — how to CREATE the fp4 (fp8 → mxfp4 experts)
+- inclusionAI's **offline quant tool is NOT public** — not in `Ling`,
+  `vllm-ling-v3`, or `ling-cookbook` (only CUDA kernels
+  `csrc/.../quantization/fp4/mxfp4_experts_quant.cu` + tests). Likely internal
+  (Ant/Quark).
+- vLLM **online mxfp4** (`tests/quantization/test_online_mxfp4.py`,
+  `vllm/model_executor/layers/quantization/online/mxfp4.py`) quantizes
+  **bf16/fp16 → mxfp4**, NOT fp8. Our 3.1-flash experts are F8_E4M3, so
+  `--quantization mxfp4` does not apply directly (it expects bf16 weights).
+- So "just flag our fp8 to mxfp4" does NOT work — the source dtype is wrong.
+
+### Realistic path to a smaller 3.1-flash (fp4)
+Write our own **fp8-block-experts → mxfp4-experts** converter (exactly what
+inclusionAI did for 3.0). The transform is fully determined:
+1. From the NAS checkpoint: expert tensors `F8_E4M3` + their fp8 block scales →
+   **dequant to bf16** (using `weight_scale_inv`).
+2. **Requant to MXFP4** (group 32, e8m0 scale). Reference impl in vLLM:
+   `tests/quantization/reference_mxfp4.py` + kernel `mxfp4_experts_quant.cu`.
+3. Keep non-expert tensors as-is (dense fp8, the 17 bf16 modules); write the
+   config with the `routed_experts_*` keys (copy 3.0-flash-fp4's, adjust for 72
+   layers).
+4. Bench the quality delta (MXFP4 experts → small cost).
+Well-defined but a real engineering task — the natural **project #2** after the
+quantization warm-up.
+
+### Alternatives for "smaller" now
+- **GGUF Q4 via llama.cpp** — `AtomicChat/Ling-3.1-flash-GGUF` already on HF +
+  cookbook `dgx-spark-llamacpp-ling-3-flash-q4-gguf.ipynb` (for 3.0). Smallest
+  cut, but a different stack (llama.cpp, not our vLLM serve line).
+- **Wait for an official 3.1-flash fp4** — inclusionAI has fp4 only for 3.0 so
+  far; 3.1 may follow.
+
+### Learning plan (operator wants to learn NVFP4/fp4 PTQ — no prior experience)
+1. **Warm-up on Qwen4-27B** (when it lands) — clean bf16 source + an official
+   ModelOpt recipe exists for the Qwen line → run the whole
+   install → calibrate → recipe YAML → export → `vllm serve --quantization
+   modelopt_fp4` → bench-delta loop on ONE Spark. Low risk, covers 100% of the
+   pipeline. (Deferred until Qwen4-27B is out.)
+2. **Ling-3.1-flash fp4 converter** (project #2) — the fp8→mxfp4-experts path
+   above; the format + reference impl are the spec.
+
+Key reference files to study: `inclusionAI/Ling-3.0-flash-fp4` config.json (the
+format), `ling-cookbook` `dgx-spark-vllm-ling-3-flash-fp4` (serving),
+`vllm` `tests/quantization/reference_mxfp4.py` +
+`vllm/model_executor/layers/quantization/online/mxfp4.py` +
+`csrc/.../fp4/mxfp4_experts_quant.cu` (the mxfp4 math + kernel).