|
|
@@ -0,0 +1,77 @@
|
|
|
+# Upstream watch — 2026-10-17 (night-shift research pass)
|
|
|
+
|
|
|
+Pass across vLLM releases, the NVIDIA forums, and HF model cards, feeding the
|
|
|
+night-shift agenda (`history/night-shifts/2026-10-17-night-plan.md`).
|
|
|
+
|
|
|
+## vLLM release status — base stays v0.31.0
|
|
|
+- Latest **release** is still **v0.31.0** (2026-10-05). v0.32.0 is only at
|
|
|
+ **rc2** (rc0/rc1/rc2 tags, no stable). No re-pin of the kindling base yet.
|
|
|
+- Keep watching for v0.32.0 stable (would trigger the base re-pin + overlay
|
|
|
+ rebase + flashinfer bump, bench-gated vs the 95/100 kindling 2x anchor).
|
|
|
+
|
|
|
+## New NVIDIA-official FP4 checkpoints (all need the b12x container)
|
|
|
+NVIDIA shipped a family of "…-FP4-Local" QAD checkpoints quantized with
|
|
|
+**nvidia-modelopt v0.47.0** via **quantization-aware distillation (QAD)**. Their
|
|
|
+NVFP4 scale factors use a **new lossless-compression format that upstream vLLM
|
|
|
+does NOT yet support** — you MUST run the `eugr/spark-vllm-b12x` container
|
|
|
+(b12x backend + `--load-format b12x`), not stock vLLM. Digest-pinned
|
|
|
+`eugr/spark-vllm-b12x@sha256:03a95654…bb3595` (arm64, built 2026-10-15).
|
|
|
+
|
|
|
+- **nvidia/GLM-5.3-Flash-FP4-Local** — official NVFP4 GLM-5.3-Flash. **2x DGX
|
|
|
+ Spark**, 500K ctx, MTP n=5 (`moe_backend:humming, attention_backend:B12X`),
|
|
|
+ FP8 KV (~2M tokens), `--attention-backend B12X --block-size 256
|
|
|
+ --moe/--linear/--load-format b12x`, `--reasoning-parser glm45
|
|
|
+ --tool-call-parser glm47`, GMU 0.87. Card quality 91.3% / 92.4%. This is the
|
|
|
+ official NVFP4 counterpart to our kindling 2x lane (our own r35 build) — a
|
|
|
+ direct A/B vs the 95/100 anchor. Recipe: `glm53-flash-fp4-local-2x` (recipes
|
|
|
+ `d5276db`). NOTE: 2x → needs 2 free Sparks; 2nodes-dac is busy with the live
|
|
|
+ kindling serve.
|
|
|
+- **nvidia/Qwen3.8-Flash-Next-FP4-Local** — 1x Spark, b12x, MTP n=4, FP8 KV,
|
|
|
+ 262K, GMU 0.8, `VLLM_ENGINE_READY_TIMEOUT_S=3600` (b12x autotune > 600 s).
|
|
|
+ Forum hardmode 89/100, ~49 t/s decode. Recipe `qwen38-flash-next-fp4-local`.
|
|
|
+- **nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4** (75 GB, 1x Spark, VL,
|
|
|
+ 1M ctx, LiLiCorr dflash-7 drafter, mamba flags, `qwen3_coder`/`nemotron_v3`)
|
|
|
+ and **Nemotron-3.5-Bolt-74B-A9B-24GB** (24 GB, QAD 2-bit experts, 1x Spark,
|
|
|
+ `ultra_v3` parser plugin). Recipes `nemotron-35-super-vl-120b-nvfp4`,
|
|
|
+ `nemotron-35-bolt-74b-24gb`. Forum: Super-VL 90/100, Bolt 88/100.
|
|
|
+
|
|
|
+## GB10 hardware gotchas (both directly bit us this session)
|
|
|
+- **Stuck GPU clock (forum 386060, confirms 376039/376239/366944/…)**: GB10 SM
|
|
|
+ clock can stick at ~660 MHz (max 3003, app clock 2418) at ~16 W with
|
|
|
+ **throttle reasons 0x0**; `nvidia-smi -lgc` is accepted but ignored and a
|
|
|
+ **reboot does not clear it**. Symptom = anomalously low throughput that looks
|
|
|
+ like "hot state" but is a hard clock cap. **Remedy** (community, not NVIDIA-
|
|
|
+ documented): power off, **unplug AC + USB**, hold the power button ~10 s, wait
|
|
|
+ a few minutes, reconnect (one report cleared to 2080 MHz / 77 TFLOPS). → When
|
|
|
+ a lane underperforms, check SM clock under load (expect ~2000–2400, not ~660)
|
|
|
+ before blaming the recipe; a smart-plug cycle may not be enough (needs the
|
|
|
+ hold-power-button drain).
|
|
|
+- **GPU/unified memory is NOT cgroup-contained on GB10 (forum 386057)**: with
|
|
|
+ the 580 driver `nvidia-smi` reports `Memory-Usage: Not Supported` on the iGPU,
|
|
|
+ and `docker --memory` / `MemoryMax=` do **not** cap `cudaMalloc`/unified
|
|
|
+ memory. When the unified pool is exhausted the **whole host stalls** (not just
|
|
|
+ the container). This is exactly why the Ling fp4 converter OOM-killed on dgx1
|
|
|
+ while the GLM-5.3 kindling 2x serve held ~119/121 GB. → **Never run a heavy
|
|
|
+ host job (conversion, big load) on a node that has a live serve.** Use a clean
|
|
|
+ node (we moved to dgx5, 117 GB free). The reporter backported 615's `memacct`
|
|
|
+ to 580 for per-cgroup GPU-memory accounting (not adopted; noted).
|
|
|
+
|
|
|
+## Other forum items (noted, not lanes)
|
|
|
+- **Muse Spark Open 816B on 2 Sparks (386058)**: llama.cpp RPC over 200 GbE
|
|
|
+ (RoCEv2), UD-IQ1_M 194.6 GiB / 6 shards, 32K ctx, q8_0 K/V, **6.8–7.0 tok/s**
|
|
|
+ single request. Two nodes is the floor (one can't hold weights; a third buys
|
|
|
+ only context). Batch-only, llama.cpp stack — not a vLLM lane.
|
|
|
+- DGPP thread 383406 at 108 posts (1 new since the 10-16 pass; no material
|
|
|
+ change). Our pin `3ad63d6b…`, HEAD observed `d9311e72…` — bump pending a
|
|
|
+ supervised re-bench.
|
|
|
+
|
|
|
+## Ling-3.1-flash fp8→fp4 (our own converter — in progress)
|
|
|
+No official 3.1-flash fp4 exists (inclusionAI has fp4 only for 3.0). We wrote a
|
|
|
+converter (fp8-block routed experts → MXFP4, "even" scale mode, byte-exact vs
|
|
|
+vLLM `reference_mxfp4.py` dq). Prototype validated (my dq == reference dq,
|
|
|
+relerr 0.0000; ~10.8% mxfp4 loss vs bf16; experts ~53% of fp8 bytes). Full
|
|
|
+54-shard conversion running on dgx5 → NAS
|
|
|
+`models--inclusionAI--Ling-3.1-flash-fp4`. Target ~110 GB → **one Spark**
|
|
|
+(vs 534 GB fp8 → 8). Serve + bench next. Format + math spec:
|
|
|
+`specs/tech-architecture/upstream-watch-2026-10-16.md` ("making 3.1-flash
|
|
|
+smaller").
|
