|
|
@@ -204,3 +204,44 @@ gap: `tool_choice: "required"` returns HTTP 500.
|
|
|
Verdict: the EP2/1M SGLang lane stays the interactive default; Beam is the
|
|
|
quality leader for hard batch work; TensorFold remains the 2-node batch shape;
|
|
|
FLARE is not viable for us at this package revision.
|
|
|
+
|
|
|
+## DeepInfer (deepseek-ai/DeepInfer) — assessed 2026-10-21, not runnable on GB10
|
|
|
+
|
|
|
+DeepSeek open-sourced their production inference framework (14.9k stars, Apache
|
|
|
+2.0). It targets exactly our model family — "Supported models:
|
|
|
+DeepSeek-V4.1-Flash", default checkpoint `deepseek-ai/DeepSeek-V4.1-Flash-1017`
|
|
|
+— with optimised kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect,
|
|
|
+TileLang), attention-DP/CP + MoE EP with EPLB, DSpark with confidence-scheduled
|
|
|
+verification, SWA bounded replay, async scheduling, batch-invariant inference,
|
|
|
+deterministic sampling and structured outputs.
|
|
|
+
|
|
|
+**Three blockers, in order of severity:**
|
|
|
+
|
|
|
+1. **Offline inference only.** The README states the release is offline and that
|
|
|
+ a complete service still needs "a KV cache system", "VE/Prefill/Decode
|
|
|
+ disaggregation" and "load balancing across data-parallel replicas" — i.e. no
|
|
|
+ server. There is no HTTP API to point an agent at; it is a Python library
|
|
|
+ (`examples/offline_inference.py`).
|
|
|
+2. **SM100-only.** `deep_cluster/hardware/device.py::is_sm_major_10()` gates on
|
|
|
+ compute capability **major 10** (SM100/101/103) and
|
|
|
+ `get_bf16_tflops_limit()` raises `RuntimeError("unknown GPU type")` otherwise.
|
|
|
+ GB10 is **SM121**. The vendored kernels (FlashMLA/DeepGEMM/DeepEP) are
|
|
|
+ SM90/SM100-targeted too.
|
|
|
+3. **Build cost.** `DEEPINFER_ACCELERATOR=nvidia ./build.sh` needs GCC >= 13,
|
|
|
+ CUDA 13, Python 3.12, four vendored CUDA projects built from submodules, plus
|
|
|
+ an NCCL 2.32.3 force-reinstall; arm64 is not a documented target.
|
|
|
+
|
|
|
+**Verdict:** not usable this shift, and a port is a project (kernels to sm121 +
|
|
|
+write the serving layer), not an evening. What *is* worth stealing:
|
|
|
+
|
|
|
+- **SWA bounded replay** (`DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact`,
|
|
|
+ `..._KEEP_TOKENS=140`): replay only a bounded suffix of the decoder prefill,
|
|
|
+ keeping at least the sliding-window span. This is a prefill-latency idea we
|
|
|
+ could look for in the SGLang/VL stack for our 1017 lane.
|
|
|
+- **Confidence-scheduled verification** (`DEEPINFER_ENABLE_MTP_VARLEN=1` +
|
|
|
+ `DEEPINFER_USE_NEXTN_DECODE`): schedule verification by draft confidence — the
|
|
|
+ same family as the "adaptive verification" knob Beam's DFlash2 wanted.
|
|
|
+- **Batch-invariant inference** — determinism across batch composition; relevant
|
|
|
+ if we ever need reproducible evals.
|
|
|
+
|
|
|
+Re-check when the repo grows a serving path or an sm120/sm121 target.
|
