Главная Обзор Помощь
Вход
sm121
/
launcher
2
0
Ответвить 0
Файлы Задачи 0 Запросы на слияние 0 Вики
Ветка: main
Ветки Метки
launcher
/
history
/
night-shifts
/
2026-10-23.md

2026-10-23.md 13 KB
Постоянная ссылка История Исходник

Night shift 2026-10-22 → 23 — report

Stand dgx5–8 (production dgx1–4 untouched all night). Plan: 2026-10-22-night-plan.md.

1. The headline: the full GLM-5.3 works, and speculation is what makes it usable

glm53-mxfp4-fp8-4x-r39 — RedHatAI/GLM-5.3-MXFP4xFP8 (740B total / ~40B active, 78 layers, DSA, 1M native window) on four Sparks at TP4.

decode c1 decode c2 (agg) score (v2.8.0) median turn
first boot, no speculation 19.0 t/s — 88/100 8.8 s
MTP k=3 33.4–34.9 t/s 47.6 t/s — —
MTP k=5 33.1–37.3 t/s 47.9 t/s — —
MTP k=3 @ 131072 ctx ~31 t/s (benchy) 36.5 t/s 90/100 (175/194) 5.7 s
reference: GLM-5.3-Flash 2x 52 t/s 51 95/100 3.6 s
reference: production 1017 4x (SGLang EP2) 48–51 t/s 64–68 86–87/100 1.4 s

So the biggest model we can run now scores 90/100 with 5.7 s turns — better quality than production, on four nodes nobody else uses, at ~60% of prod's decode speed. k=5 buys nothing over k=3: drafts are accepted 87–93% of the time (Mean acceptance length 4.36–4.66 of 5), so verification, not the ladder, is the limit.

Boot chain (each failure cost a 30-minute reload; all recorded in the recipe header):

  1. kindling spark-glm53:v10 — multi-node worker init dies (is_in_the_same_node: gloo/tcp), the fork is TP2-shaped.
  2. r39 + --attention-backend B12X — TypeError: DeepseekV32B12xAttention got an unexpected keyword 'index_group_builder' (the re-cut indexer API is newer than r39's class).
  3. r39 + --linear-backend marlin — prepare_fp8_layer_for_marlin trips on the ReplicatedLinear modules the re-cut puts fp8_block on.
  4. r39 default dispatch — mm_fp4 does not support backend 'cute-dsl' with capability 121.
  5. VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel — MXFP4 linears fall to Marlin, fp8_block keeps its own backend. Boots.

Karmic A/B (spark-glm53-karmic:local, the local-inference-lab fork): native B12X_MXFP4_MXFP8 MoE experts vs r39's MarlinExperts = 19.2 vs 19.0 t/s. At 99 GiB of weights per rank decode is bandwidth-bound; a faster GEMM is not the lever, speculation is.

2. Two hard limits found by measurement

DCP4 does not exist for this checkpoint. Explicitly selecting the backend answers why:

ValueError: Selected backend AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120
is not valid for this configuration. Reason: ['DCP not supported']

Auto-selection finds no MLA backend at all once DCP is on; same on both forks, block 64 and 256. The DSA/sparse-MLA path has no context-parallel implementation in our images, so the 512K/1M windows of the Flash sibling are out of reach. The window is therefore KV-pin-bound only (52.7 KB/token/rank; 146,944 tokens at a 7.5 GiB pin).

And the KV window is not the usable prompt length. The needle ladder:

prompt result
40,269 tok ✅ correct, TTFT ~85 s
~120K tok ❌ RPC call to sample_tokens timed out (even with VLLM_RPC_TIMEOUT=3600) → EngineDead, and dgx6 + dgx7 wedged (ping alive, sshd starved)
~150K tok (earlier, 196K window) ❌ same, plus the same wedge

Recovery both times: HA plug cycle of the two nodes (off 35 s → on), back in ~1 min, no data loss (weights live on the NAS). The cause is not the KV pool: after 98.86 GiB of weights a Spark has ~22 GiB, and a long prefill's scratch does not fit — smaller chunks and a bigger RPC budget do not help.

Practical envelope for this lane: ≤64K prompts (the 131072 window buys queueing headroom, not a 128K prompt); 8K→64K ladders only. Real 128K+ needs TP8, i.e. halving the weights per rank.

3. FLARE — works, and is honest about where it stands

NVIDIA published NVIDIA/flare-local-inference on 2026-10-20; the full procedure and the six failure modes are in specs/tech-architecture/flare-bringup.md. The production checkpoint (deepseek-ai/DeepSeek-V4.1-Flash-1017, pinned revision) prepared at TP4 across dgx5–8 and served by the Rust runtime: 1M context advertised, 24 GiB KV per rank (28.9M tokens), two RoCE rails per peer. First boot: prefill ~1,430–1,700 t/s, decode 19.2 t/s c1 / 29.6 c2 — ~2.4× slower than our SGLang EP2 production lane on the same model, with nothing tuned. Their upstream moved 12+ commits during the night, including "Preset DeepSeek V4.1 packed GPU speculation" and adaptive verification scoring; that is the obvious next pass.

The agents were stopped and disabled afterwards (unit files kept), and a check confirmed nothing returns at boot.

4. Dashboard / add-on: five releases, one honest post-mortem

1.3.4 → 1.3.9. Features: the recipes panel's Model column now has the same hover affordances as the recipe name — ⧉ copies the model id, HF opens the checkpoint page on the Hub (only for org/name ids) — plus an inline SVG favicon on both pages, a build-version badge under the dashboard title, and a webui entry so the HA add-on page has an "OPEN WEB UI" button.

The release chain itself is worth remembering:

version what went wrong
1.3.4 built from a launcher commit that never reached gitea (a chained git push -q failed silently)
1.3.5 rebuild: buttons + icon land, no badge yet
1.3.6 badge present but shows vdev — a Docker ARG never reaches the compiler
1.3.7 ENV in the builder — still dev: the warm cargo cache (target/ on a cache mount) served the old literal
1.3.8 cargo:rerun-if-env-changed — still dev (the cached fingerprint wins)
1.3.9 the version is generated by build.rs into OUT_DIR, and the CI log prints the value it baked

Verified in the published image, not just in source: docker run … sm121 web …; curl /api/state → {"version":"1.3.9", …}.

5. Research (the night's reading list)

  • RedHatAI/GLM-5.3-speculator.dspark — 2.5 GB, 3-layer Qwen3 draft fed by hidden states from target layers 2/20/39/58/75, up to 8 drafts, 1M context, 41.16% acceptance / 4.29 accepted length at 524K–1M. Downloaded; on r39 it passes config and draft-backend selection, then dies in qwen3_dflash._build_context_kv_buffers — the speculator targets vLLM 0.32 / speculators 0.7.x. Retry belongs on a vLLM-main image (eugr nightly, already staged on two nodes).
  • vLLM 0.32.0 released; RedHatAI/GLM-5.3-MXFP4xFP8 re-cut again on 22.10 14:05 (150 files).
  • tunamitom/b12x@76b76347 — MXFP4-A4 prefill for MoE (CuTe mxfp4_mma_m16n8k64_f32_e2m1_ue8m0, dynamic E8M0 activation scales, decode stays exact W4A16, switch at B12X_W4A16_MXFP4_PREFILL_MIN_TOKENS=512), reported ~+10% prefill and "should work for Flash too". Directly relevant to our prefill pain → todo #23.
  • madeby561/vllm:mimo-v26-flash-b12x-20260923-rc4 (12.9 GB) and its opus55/kk-winner siblings → todo #22. Caveat: the tag is amd64-only, so a Spark needs an arm64 manifest or a source build.
  • FLARE, sparkring (sparkring, sparkcache, net-sircl, glm53-flash-tp4-spark), local-inference-lab blackwell-llm-docker (92★) and the TensorFold family all moved; nothing newer than what we already track.

6. State of the fleet in the morning

dgx5–8: free (four nodes, no containers, all online). dgx1–4: production 1017 untouched. Add-on: 1.3.9 published (buttons, icon, version badge, OPEN WEB UI button). All work committed and pushed: launcher 60ffdda, recipes 2b7b4c7, add-on c6fa22f.

7. What is worth doing next

  1. DSpark on a vLLM-main image (eugr nightly staged): the 2.5 GB speculator is the single biggest remaining lever on the full GLM-5.3 (MTP already took 19 → 35 t/s; a real drafter with 4.3 accepted tokens should go further).
  2. b12x A4 prefill (todo #23): +10% prefill claimed, and prefill is exactly what limits our usable prompt length.
  3. FLARE tuning pass on 1017 (--scheduler-policy throughput, --max-concurrency, adaptive verification) to see whether the Rust runtime can close the 2.4× decode gap on our own production model.

  4. Only then the quality question: is 90/100 on four otherwise-idle nodes worth a production slot? The Flash 2x lane still leads at 95/100 on two nodes.

    Engram локализация — доведено (вечер 2026-10-23)

Раздача шардов. rsync по фабрике (192.168.200.x, ~470–490 MB/s) с dgx5 на dgx6/7/8: 2×101.5 GB на ноду, 16:48 / 16:58 / 17:07 UTC. Все четыре ноды держат 189 GiB локальных копий (~/models/dsv41-1017-local), других остатков тех же весов нет — ~/models содержит только этот каталог, ~/hf-local отсутствует. Свободно: dgx5 239G, dgx6 66G, dgx7 49G, dgx8 76G (dgx7 — самое узкое место).

Сделано в лаунчере (новый staging-kind). [[staging]] kind = "local-view": файлы по local-глобам становятся настоящими локальными копиями, всё остальное — симлинки на контейнерный путь общего кэша; serve монтирует вид в /root/.cache/hf-view и переписывает model. Плюс staging::preflight — до любой копии и до первого контейнера опрашивает каждую ноду (measure: сколько надо / сколько есть, с учётом уже разложенного и min_free-запаса) и отказывает с именем ноды, репозиторием, размером и каталогом. Три бага, найденных на живом прогоне: симлинки указывали на сам вид (нужен общий кэш /root/.cache/huggingface), вид раскладывается в dst/<repo>/, и при отсутствии леджера проход перекопировал 190 GB вместо того, чтобы принять уже разложенное дерево по хешам.

Результат (1017 TP4, DSpark k5, сток vLLM 0.32):

engram на NAS engram локально прод SGLang
prefill pp2048 c1 750 t/s 2,927 t/s 2,800–3,100
prefill (12.6K уникальный промпт) — 3,957 t/s —
decode c1 / c2 30.3 / — 37.0 / 46.7 47.8–50.7 / 64.2–68.3
чтения с /mnt/nas за прогон +258 MB на запрос 0 MB —

Бенч v2.8.0 (2026-10-23_175600_engram-local): 91/100 (177/194), Hard Mode 48/56 (86%), Safety 85%, Responsiveness 50 (медианный ход 3.0 s), 1 safety-critical — TC-92 (tenant isolation). Прод на том же моделе — 86–87/100.

Вывод: разрыв в префилле был I/O, а не стек движка; остаётся декод (37.0 c1 против прод 47.8–50.7) при уже равном префилле.

Диск: аудит и чистка (вечер 2026-10-23)

Вопрос оператора «нода 916 G, ОС ~100 G, куда остальное» разобрался вскрытием du -x -d1 / по нодам. Настоящие пожиратели:

  1. docker: остановленные контейнеры держали образы. docker image prune -a не трогает образ, на который ссылается любой контейнер, включая Exited (у нас их висело 12–15 на ноду). Поэтому docker system df показывал «211–245 G reclaimable», а место не возвращалось. Лечение: docker container prune -f && docker image prune -af → +276…+287 G/ноду.
  2. ~/sm121-cache 91 G × 4 (90 G — тарболл закрытого GLM-5.3 kindling) и ~/ns/1017-tf 95 G × 2 (TensorFold) — первый rm -rf без sudo молча не сработал (root-owned), поэтому отчёт о чистке был неверным. Только с sudo освободилось.
  3. /srv/gr8 86 G × 2 — остатки движка gr8.
  4. ~/.cache/uv 24–30 G на ноде (кэш uv-тулчейна) — снесён.
  5. flare-src нужен только контроллеру (dgx5) — с остальных удалён, бинарь копируется; vllm-judge удалён (переделывать заново).

Итог: свободно dgx5 557 G, dgx6 605 G, dgx7 607 G, dgx8 597 G (осталось занято: engram 190 G/ноду, docker ~30–50 G, ОС ~65–95 G). Места хватает на gb10-пакет (511 G контроллеру + срезы пирам).