Página Principal Explorar Ajuda
Iniciar Sessão
sm121
/
launcher
2
0
Fork 0
Ficheiros Problemas 0 Pull Requests 0 Wiki
Árvore: 709a8ea749
Ramos Etiquetas
launcher
/
history
/
night-shifts
/
2026-09-08.md

2026-09-08.md 19 KB
Histórico Em bruto

Night shift 2026-09-08

RoCEnante 2x A/B (r27-spark, 4nodes-switch) — DONE

Goal: operator asked to (1) make VLLM_ENABLE_ROCE_ALLREDUCE=1 the default in the r27 recipes and (2) test the 2x variant (the 4x DCP4 A/B from 09-07 showed decode +31…+90%).

Timeline (UTC; node clocks are +2 h ahead — keep that in mind when reading node-side log timestamps):

  • ~09:19 production 4x aiden stopped (operator OK) for the 2x work.
  • Arm A (NCCL control, glm53-flash-r27-spark as-is): 2x on dgx1+dgx2, bench c1,2 seed 42 → 158/176 (90).
  • Recipe rollout: VLLM_ENABLE_ROCE_ALLREDUCE=1 added + --disable-custom-all-reduce removed in both r27 recipes; the interim glm53-flash-r27-4x-dcp4-roce clone deleted, its reports consolidated into glm53-flash-r27-4x-dcp4/reports/ (commit fb3fbf5).
  • Arm B attempt 1 (10:13): picked dgx1+dgx3; the dgx3 worker container died within ~2 min, the rank-0 head hung in NCCL init (orphan). First sighting of the stop-timing race: sm121 stop returned before the previous (production 4x aiden) container had fully released the GPU on dgx3.
  • Operator directive: stop must actually wait for the containers to be gone; --no-wait to opt out. Implemented: stop now runs a per-node verification poll (30 × 2 s) after docker stop and errors if labeled containers are still running (commit a33cfa8). The web dashboard stop job uses the waiting variant.
  • Arm B attempt 2 (10:24, dgx2+dgx3): same dgx3 worker death.
  • Arm B attempt 3 (10:54, dgx1+dgx2): both workers survived; RoCEnante activated (world=2, hcas=rocep1s0f0,roceP2p1s0f0, dispatch ['B12X_ROCENANTE','PYNCCL']). Bench ran to ~43% and the engine died: TileLang trace crash in glm5next/nvidia/ops/glm_kpool.py ('int' object has no attribute 'to' at block.to(tl.int64)) → EngineDeadError, clean shutdown. NOT RoCEnante-specific on its face (the NCCL arm completed the same scenarios fine — but see below), one-off.
  • Arm B re-run (11:23, dgx1+dgx2, RoCEnante): clean full run → 162/176 (92), 100% completion, zero 500s. Crash did not reproduce.

2x A/B result (dev46, seed 42, c1,2):

cell NCCL tg RoCE tg Δ tg
d0 c1 23.4 27.4 +17%
d0 c2 33.5 33.3 −0.6%
d4096 c1 25.3 24.4 −3.6%
d4096 c2 31.8 30.6 −3.8%
d8192 c1 25.2 24.7 −2.0%
d8192 c2 29.1 29.4 +1.0%

Quality: 158 → 162 (+4). Scenario flips: up TC-03/33/49/61/62/68/75 (TC-33 fail→pass); down TC-40/50 (pass→partial), TC-88 (pass→fail). Conclusion: at 2x RoCEnante is a quality win and a small d0-c1 decode win; the 4x DCP4 decode win (+31…+90%) is the headline. RoCEnante is now the default in both r27 recipes.

sm121 bench (operator request)

"Given how we run our benches, should we add a command to the launcher that does all the dirty work?" — yes. Implemented (commit 1ef3eeb): sm121 bench --recipe R --cluster C --label L [--concurrency 1,2] [--seed 42] — waits for API readiness (900 s), runs tool-eval-bench (PATH) in a scratch cwd under ~/.cache/sm121/bench/, lands <UTC-date>_<label>.{md,json,log} into recipes/<lane>/reports/, prints the final score + perf rows. -d dry-run prints the exact command. Served name/port come from the recipe (--served-model-name in args / port). The same commit moves the stop handler from main.rs to docker/stop.rs (line-budget).

r28 full-stack evaluation + version profiles (evening shift)

Operator request: a build_profile entity — a named pin set referenced from recipes instead of copy-pasting the whole [build.versions] table. The half-cut r28 recipe (vLLM advanced, b12x/LMCache left at r27) was exactly the failure mode this kills.

  • Feature (15e9b62): build_profile = "<name>" → docker-images/<dir>/ profiles/<name>.env (same KEY=VAL format); merge base < profile < recipe [build.versions]; merged file written to the node before build.sh and folded into the sm121.checksum label. profiles/glm53-r28.env carries the full r28 lock (vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a — all three forks cut simultaneously 09-06, strictly ahead of r27 pins).
  • TOML context trap (b5f0662): the first build_profile edit landed after the [[extra_mounts]] block → the bare key parsed into the LAST mount table and serde silently ignored it; the build ran with base pins while the recipe "had" the override. Bare keys must stay top-level (tables like [build.versions] reset the context and are safe anywhere). Verified end-to-end with a live load_recipe + build_checksum probe before re-running the build; pitfall documented in both skills.
  • Build: ~6 min (buildx cache), label 14271940…; image verified in-container (vllm r28, lmcache 934c287a, b12x 1.3.0 — the r28 b12x moved to static versioning, so the scm pretend no longer applies; pin still enforced by the HEAD+tree gates).
  • Memory gate flake: first serve died at worker init — Free memory 107.86 < 0.9×121.63 = 109.46 GiB (the gpu_memory_utilization admission gate, vLLM #48140): docker load of the 16 GB image + ~12 GiB of starting vLLM processes pushed the check under the line. The launcher's existing drop_caches (post-sync, pre-container, default on) ran but the margin is still razor-thin; a retry with settled cache booted fine. Follow-up considered: GMU 0.88 in the 4x recipes (gate → 107.0 GiB; KV is explicitly pinned to 16 GiB so capacity is unchanged).
  • Bench (full r28, seed 42, hardmode, c1,2): 155/176 (88/100) vs r27 RoCE 160/176 and mixed-stack r28 159/176; median turn 3.3 s (r27 3.9 s — the r28 scheduler win holds). 8 hard scenarios lost points vs r27 (TC-21/ 49/51/61/67/80/82/88); TC-80+TC-88 failed only on full r28. But: r27 itself spread 155–162 across its A/B runs, and the r28 b12x M8/M1 changes are layout-only ("no model math, scale precision, or output reduction is changed"), so 155 is inside the observed run-to-run band. Re-run queued to pin the variance before judging the stack.
  • Re-run: 159/176 — the 155 was variance, not a regression: 8 scenarios flipped between the two identical-stack runs (incl. TC-80/TC-88, which now PASS). Full r28 (155–159) and r27 (155–162) are statistically tied on quality; r28 wins on speed (median turn 3.0–3.3 s vs 3.9 s; TTFT ~3.6× better per the mixed run). Full r28 stack = production candidate. Also hardened startup: GMU 0.90 → 0.88 in all r27/r28 4x/2x recipes (ff1f727) so the gpu_memory_utilization admission gate (107.0 GiB, not 109.5) survives the ~12 GiB of starting vLLM processes on top of drop_caches (which already ran post-sync, pre-container, default on).

Pending: (1) b12x O_DIRECT loader test — DONE, blocked upstream: --load-format b12x boots the loader (O_DIRECT, 44 shards, 92.6 GB physical, 404.1 s execution, 40.07/44.59 GiB into write-combined shared storage) but dies at b12x/loader/_checkpoint.py:370: ValueError: checkpoint destination was not allocated as a weight: model.language_model.layers.45.eh_proj.weight — the GLM/Step-3.5-MTP eh_proj linear is missing from the pool allocation plan. Report to the fork owner; keep --load-format instanttensor for GLM-5.3 NVFP4 meanwhile. Test recipe kept: glm53-flash-r28-4x-dcp4-b12xload (re-test once the fork fix lands, then delete). (2) Discord env vars — TESTED: --mamba-cache-mode align + VLLM_GLM53_SPLIT_TARGET_BLOCK_SIZE=2048 / _MAMBA_BLOCK_SIZE=256 boots clean (split2048 recipe): KV 8,269,149 tokens (+2.6% vs 8,060,928 standard at the same 16 GiB pin — better page packing), bench 156/176 / 3.1 s — inside the standard-r28 noise band (155/159), so neutral for sequential serving; the win (256-token recurrent checkpoints → finer prefix reuse) only shows under high prefix-cache hit workloads. Keep OFF by default; it is the #646 page geometry the DFlash2 path expects.

GLM-5.3-NVFP4-Spark 386B boot test (operator request) — NOT 4x-capable

local-inference-lab/GLM-5.3-NVFP4-Spark (386B params, GlmMoeDsaForCausalLM — registered in the fork under vllm/models/deepseek_v32, 78 layers, hidden 6144, 8 experts/tok, 1M context): 406 GiB of safetensors on disk (84 shards), ~101.5 GiB per node on 4x. First load pass completed (294 s at 1.5 GB/s), the engine then restarted and began a SECOND full load while the first copy was still resident → 2×101.5 > 121.63 GiB → all four nodes went OOM-thrashing (0 GiB available, sshd unreachable on 3 of 4). Kill issued; the engine kept RESPAWNING OOM-killed workers (the container stayed up), so the nodes never recovered on their own — operator approved a power cycle via HA (switch.dgx1-4 off → on, 2026-09-09 ~05:30 UTC). All four back clean: 116–117 GiB available, 0 sm121 containers.

Answer to the operator: 4 nodes is not enough — a single copy already leaves ~20 GiB/node, which the runtime (vLLM processes ~12 GiB + KV + activations + JIT) consumes to zero. 6+ nodes is the safe shape (67.6 GiB/node weights → 54 GiB headroom); 5 nodes (dgx5, currently off, 88.43 in config) would fit at ~81 GiB/node with ~40 GiB headroom if it joins the RoCE cluster. Keep the glm53-spark-nvfp4-4x recipe as a reference only (6 GiB KV pin, r28 image) — do not run on 4x.

Ops notes / lessons

  • Node clocks on the Sparks run +2 h ahead of UTC; the launcher's own INFO lines are UTC. Compare timestamps across both before drawing conclusions.
  • The 12:0x stop && serve restore chain broke silently: time stop | tail in a backgrounded list swallowed the stop's output, and the serve half never ran — production sat down 07:18→12:21 without a notice. Rule: background chains that chain state-changing commands must log to a file both halves read, or be run foreground; verify the second half actually started. The 12:21 direct serve started fine (4/4, loading at 12:23).
  • Per operator: the 4x node is not needed right now — the 12:21 4x aiden serve was stopped again at 12:24 and left down. Cluster is idle/clean; restore with sm121 serve --recipe glm53-flash-aiden-4x --cluster 4nodes-switch when the operator wants the agent back.

09-09: 2x на 2nodes-dac (DAC A/B, по запросу оператора) + bench pty-фикс

Оператор: 2x-тесты прогнать и на 2nodes-dac — прямое DAC-соединение может давать скорость.

  • glm53-flash-r28-spark (r28 image + RoCE-ON) на 2nodes-dac (100.x fabric, dgx1+dgx2). RoCEnante live и на DAC (b12x_roce_all_reduce.py: "all-reduce is live", "all-gather is live") — CX-7 RoCE не требует коммутатора. Загрузка 304.5 s (91.4 GB/s), KV 2,206,618 tokens (10 GiB pin).
  • Bench 2x DAC: 158/176 (90/100), 1 safety fail (TC-43), median turn 5.5 s. Perf d0: c1 1,475 pp / 27.2 tg / TTFT 1,362 ms, c2 1,396 / 35.2 / 1,949.
  • Вывод: прямое DAC не даёт преимущества — цифры в точности совпадают с r27 2x RoCE A/B от 09-08 на той же DAC-фабрике (armB: 1,500 / 27.4 / 1,398 — ±2%). Для TP2 доля inter-node allreduce мала, коммутатор vs прямая пара — не различимо. RoCE-ON остаётся дефолтом (на NCCL tg ниже: 23.4 vs 27.4).
  • Сервер остановлен, кластер чист (0 sm121-контейнеров).

09-09: sm121 bench вешался после perf-фазы (2x) — pty-фикс

Симптом: 3/3 прогona sm121 bench на 2x-сервере вешались сразу после llama-benchy perf-фазы (child tool-eval-bench: 0% CPU, socket ESTABLISHED, rich-вывод заморожен; сервер при этом здоров — /v1/completions отвечает). 4x-прогоны через тот же launcher работали (4/4) — выглядело как "2x-специфика".

Изоляция (A/B на живой 2x-сервере):

  • A: ручная ручка с redirect в файл → работает (94/100 c1, до TC-88).
  • B: через | tee → "смерть" [Errno 32] Broken pipe — но это артефакт теста (tee писал в closed capture → tee умер → pipe разорвался), не баг.
  • C: script -q /dev/null (pty) → работает (TC-14/15 за 3 мин, прошёл переход perf→scenarios, где A-вариант с pipe вешался).

Диагноз: rich Live-дисплей tool-eval-bench deadlock'ится на non-tty stdout pipe в переходе perf→scenarios (race в рендер-цикле); на tty (pty) — чисто. 4x "работал" по luck (timing-зависимая race).

Фикс (bench.rs): child теперь запускается под /usr/bin/script -q /dev/null (BSD form, macOS; fallback на прямой exec если script absent). Гейты зелёные (271 тестов). Подтверждено живым прогоном: 2x DAC bench дошёл до конца (158/176) — переход прошёл без зависания.

09-09: 386B full model (GLM-5.3-NVFP4-Spark) — не влезает в 4 nodes

Оператор: погонять full model local-inference-lab/GLM-5.3-NVFP4-Spark (406 GiB, 78 layers, hidden 6144, 8 experts/tok, 1M ctx, GlmMoeDsaForCausalLM = fork registry) на 4 nodes.

  • Рецепт glm53-spark-nvfp4-4x (r28 image, KV 6 GiB, GMU 0.88) создан.
  • OOM, доказан эмпирически: weights 406.1 GB / 4 = 101.5 GiB/node + engine overhead (vLLM workers + docker daemon ~12 GiB) ≈ 113.5+ GiB > 121.63 GiB HBM — при этом vLLM делает двойной load (первый pass 294 s @ 1.5 GB/s физ 92.6 GB, engine рестартует и грузит второй раз, пока первая копия ещё в памяти) → 2×101.5 > 121.63 → 4 nodes ушли в OOM-thrash (0 GiB avail, sshd down на 3 из 4).
  • vLLM engine респаунил OOM-killed воркеров бесконечно (контейнер "Up"), ноды сами не восстановились → HA power-cycle (switch.dgx1-4 off→on через HA API, токен из config) — все 4 восстановились чисто (116-117 GiB free, 0 контейнеров).
  • Вывод: 4 nodes не хватает (даже с идеальным распределением weights 101.5
    • overhead не помещаются в 121.63 с безопасным запасом). Нужно 6+ nodes (67.6 GiB/node → 54 GiB headroom) или 5 nodes (81.2 GiB/node → 40 GiB headroom) при подключении dgx5 (192.168.88.43, сейчас off) к RoCE-фабрике. Рецепт оставлен как reference; не запускать на 4x.

09-09: DFlash2 variant для full model — не существует

Оператор: "прогони variant с DFlash2, раз full не влезает". Проверено HF: local-inference-lab имеет только GLM-5.3-NVFP4-Spark (435.6 GB) и GLM-5.3-NVFP4 (464.9 GB) — нет DFlash2-варианта full model (DFlash2-чеки = 1.3 GB draft-модели для Flash-версии: GLM-5.3-Flash-DFlash2*). DFlash2 — speculative drafter, он добавляет памяти (draft model + drafter KV), а не экономит — для full model это усугубило бы OOM. Ответ оператору: variant не существует; для full model нужно расширение кластера (см. выше).

09-09: DFlash2 на r28 — всё ещё blocked (dtype mismatch, уточнение)

Оператор: "проверь, работает ли DFlash сейчас". Пересобрал рецепт glm53-flash-r28-dflash2-2x из git-истории (b3babad), перецелил на r28 image + build_profile. Запуск → BLOCKED, новая точка отказа:

ValueError: No valid attention backend found for kv_cache_dtype=fp8_ds_mla, use_mla=False, block_size=256

Диагностика по исходникам r28 (vLLM 62f6385c):

  • DFlash2 drafter (qwen3_dflash.py) получает KV-dtype от общего engine config (get_kv_quant_mode(self.kv_cache_dtype)), а engine GLM-5.3 работает в fp8_ds_mla.
  • DFlash-аттеншн-бэкенд (vllm/v1/attention/backends/dflash_attn/__init__.py:219,227) требует bf16: TypeError: q and out must be bfloat16 / K and V caches must be bfloat16.
  • r28 commit fa4720ea ("restore GLM boundaries across DCP and DFlash groups") починил prefix-cache/границы, но не dtype-констрейн.
  • В Aiden lane DFlash2 работает в их вилке — у них есть override на bf16 KV для drafter. В нашей r27/r28 вилке per-drafter dtype override нет.

Статус: blocked в форке — нужен либо DFlashAttention с fp8_ds_mla, либо per-drafter bf16-KV override. Отчёт для владельца форка (voipmonitor) в очереди. r27 DFlash2 (прошлый сессии-блокер) — тот же корень, не регресс.

09-09: Прод переключён на full r28 (решение оператора) + upstream-описания

Оператор: (1) поднимать прод на r28; (2) dgx5 — пока только 10G, в кластер не подключаем (386B остаётся ждёт решения об инфраструктуре); (3) подготовить описания проблем, но "странно, они говорят что всё живое" — смотреть в сторону sparkring.

  • Прод = glm53-flash-r28-4x-dcp4 на 4nodes-switch (заменён aiden-4x). Запуск: weights 184G + MTP 12.3G (instanttensor), RoCEnante live, KV 8,060,928 tokens, API подтверждён completion-запросом. ~09:00 UTC.
  • Sparkring-разведка (FujitsuPolycom/sparkring):
    • SparkRing = SIRCL + RoCEnante (наш RoCEnante — из их third_party/b12x_roce)
    • patched NCCL; switchless rings 2x/4x/6x.
    • DFlash2 в sparkring — RETIRED profile (JJ R8/R10: FP8 KV + BF16 DFlash2 depth 7, vLLM dev/jovian-judgement tag e02b1746 + b12x 9ae41c5c) — замещён native MTP3, который и есть наш текущий прод-путь (MTP k=5, head in-checkpoint). Т.е. "всё живое" — потому что их текущий путь не трогает ни b12x-loader, ни DFlash2.
    • 386B full model: DFlash2-варианта не существует (1.3 GB draft-модели только для 184G Flash); drafter добавляет память, не экономит.
  • Upstream-описания (для voipmonitor/local-inference-lab): specs/tech-architecture/upstream-issues-2026-09-09.md — (1) b12x O_DIRECT loader: eh_proj.weight missing from pool allocation plan (b12x/loader/_checkpoint.py:370) — small fix if plan is config-driven; (2) DFlash2: bf16-KV drafter wiring exists in JJ R8 line, absent in r27/r28 base — вопрос владельцу: native MTP — long-term path? (тогда не чиним), либо port из JJ R8; (3) 386B = capacity note, не баг.
  • Cluster: прод 4x r28 serving; 2nodes-dac свободен; dgx5 off (10G only).