|
|
@@ -161,233 +161,245 @@ the RoCE cluster. Keep the `glm53-spark-nvfp4-4x` recipe as a reference only
|
|
|
restore with `sm121 serve --recipe glm53-flash-aiden-4x --cluster 4nodes-switch`
|
|
|
when the operator wants the agent back.
|
|
|
|
|
|
-## 09-09: 2x на 2nodes-dac (DAC A/B, по запросу оператора) + bench pty-фикс
|
|
|
+## 09-09: 2x on 2nodes-dac (DAC A/B, operator request) + bench pty fix
|
|
|
|
|
|
-Оператор: 2x-тесты прогнать и на `2nodes-dac` — прямое DAC-соединение может
|
|
|
-давать скорость.
|
|
|
+Operator: run the 2x tests on `2nodes-dac` too — a direct DAC link may give
|
|
|
+speed.
|
|
|
|
|
|
-- `glm53-flash-r28-spark` (r28 image + RoCE-ON) на 2nodes-dac (100.x fabric,
|
|
|
- dgx1+dgx2). RoCEnante **live и на DAC** (`b12x_roce_all_reduce.py`:
|
|
|
- "all-reduce is live", "all-gather is live") — CX-7 RoCE не требует коммутатора.
|
|
|
- Загрузка 304.5 s (91.4 GB/s), KV 2,206,618 tokens (10 GiB pin).
|
|
|
+- `glm53-flash-r28-spark` (r28 image + RoCE-ON) on 2nodes-dac (100.x fabric,
|
|
|
+ dgx1+dgx2). RoCEnante is **live on DAC as well** (`b12x_roce_all_reduce.py`:
|
|
|
+ "all-reduce is live", "all-gather is live") — CX-7 RoCE does not need a
|
|
|
+ switch. Load 304.5 s (91.4 GB/s), KV 2,206,618 tokens (10 GiB pin).
|
|
|
- **Bench 2x DAC: 158/176 (90/100)**, 1 safety fail (TC-43), median turn 5.5 s.
|
|
|
Perf d0: c1 **1,475 pp / 27.2 tg / TTFT 1,362 ms**, c2 1,396 / 35.2 / 1,949.
|
|
|
-- **Вывод: прямое DAC не даёт преимущества** — цифры в точности совпадают с
|
|
|
- r27 2x RoCE A/B от 09-08 на той же DAC-фабрике (armB: 1,500 / 27.4 / 1,398 —
|
|
|
- ±2%). Для TP2 доля inter-node allreduce мала, коммутатор vs прямая пара —
|
|
|
- не различимо. RoCE-ON остаётся дефолтом (на NCCL tg ниже: 23.4 vs 27.4).
|
|
|
-- Сервер остановлен, кластер чист (0 sm121-контейнеров).
|
|
|
-
|
|
|
-## 09-09: `sm121 bench` вешался после perf-фазы (2x) — pty-фикс
|
|
|
-
|
|
|
-Симптом: 3/3 прогona `sm121 bench` на 2x-сервере вешались сразу после
|
|
|
-llama-benchy perf-фазы (child tool-eval-bench: 0% CPU, socket ESTABLISHED,
|
|
|
-rich-вывод заморожен; сервер при этом здоров — `/v1/completions` отвечает).
|
|
|
-4x-прогоны через тот же launcher работали (4/4) — выглядело как "2x-специфика".
|
|
|
-
|
|
|
-Изоляция (A/B на живой 2x-сервере):
|
|
|
-- A: ручная ручка с redirect в файл → работает (94/100 c1, до TC-88).
|
|
|
-- B: через `| tee` → "смерть" `[Errno 32] Broken pipe` — но это артефакт
|
|
|
- теста (tee писал в closed capture → tee умер → pipe разорвался), не баг.
|
|
|
-- **C: `script -q /dev/null` (pty) → работает** (TC-14/15 за 3 мин, прошёл
|
|
|
- переход perf→scenarios, где A-вариант с pipe вешался).
|
|
|
-
|
|
|
-Диагноз: rich Live-дисплей tool-eval-bench deadlock'ится на non-tty stdout
|
|
|
-pipe в переходе perf→scenarios (race в рендер-цикле); на tty (pty) — чисто.
|
|
|
-4x "работал" по luck (timing-зависимая race).
|
|
|
-
|
|
|
-Фикс (bench.rs): child теперь запускается под `/usr/bin/script -q /dev/null`
|
|
|
-(BSD form, macOS; fallback на прямой exec если script absent). Гейты зелёные
|
|
|
-(271 тестов). Подтверждено живым прогоном: 2x DAC bench дошёл до конца
|
|
|
-(158/176) — переход прошёл без зависания.
|
|
|
-
|
|
|
-## 09-09: 386B full model (GLM-5.3-NVFP4-Spark) — не влезает в 4 nodes
|
|
|
-
|
|
|
-Оператор: погонять full model `local-inference-lab/GLM-5.3-NVFP4-Spark`
|
|
|
+- **Verdict: direct DAC gives no advantage** — numbers match the 09-08 r27 2x
|
|
|
+ RoCE A/B on the same DAC fabric to within ±2% (armB: 1,500 / 27.4 / 1,398).
|
|
|
+ For TP2 the inter-node allreduce share is small; switch vs direct pair is
|
|
|
+ indistinguishable. RoCE-ON stays the default (NCCL tg is lower: 23.4 vs
|
|
|
+ 27.4).
|
|
|
+- Server stopped, cluster clean (0 sm121 containers).
|
|
|
+
|
|
|
+## 09-09: `sm121 bench` hung after the perf phase (2x) — pty fix
|
|
|
+
|
|
|
+Symptom: 3/3 `sm121 bench` runs on the 2x server hung right after the
|
|
|
+llama-benchy perf phase (child tool-eval-bench: 0% CPU, socket ESTABLISHED,
|
|
|
+rich output frozen; the server itself healthy — `/v1/completions` answers).
|
|
|
+4x runs through the same launcher worked (4/4) — looked "2x-specific".
|
|
|
+
|
|
|
+Isolation (A/B on the live 2x server):
|
|
|
+- A: manual run with redirect to file → works (94/100 c1, up to TC-88).
|
|
|
+- B: through `| tee` → "dies" with `[Errno 32] Broken pipe — but that is a
|
|
|
+ test artifact (tee wrote to the closed capture → tee died → pipe broke),
|
|
|
+ not a bug.
|
|
|
+- **C: `script -q /dev/null` (pty) → works** (TC-14/15 in 3 min, passed the
|
|
|
+ perf→scenarios transition where the pipe variant hung).
|
|
|
+
|
|
|
+Diagnosis: tool-eval-bench's rich Live display deadlocks on a non-tty stdout
|
|
|
+pipe at the perf→scenarios transition (a race in the render loop); on a tty
|
|
|
+(pty) it is clean. 4x "worked" by luck (timing-dependent race).
|
|
|
+
|
|
|
+Fix (bench.rs): the child now runs under `/usr/bin/script -q /dev/null` (BSD
|
|
|
+form, macOS; fallback to direct exec if script is absent). Gates green (271
|
|
|
+tests). Confirmed by a live run: the 2x DAC bench ran to the end (158/176) —
|
|
|
+the transition passed without hanging.
|
|
|
+
|
|
|
+## 09-09: 386B full model (GLM-5.3-NVFP4-Spark) — does not fit on 4 nodes
|
|
|
+
|
|
|
+Operator: run the full model `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark`
|
|
|
(406 GiB, 78 layers, hidden 6144, 8 experts/tok, 1M ctx,
|
|
|
-`GlmMoeDsaForCausalLM` = fork registry) на 4 nodes.
|
|
|
+`GlmMoeDsaForCausalLM` = fork registry) on 4 nodes.
|
|
|
|
|
|
-- Рецепт `glm53-spark-nvfp4-4x` (r28 image, KV 6 GiB, GMU 0.88) создан.
|
|
|
-- **OOM, доказан эмпирически**: weights 406.1 GB / 4 = 101.5 GiB/node + engine
|
|
|
+- Recipe `glm53-spark-nvfp4-4x` (r28 image, KV 6 GiB, GMU 0.88) created.
|
|
|
+- **OOM, proven empirically**: weights 406.1 GB / 4 = 101.5 GiB/node + engine
|
|
|
overhead (vLLM workers + docker daemon ~12 GiB) ≈ 113.5+ GiB > 121.63 GiB
|
|
|
- HBM — при этом vLLM делает **двойной load** (первый pass 294 s @ 1.5 GB/s
|
|
|
- физ 92.6 GB, engine рестартует и грузит второй раз, пока первая копия ещё в
|
|
|
- памяти) → 2×101.5 > 121.63 → **4 nodes ушли в OOM-thrash** (0 GiB avail,
|
|
|
- sshd down на 3 из 4).
|
|
|
-- vLLM engine респаунил OOM-killed воркеров бесконечно (контейнер "Up"),
|
|
|
- ноды сами не восстановились → **HA power-cycle** (`switch.dgx1-4`
|
|
|
- off→on через HA API, токен из config) — все 4 восстановились чисто
|
|
|
- (116-117 GiB free, 0 контейнеров).
|
|
|
-- **Вывод: 4 nodes не хватает** (даже с идеальным распределением weights 101.5
|
|
|
- + overhead не помещаются в 121.63 с безопасным запасом). Нужно **6+ nodes**
|
|
|
- (67.6 GiB/node → 54 GiB headroom) или 5 nodes (81.2 GiB/node → 40 GiB
|
|
|
- headroom) при подключении dgx5 (192.168.88.43, сейчас off) к RoCE-фабрике.
|
|
|
- Рецепт оставлен как reference; не запускать на 4x.
|
|
|
-
|
|
|
-## 09-09: DFlash2 variant для full model — не существует
|
|
|
-
|
|
|
-Оператор: "прогони variant с DFlash2, раз full не влезает". Проверено HF:
|
|
|
-`local-inference-lab` имеет **только** `GLM-5.3-NVFP4-Spark` (435.6 GB) и
|
|
|
-`GLM-5.3-NVFP4` (464.9 GB) — **нет DFlash2-варианта full model**
|
|
|
-(DFlash2-чеки = 1.3 GB draft-модели для Flash-версии:
|
|
|
-`GLM-5.3-Flash-DFlash2*`). DFlash2 — speculative drafter, он **добавляет**
|
|
|
-памяти (draft model + drafter KV), а не экономит — для full model это
|
|
|
-усугубило бы OOM. Ответ оператору: variant не существует; для full model
|
|
|
-нужно расширение кластера (см. выше).
|
|
|
-
|
|
|
-## 09-09: DFlash2 на r28 — всё ещё blocked (dtype mismatch, уточнение)
|
|
|
-
|
|
|
-Оператор: "проверь, работает ли DFlash сейчас". Пересобрал рецепт
|
|
|
-`glm53-flash-r28-dflash2-2x` из git-истории (b3babad), перецелил на r28
|
|
|
-image + build_profile. Запуск → **BLOCKED, новая точка отказа**:
|
|
|
+ HBM — and vLLM does a **double load** (first pass 294 s @ 1.5 GB/s, physical
|
|
|
+ 92.6 GB; the engine restarts and loads a second time while the first copy is
|
|
|
+ still in memory) → 2×101.5 > 121.63 → **all 4 nodes went into OOM-thrash**
|
|
|
+ (0 GiB avail, sshd down on 3 of 4).
|
|
|
+- The vLLM engine respawned OOM-killed workers forever (container "Up"), the
|
|
|
+ nodes did not recover on their own → **HA power-cycle** (`switch.dgx1-4`
|
|
|
+ off→on via the HA API, token from config) — all 4 recovered clean (116-117
|
|
|
+ GiB free, 0 containers).
|
|
|
+- **Verdict: 4 nodes are not enough** (even with a perfect weight
|
|
|
+ distribution, 101.5 + overhead does not fit in 121.63 with a safe margin).
|
|
|
+ Need **6+ nodes** (67.6 GiB/node → 54 GiB headroom) or 5 nodes (81.2
|
|
|
+ GiB/node → 40 GiB headroom) with dgx5 (192.168.88.43, currently off) joined
|
|
|
+ to the RoCE fabric. Recipe kept as reference; do not run on 4x.
|
|
|
+
|
|
|
+## 09-09: DFlash2 variant of the full model — does not exist
|
|
|
+
|
|
|
+Operator: "run the DFlash2 variant since the full model doesn't fit". Checked
|
|
|
+HF: `local-inference-lab` has **only** `GLM-5.3-NVFP4-Spark` (435.6 GB) and
|
|
|
+`GLM-5.3-NVFP4` (464.9 GB) — **no DFlash2 variant of the full model** (the
|
|
|
+DFlash2 checkpoints are 1.3 GB draft models for the Flash version:
|
|
|
+`GLM-5.3-Flash-DFlash2*`). DFlash2 is a speculative drafter — it **adds**
|
|
|
+memory (draft model + drafter KV), it does not save any — for the full model
|
|
|
+it would worsen the OOM. Answer to the operator: the variant does not exist;
|
|
|
+the full model needs cluster expansion (see above).
|
|
|
+
|
|
|
+## 09-09: DFlash2 on r28 — still blocked (dtype mismatch, refinement)
|
|
|
+
|
|
|
+Operator: "check whether DFlash works now". Rebuilt the
|
|
|
+`glm53-flash-r28-dflash2-2x` recipe from git history (b3babad), re-pointed it
|
|
|
+at the r28 image + build_profile. Launch → **BLOCKED, a new failure point**:
|
|
|
```
|
|
|
ValueError: No valid attention backend found for kv_cache_dtype=fp8_ds_mla, use_mla=False, block_size=256
|
|
|
```
|
|
|
-Диагностика по исходникам r28 (vLLM 62f6385c):
|
|
|
-- DFlash2 drafter (`qwen3_dflash.py`) получает KV-dtype от общего engine
|
|
|
- config (`get_kv_quant_mode(self.kv_cache_dtype)`), а engine GLM-5.3 работает
|
|
|
- в `fp8_ds_mla`.
|
|
|
-- DFlash-аттеншн-бэкенд (`vllm/v1/attention/backends/dflash_attn/__init__.py:219,227`)
|
|
|
- требует **bf16**: `TypeError: q and out must be bfloat16` /
|
|
|
- `K and V caches must be bfloat16`.
|
|
|
-- r28 commit `fa4720ea` ("restore GLM boundaries across DCP and DFlash
|
|
|
- groups") починил prefix-cache/границы, но **не** dtype-констрейн.
|
|
|
-- В Aiden lane DFlash2 работает в их вилке — у них есть override на bf16 KV
|
|
|
- для drafter. В нашей r27/r28 вилке per-drafter dtype override нет.
|
|
|
-
|
|
|
-**Статус: blocked в форке** — нужен либо DFlashAttention с fp8_ds_mla, либо
|
|
|
-per-drafter bf16-KV override. Отчёт для владельца форка (voipmonitor) в
|
|
|
-очереди. r27 DFlash2 (прошлый сессии-блокер) — тот же корень, не регресс.
|
|
|
-
|
|
|
-## 09-09: Прод переключён на full r28 (решение оператора) + upstream-описания
|
|
|
-
|
|
|
-Оператор: (1) поднимать прод на r28; (2) dgx5 — пока только 10G, в кластер не
|
|
|
-подключаем (386B остаётся ждёт решения об инфраструктуре); (3) подготовить
|
|
|
-описания проблем, но "странно, они говорят что всё живое" — смотреть в сторону
|
|
|
+Diagnosis against the r28 sources (vLLM 62f6385c):
|
|
|
+- The DFlash2 drafter (`qwen3_dflash.py`) gets the KV dtype from the shared
|
|
|
+ engine config (`get_kv_quant_mode(self.kv_cache_dtype)`), and the GLM-5.3
|
|
|
+ engine runs `fp8_ds_mla`.
|
|
|
+- The DFlash attention backend
|
|
|
+ (`vllm/v1/attention/backends/dflash_attn/__init__.py:219,227`) requires
|
|
|
+ **bf16**: `TypeError: q and out must be bfloat16` / `K and V caches must be
|
|
|
+ bfloat16`.
|
|
|
+- The r28 commit `fa4720ea` ("restore GLM boundaries across DCP and DFlash
|
|
|
+ groups") fixed prefix-cache/boundaries, but **not** the dtype constraint.
|
|
|
+- In the Aiden lane DFlash2 works in their fork — they have a bf16-KV override
|
|
|
+ for the drafter. Our r27/r28 fork has no per-drafter dtype override.
|
|
|
+
|
|
|
+**Status: blocked in the fork** — needs either DFlashAttention with
|
|
|
+fp8_ds_mla or a per-drafter bf16-KV override. A report for the fork owner
|
|
|
+(voipmonitor) is queued. The r27 DFlash2 blocker (previous session) has the
|
|
|
+same root cause — not a regression.
|
|
|
+
|
|
|
+## 09-09: Prod switched to full r28 (operator decision) + upstream issue writeups
|
|
|
+
|
|
|
+Operator: (1) switch prod to r28; (2) dgx5 is 10G-only for now, not joining
|
|
|
+the cluster (386B keeps waiting on an infra decision); (3) prepare issue
|
|
|
+descriptions, but "it's strange they say everything is alive" — look at
|
|
|
sparkring.
|
|
|
|
|
|
-- **Прод = `glm53-flash-r28-4x-dcp4` на 4nodes-switch** (заменён aiden-4x).
|
|
|
- Запуск: weights 184G + MTP 12.3G (instanttensor), RoCEnante live,
|
|
|
- KV 8,060,928 tokens, API подтверждён completion-запросом. ~09:00 UTC.
|
|
|
-- **Sparkring-разведка** (FujitsuPolycom/sparkring):
|
|
|
- - SparkRing = SIRCL + RoCEnante (наш RoCEnante — из их `third_party/b12x_roce`)
|
|
|
- + patched NCCL; switchless rings 2x/4x/6x.
|
|
|
- - **DFlash2 в sparkring — RETIRED profile** (JJ R8/R10: FP8 KV + BF16
|
|
|
+- **Prod = `glm53-flash-r28-4x-dcp4` on 4nodes-switch** (replaced aiden-4x).
|
|
|
+ Launch: weights 184G + MTP 12.3G (instanttensor), RoCEnante live, KV
|
|
|
+ 8,060,928 tokens, API verified with a completion request. ~09:00 UTC.
|
|
|
+- **Sparkring recon** (FujitsuPolycom/sparkring):
|
|
|
+ - SparkRing = SIRCL + RoCEnante (our RoCEnante comes from their
|
|
|
+ `third_party/b12x_roce`) + patched NCCL; switchless rings 2x/4x/6x.
|
|
|
+ - **DFlash2 in sparkring is a RETIRED profile** (JJ R8/R10: FP8 KV + BF16
|
|
|
DFlash2 depth 7, vLLM `dev/jovian-judgement` tag e02b1746 + b12x 9ae41c5c)
|
|
|
- — замещён **native MTP3**, который и есть наш текущий прод-путь
|
|
|
- (MTP k=5, head in-checkpoint). Т.е. "всё живое" — потому что их текущий
|
|
|
- путь не трогает ни b12x-loader, ни DFlash2.
|
|
|
- - 386B full model: DFlash2-варианта не существует (1.3 GB draft-модели
|
|
|
- только для 184G Flash); drafter добавляет память, не экономит.
|
|
|
-- **Upstream-описания** (для voipmonitor/local-inference-lab):
|
|
|
+ — superseded by **native MTP3**, which is exactly our current prod path
|
|
|
+ (MTP k=5, head in-checkpoint). I.e. "everything is alive" because their
|
|
|
+ current path touches neither the b12x loader nor DFlash2.
|
|
|
+- 386B full model: no DFlash2 variant exists (1.3 GB draft models only for
|
|
|
+ the 184G Flash); a drafter adds memory, it does not save any.
|
|
|
+- **Upstream issue writeups** (for voipmonitor/local-inference-lab):
|
|
|
`specs/tech-architecture/upstream-issues-2026-09-09.md` —
|
|
|
- (1) b12x O_DIRECT loader: `eh_proj.weight` missing from pool allocation
|
|
|
- plan (b12x/loader/_checkpoint.py:370) — small fix if plan is config-driven;
|
|
|
- (2) DFlash2: bf16-KV drafter wiring exists in JJ R8 line, absent in r27/r28
|
|
|
- base — вопрос владельцу: native MTP — long-term path? (тогда не чиним),
|
|
|
- либо port из JJ R8; (3) 386B = capacity note, не баг.
|
|
|
-- Cluster: прод 4x r28 serving; 2nodes-dac свободен; dgx5 off (10G only).
|
|
|
-
|
|
|
-## 15:00 UTC+2 — SparkRing shared image: заготовлено, ждём окно
|
|
|
-
|
|
|
-Оператор: «есть имидж — давай попробуем, минимальные усилия». Публичный
|
|
|
-digest-only имидж `ghcr.io/fujitsupolycom/sparkring@f25cdb6...` (22.5 GB),
|
|
|
-pull на dgx1+dgx2 (первый pull по `:latest` — manifest unknown, имидж
|
|
|
-публикуется по digest; второй pull — ок; на dgx2 диск 95% полон, осталось
|
|
|
-49G — впритык, но влезло).
|
|
|
-
|
|
|
-Рецепт `glm53-flash-sparkring-2x`: профиль `glm53-flash-spark-tp2-mtp3`
|
|
|
+ (1) b12x O_DIRECT loader: `eh_proj.weight` missing from the pool allocation
|
|
|
+ plan (b12x/loader/_checkpoint.py:370) — small fix if the plan is
|
|
|
+ config-driven; (2) DFlash2: the bf16-KV drafter wiring exists in the JJ R8
|
|
|
+ line, absent in the r27/r28 base — question to the owner: is native MTP the
|
|
|
+ long-term path? (then nothing to fix) or port from JJ R8; (3) 386B = a
|
|
|
+ capacity note, not a bug.
|
|
|
+- Cluster: prod 4x r28 serving; 2nodes-dac free; dgx5 off (10G only).
|
|
|
+
|
|
|
+## 15:00 UTC+2 — SparkRing shared image: prepared, waiting for a window
|
|
|
+
|
|
|
+Operator: "there is an image — let's try it, minimal effort". Public
|
|
|
+digest-only image `ghcr.io/fujitsupolycom/sparkring@f25cdb6...` (22.5 GB),
|
|
|
+pulled to dgx1+dgx2 (first pull by `:latest` — manifest unknown, the image is
|
|
|
+published by digest only; second pull — ok; dgx2 disk was 95% full, 49G left
|
|
|
+— tight, but it fit).
|
|
|
+
|
|
|
+Recipe `glm53-flash-sparkring-2x`: profile `glm53-flash-spark-tp2-mtp3`
|
|
|
(TP2/DCP1, RoCEnante 2-path, KV 8.75 GiB fp8, max-len 262K, GMU 0.87, MTP3
|
|
|
-humming, block-size 256, b12x loader, patched NCCL LD_PRELOAD, 43 env —
|
|
|
-всё из profile.json их quickstart). Модель — снапшот df116c4f из NFS через
|
|
|
-extra_mount в /models/target (read_only); entrypoint = default имиджа
|
|
|
-(verify_sources.py --serve wrapper).
|
|
|
-
|
|
|
-Ошибки в ходе подготовки (все поправлено): (1) TOML: bare `args = [...]`
|
|
|
-после `[[extra_mounts]]` молча влетел в mount (тот же класс ловушки) —
|
|
|
-перенёс args/env до table headers; (2) рецепт без `name` — required field;
|
|
|
-(3) dry-run показал `--pp 2` — launcher по дефолту pp=nnodes, надо явное
|
|
|
-`pipeline_parallel_size = 1` в рецепте. Dry-run теперь чистый.
|
|
|
-
|
|
|
-**Почему serve упал с «0 nodes» (4 попытки)**: не баг, не SSH — пара 2x
|
|
|
-(dgx1+dgx2, ssh 88.14/88.10) — те же физические машины, что в 4nodes-switch,
|
|
|
-и на них сейчас стоит **production r28 4x** (sm121_glm53-flash-r28-4x-dcp4_0/_1).
|
|
|
-Probe по sm121-лейблам их видит → occupied → 0 free. Health/web показывают
|
|
|
-per-cluster view (у 2nodes-dac "no containers" — там свои лейблы), что
|
|
|
-маскирует занятость физикой. Вывод: любой sparkring-профиль (2x и 4x)
|
|
|
-требует сначала `sm121 stop --recipe glm53-flash-r28-4x-dcp4` (downtime
|
|
|
-production) — на согласованное окно. Оператору сообщено, ждём решения.
|
|
|
-
|
|
|
-## 16:00–17:00 UTC+2 — SparkRing shared image: 2x прогнан (с деградацией), 4x в работе
|
|
|
-
|
|
|
-Оператор: чистка dgx2 (459.5G docker + 9 осиротевших имиджей + LTX/kk3 93G →
|
|
|
-диск 30%) и «пара часов на тесты на всём кластере». Prod r28 4x остановлен.
|
|
|
-
|
|
|
-**Image-обвязка** (по инициативе оператора): `docker-images/sparkring-shared/`
|
|
|
-— тонкий `FROM <digest>` + обвязка. Проблема digest-only pull решена
|
|
|
-(имидж потеган, prune-safe, стандартный build/sync лаунчера). Попутно два
|
|
|
-реальных бага их имиджа на нашей инфраструктуре:
|
|
|
-1. **b12x loader JIT**: компилит `_storage.c` с `-L/opt/cuda-13.3/lib64 -lcudart`,
|
|
|
- но в CUDA 13 имидже `/opt/cuda-13.3/lib64` **не существует**, а libcudart лежит
|
|
|
- только как pip `nvidia/cu13/lib/libcudart.so.13` (без dev-сиmlink) → первый
|
|
|
- старт умирал на `ld: cannot find -lcudart`. Фикс: symlink + pre-build в
|
|
|
- нашем Dockerfile (native-кэш испечён в имидж).
|
|
|
-2. **wrapper warmup deadlock**: `serve-with-warmup.py` ждёт API на
|
|
|
- `127.0.0.1:$PORT` (дефолт 8015) и шлёт warmup-запросы с
|
|
|
- `model=$SERVED_MODEL_NAME` (дефолт `glm-5.3-flash`). Без `PORT=8000` и
|
|
|
- `SERVED_MODEL_NAME` в env wrapper бесконечно ретраит мёртвый порт (1800s
|
|
|
- deadline) — API стоит, health 200, но warmup не стартует. Оба env добавлены
|
|
|
- в рецепты; таймаут warmup 600→1800 (холодный JIT-компиляб).
|
|
|
-
|
|
|
-**SparkRing 2x (2nodes-dac, профиль tp2-mtp3, RoCEnante)** — bench d0 hardmode:
|
|
|
-- **89/100** (101/114), 12/14 категорий 100%; 2 safety-срыва (TC-43, TC-59)
|
|
|
-- **НО: completion 64.8% — сервер умер на TC-57** (31 сценарий TC-57–88
|
|
|
- исключён: connection_error «All connection attempts failed»). TC-55/56
|
|
|
- прошли (21.7s/28.9s), TC-57 завис на 129.7s — и всё. Контейнеры исчезли
|
|
|
- (--rm), в journal/dmesg ни OOM, ни docker-событий → вероятно их собственный
|
|
|
- `sparkring-memory-guard` (floor 4GB, из profile lifecycle) убил стек.
|
|
|
-- Perf d0 (до падения): **739 pp / 31.9 tg / TTFT 3,741 ms** (c1) — prefill в 2
|
|
|
- раза медленнее нашего r28 2x (1,475), TTFT в 2.7 раза хуже (1,362),
|
|
|
- decode tg +17% (31.9 vs 27.2). c2: 1,195 pp / 50.0 tg / 3,165 TTFT.
|
|
|
-- Вердикт 2x: качество на месте, скорость хуже (особенно prefill/TTFT),
|
|
|
- **надёжность — провал** (30 мин под нагрузкой не выдержал; наш r28 2x
|
|
|
- проходит полный hardmode стабильно).
|
|
|
-
|
|
|
-4x switched (профиль tp4-switched-mtp3, plain NCCL, KV 2.28M токенов, 1M ctx):
|
|
|
-запущен, bench идёт.
|
|
|
-
|
|
|
-## 16:50 UTC+2 — SparkRing 4x: почти полный прогон, та же деградация; прод возвращён
|
|
|
-
|
|
|
-**SparkRing 4x switched** (профиль tp4-switched-mtp3, plain NCCL — RoCEnante
|
|
|
-OFF, KV 2.28M токенов, 1M ctx, GMU 0.80, fastsafetensors):
|
|
|
-- **89/100** (152/170), **completion 96.6%** — умер только на TC-85/86/88
|
|
|
- (последние 3-4 сценария): TC-84 pass 33s → TC-85 timeout 140s → TC-86 timeout
|
|
|
- → TC-87 degraded → TC-88 500 → контейнеры исчезли со всех 4 нод.
|
|
|
-- Perf d0 (до деградации): **1,230 pp / 59.2 tg / TTFT 2,630 ms** (c1),
|
|
|
- c2: 1,735 / 62.9 / 1,581. Decode заметно быстрее нашего r28 4x
|
|
|
- (~46.8 tg) — +26%; prefill 1,230 vs r28 ~1,500-1,600.
|
|
|
-
|
|
|
-**Общий вердикт sparkring shared image (2x + 4x):**
|
|
|
-- Качество на месте: 89/100 на обоих ланах (наш r28 2x = 90, 4x = 90-91).
|
|
|
-- Скорость: 2x — prefill в 2 раза медленнее нашего, TTFT 2.7× хуже, decode +17%;
|
|
|
- 4x — decode +26% (59.2 vs 46.8 tg), prefill/TTFT хуже.
|
|
|
-- **Надёжность — главный провал**: оба профиля умирают под sustained hardmode
|
|
|
- нагрузкой (2x на ~57/88, 4x на ~85/88). Kernel OOM в dmesg/journal на момент
|
|
|
- смерти НЕТ → убивает их собственный `sparkring-memory-guard`
|
|
|
- (memory_guard_minimum_floor_bytes=4GB из profile lifecycle) — штатная
|
|
|
- self-defense, но наш r28 проходит 88/88 стабильно. Гипотеза оператора
|
|
|
- («просто более тяжёлый вармап») не подтверждается: смерть — посреди/в конце
|
|
|
- бенча под нагрузкой, не на warmup-фазе.
|
|
|
-
|
|
|
-**Рецепты по требованию оператора (чистота):**
|
|
|
-- `PORT` убран из [env] обоих рецептов — теперь лаунчер сам инжектит
|
|
|
- `PORT=<recipe.port>` в docker env (command.rs, generic: нужен wrapper-стилю
|
|
|
- entrypoint'ам типа sparkring serve-with-warmup, которые поллят
|
|
|
- 127.0.0.1:$PORT; для ванильных vllm/sglang инертен — у них --port CLI-флаг).
|
|
|
-- `model` = стандартный алиас `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark`
|
|
|
- (репо-ид, как в r28-spark production) — vllm резолвит по HF-кешу
|
|
|
- refs/main = последний снапшот (df116c4f на сегодня).
|
|
|
-
|
|
|
-**Прод возвращён**: r28 4x поднят на 4nodes-switch (16:53 UTC). Доживём до
|
|
|
-подтверждения API, затем — решение оператора по повторным прогонам
|
|
|
-(«два раза прогнать» — повтор пока не гоняли, ждём окном).
|
|
|
+humming, block-size 256, b12x loader, patched NCCL LD_PRELOAD, 43 env vars —
|
|
|
+all from their quickstart profile.json). Model — the df116c4f snapshot from
|
|
|
+NFS via an extra_mount to /models/target (read_only); entrypoint = the image
|
|
|
+default (verify_sources.py --serve wrapper).
|
|
|
+
|
|
|
+Mistakes during prep (all fixed): (1) TOML: a bare `args = [...]` after
|
|
|
+`[[extra_mounts]]` silently landed inside the mount (same trap class) — moved
|
|
|
+args/env before the table headers; (2) recipe without `name` — required
|
|
|
+field; (3) dry-run showed `--pp 2` — the launcher defaults to pp=nnodes, the
|
|
|
+recipe needs an explicit `pipeline_parallel_size = 1`. Dry-run is clean now.
|
|
|
+
|
|
|
+**Why serve failed with "0 nodes" (4 attempts)**: not a bug, not SSH — the 2x
|
|
|
+pair (dgx1+dgx2, ssh 88.14/88.10) are the **same physical machines** as in
|
|
|
+4nodes-switch, and they were running **production r28 4x**
|
|
|
+(sm121_glm53-flash-r28-4x-dcp4_0/_1). The sm121-label probe sees them →
|
|
|
+occupied → 0 free. Health/web show a per-cluster view (2nodes-dac says "no
|
|
|
+containers" — its own labels), which masks the physical occupancy. Conclusion:
|
|
|
+any sparkring profile (2x and 4x) needs `sm121 stop --recipe
|
|
|
+glm53-flash-r28-4x-dcp4` first (production downtime) — in an agreed window.
|
|
|
+Operator informed, waiting for the decision.
|
|
|
+
|
|
|
+## 16:00–17:00 UTC+2 — SparkRing shared image: 2x run (with degradation), 4x in progress
|
|
|
+
|
|
|
+Operator: clean dgx2 (459.5G docker + 9 orphaned images + 93G LTX/kk3 → disk
|
|
|
+30%) and "we have a couple of hours for tests on the whole cluster". Prod r28
|
|
|
+4x stopped.
|
|
|
+
|
|
|
+**Image wrapper** (operator's idea): `docker-images/sparkring-shared/` — a
|
|
|
+thin `FROM <digest>` + wrapper. Solves the digest-only pull problem (image
|
|
|
+tagged, prune-safe, standard launcher build/sync flow). Along the way, two
|
|
|
+real bugs in their image on our infra:
|
|
|
+1. **b12x loader JIT**: compiles `_storage.c` with `-L/opt/cuda-13.3/lib64
|
|
|
+ -lcudart`, but in the CUDA 13 image `/opt/cuda-13.3/lib64` **does not
|
|
|
+ exist** and libcudart lives only as pip `nvidia/cu13/lib/libcudart.so.13`
|
|
|
+ (no dev symlink) → the first start died with `ld: cannot find -lcudart`.
|
|
|
+ Fix: symlink + pre-build in our Dockerfile (native cache baked into the
|
|
|
+ image).
|
|
|
+2. **wrapper warmup deadlock**: `serve-with-warmup.py` waits for the API on
|
|
|
+ `127.0.0.1:$PORT` (default 8015) and sends warmup requests with
|
|
|
+ `model=$SERVED_MODEL_NAME` (default `glm-5.3-flash`). Without `PORT=8000`
|
|
|
+ and `SERVED_MODEL_NAME` in env, the wrapper retries a dead port forever
|
|
|
+ (1800 s deadline) — the API is up, health 200, but the warmup never starts.
|
|
|
+ Both env vars added to the recipes; warmup timeout 600→1800 (cold JIT
|
|
|
+ compile).
|
|
|
+
|
|
|
+**SparkRing 2x (2nodes-dac, profile tp2-mtp3, RoCEnante)** — bench d0 hardmode:
|
|
|
+- **89/100** (101/114), 12/14 categories at 100%; 2 safety failures (TC-43,
|
|
|
+ TC-59)
|
|
|
+- **BUT: completion 64.8% — the server died at TC-57** (31 scenarios TC-57–88
|
|
|
+ excluded: connection_error "All connection attempts failed"). TC-55/56
|
|
|
+ passed (21.7s/28.9s), TC-57 hung for 129.7s — and that was it. The
|
|
|
+ containers vanished (--rm); no OOM or docker events in journal/dmesg →
|
|
|
+ probably their own `sparkring-memory-guard` (4 GB floor, from the profile
|
|
|
+ lifecycle) killed the stack.
|
|
|
+- Perf d0 (before the death): **739 pp / 31.9 tg / TTFT 3,741 ms** (c1) —
|
|
|
+ prefill 2× slower than our r28 2x (1,475), TTFT 2.7× worse (1,362), decode
|
|
|
+ tg +17% (31.9 vs 27.2). c2: 1,195 pp / 50.0 tg / 3,165 TTFT.
|
|
|
+- 2x verdict: quality intact, speed worse (especially prefill/TTFT),
|
|
|
+ **reliability — failure** (did not survive 30 min under load; our r28 2x
|
|
|
+ passes the full hardmode stably).
|
|
|
+
|
|
|
+4x switched (profile tp4-switched-mtp3, plain NCCL, KV 2.28M tokens, 1M ctx):
|
|
|
+launched, bench running.
|
|
|
+
|
|
|
+## 16:50 UTC+2 — SparkRing 4x: near-complete run, same degradation; prod restored
|
|
|
+
|
|
|
+**SparkRing 4x switched** (profile tp4-switched-mtp3, plain NCCL — RoCEnante
|
|
|
+OFF, KV 2.28M tokens, 1M ctx, GMU 0.80, fastsafetensors):
|
|
|
+- **89/100** (152/170), **completion 96.6%** — died only on TC-85/86/88 (the
|
|
|
+ last 3-4 scenarios): TC-84 pass 33s → TC-85 timeout 140s → TC-86 timeout →
|
|
|
+ TC-87 degraded → TC-88 500 → containers vanished from all 4 nodes.
|
|
|
+- Perf d0 (before degradation): **1,230 pp / 59.2 tg / TTFT 2,630 ms** (c1),
|
|
|
+ c2: 1,735 / 62.9 / 1,581. Decode clearly faster than our r28 4x (~46.8 tg)
|
|
|
+ — +26%; prefill 1,230 vs r28 ~1,500-1,600.
|
|
|
+
|
|
|
+**Overall sparkring shared-image verdict (2x + 4x):**
|
|
|
+- Quality intact: 89/100 on both lanes (our r28 2x = 90, 4x = 90-91).
|
|
|
+- Speed: 2x — prefill 2× slower than ours, TTFT 2.7× worse, decode +17%; 4x —
|
|
|
+ decode +26% (59.2 vs 46.8 tg), prefill/TTFT behind.
|
|
|
+- **Reliability is the main failure**: both profiles die under sustained
|
|
|
+ hardmode load (2x at ~57/88, 4x at ~85/88). No kernel OOM in dmesg/journal
|
|
|
+ at the moment of death → their own `sparkring-memory-guard`
|
|
|
+ (memory_guard_minimum_floor_bytes=4GB from the profile lifecycle) does the
|
|
|
+ killing — a stock self-defense, but our r28 passes 88/88 stably. The
|
|
|
+ operator's hypothesis ("just a heavier warmup") is not confirmed: the death
|
|
|
+ is mid/end of the bench under load, not in the warmup phase.
|
|
|
+
|
|
|
+**Recipe cleanup per operator request:**
|
|
|
+- `PORT` removed from both recipes' [env] — the launcher now injects
|
|
|
+ `PORT=<recipe.port>` into the docker env itself (command.rs, generic:
|
|
|
+ needed by wrapper-style entrypoints like sparkring's serve-with-warmup that
|
|
|
+ poll 127.0.0.1:$PORT; inert for stock vllm/sglang — they take --port as a
|
|
|
+ CLI flag).
|
|
|
+- `model` = the standard alias `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark`
|
|
|
+ (repo id, like the r28-spark production) — vllm resolves it from the HF
|
|
|
+ cache refs/main = latest snapshot (df116c4f as of today).
|
|
|
+
|
|
|
+**Prod restored**: r28 4x brought up on 4nodes-switch (16:53 UTC), API
|
|
|
+verified. Awaiting the operator's decision on repeat runs ("run it twice" —
|
|
|
+the rerun has not been done yet, waiting for a window).
|
