repo language: history + spec sections to English (per operator)

Conversation stays Russian, repo content stays English: the 09-09
history sections (2x DAC, bench pty fix, 386B, DFlash2, prod switchover,
sparkring 2x/4x) and the sparkring spec section are rewritten in
English, plus a stray Russian operator quote in the dflash2 recipe
comment. Remaining Cyrillic in the tree is intentional: ru locale,
locale tests, and historical quotes in the July code review / bug
spec.

history/night-shifts/2026-09-08.md

@@ -161,233 +161,245 @@ the RoCE cluster. Keep the `glm53-spark-nvfp4-4x` recipe as a reference only
   restore with `sm121 serve --recipe glm53-flash-aiden-4x --cluster 4nodes-switch`
   when the operator wants the agent back.
 
-## 09-09: 2x на 2nodes-dac (DAC A/B, по запросу оператора) + bench pty-фикс
+## 09-09: 2x on 2nodes-dac (DAC A/B, operator request) + bench pty fix
 
-Оператор: 2x-тесты прогнать и на `2nodes-dac` — прямое DAC-соединение может
-давать скорость.
+Operator: run the 2x tests on `2nodes-dac` too — a direct DAC link may give
+speed.
 
-- `glm53-flash-r28-spark` (r28 image + RoCE-ON) на 2nodes-dac (100.x fabric,
-  dgx1+dgx2). RoCEnante **live и на DAC** (`b12x_roce_all_reduce.py`:
-  "all-reduce is live", "all-gather is live") — CX-7 RoCE не требует коммутатора.
-  Загрузка 304.5 s (91.4 GB/s), KV 2,206,618 tokens (10 GiB pin).
+- `glm53-flash-r28-spark` (r28 image + RoCE-ON) on 2nodes-dac (100.x fabric,
+  dgx1+dgx2). RoCEnante is **live on DAC as well** (`b12x_roce_all_reduce.py`:
+  "all-reduce is live", "all-gather is live") — CX-7 RoCE does not need a
+  switch. Load 304.5 s (91.4 GB/s), KV 2,206,618 tokens (10 GiB pin).
 - **Bench 2x DAC: 158/176 (90/100)**, 1 safety fail (TC-43), median turn 5.5 s.
   Perf d0: c1 **1,475 pp / 27.2 tg / TTFT 1,362 ms**, c2 1,396 / 35.2 / 1,949.
-- **Вывод: прямое DAC не даёт преимущества** — цифры в точности совпадают с
-  r27 2x RoCE A/B от 09-08 на той же DAC-фабрике (armB: 1,500 / 27.4 / 1,398 —
-  ±2%). Для TP2 доля inter-node allreduce мала, коммутатор vs прямая пара —
-  не различимо. RoCE-ON остаётся дефолтом (на NCCL tg ниже: 23.4 vs 27.4).
-- Сервер остановлен, кластер чист (0 sm121-контейнеров).
-
-## 09-09: `sm121 bench` вешался после perf-фазы (2x) — pty-фикс
-
-Симптом: 3/3 прогona `sm121 bench` на 2x-сервере вешались сразу после
-llama-benchy perf-фазы (child tool-eval-bench: 0% CPU, socket ESTABLISHED,
-rich-вывод заморожен; сервер при этом здоров — `/v1/completions` отвечает).
-4x-прогоны через тот же launcher работали (4/4) — выглядело как "2x-специфика".
-
-Изоляция (A/B на живой 2x-сервере):
-- A: ручная ручка с redirect в файл → работает (94/100 c1, до TC-88).
-- B: через `| tee` → "смерть" `[Errno 32] Broken pipe` — но это артефакт
-  теста (tee писал в closed capture → tee умер → pipe разорвался), не баг.
-- **C: `script -q /dev/null` (pty) → работает** (TC-14/15 за 3 мин, прошёл
-  переход perf→scenarios, где A-вариант с pipe вешался).
-
-Диагноз: rich Live-дисплей tool-eval-bench deadlock'ится на non-tty stdout
-pipe в переходе perf→scenarios (race в рендер-цикле); на tty (pty) — чисто.
-4x "работал" по luck (timing-зависимая race).
-
-Фикс (bench.rs): child теперь запускается под `/usr/bin/script -q /dev/null`
-(BSD form, macOS; fallback на прямой exec если script absent). Гейты зелёные
-(271 тестов). Подтверждено живым прогоном: 2x DAC bench дошёл до конца
-(158/176) — переход прошёл без зависания.
-
-## 09-09: 386B full model (GLM-5.3-NVFP4-Spark) — не влезает в 4 nodes
-
-Оператор: погонять full model `local-inference-lab/GLM-5.3-NVFP4-Spark`
+- **Verdict: direct DAC gives no advantage** — numbers match the 09-08 r27 2x
+  RoCE A/B on the same DAC fabric to within ±2% (armB: 1,500 / 27.4 / 1,398).
+  For TP2 the inter-node allreduce share is small; switch vs direct pair is
+  indistinguishable. RoCE-ON stays the default (NCCL tg is lower: 23.4 vs
+  27.4).
+- Server stopped, cluster clean (0 sm121 containers).
+
+## 09-09: `sm121 bench` hung after the perf phase (2x) — pty fix
+
+Symptom: 3/3 `sm121 bench` runs on the 2x server hung right after the
+llama-benchy perf phase (child tool-eval-bench: 0% CPU, socket ESTABLISHED,
+rich output frozen; the server itself healthy — `/v1/completions` answers).
+4x runs through the same launcher worked (4/4) — looked "2x-specific".
+
+Isolation (A/B on the live 2x server):
+- A: manual run with redirect to file → works (94/100 c1, up to TC-88).
+- B: through `| tee` → "dies" with `[Errno 32] Broken pipe — but that is a
+  test artifact (tee wrote to the closed capture → tee died → pipe broke),
+  not a bug.
+- **C: `script -q /dev/null` (pty) → works** (TC-14/15 in 3 min, passed the
+  perf→scenarios transition where the pipe variant hung).
+
+Diagnosis: tool-eval-bench's rich Live display deadlocks on a non-tty stdout
+pipe at the perf→scenarios transition (a race in the render loop); on a tty
+(pty) it is clean. 4x "worked" by luck (timing-dependent race).
+
+Fix (bench.rs): the child now runs under `/usr/bin/script -q /dev/null` (BSD
+form, macOS; fallback to direct exec if script is absent). Gates green (271
+tests). Confirmed by a live run: the 2x DAC bench ran to the end (158/176) —
+the transition passed without hanging.
+
+## 09-09: 386B full model (GLM-5.3-NVFP4-Spark) — does not fit on 4 nodes
+
+Operator: run the full model `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark`
 (406 GiB, 78 layers, hidden 6144, 8 experts/tok, 1M ctx,
-`GlmMoeDsaForCausalLM` = fork registry) на 4 nodes.
+`GlmMoeDsaForCausalLM` = fork registry) on 4 nodes.
 
-- Рецепт `glm53-spark-nvfp4-4x` (r28 image, KV 6 GiB, GMU 0.88) создан.
-- **OOM, доказан эмпирически**: weights 406.1 GB / 4 = 101.5 GiB/node + engine
+- Recipe `glm53-spark-nvfp4-4x` (r28 image, KV 6 GiB, GMU 0.88) created.
+- **OOM, proven empirically**: weights 406.1 GB / 4 = 101.5 GiB/node + engine
   overhead (vLLM workers + docker daemon ~12 GiB) ≈ 113.5+ GiB > 121.63 GiB
-  HBM — при этом vLLM делает **двойной load** (первый pass 294 s @ 1.5 GB/s
-  физ 92.6 GB, engine рестартует и грузит второй раз, пока первая копия ещё в
-  памяти) → 2×101.5 > 121.63 → **4 nodes ушли в OOM-thrash** (0 GiB avail,
-  sshd down на 3 из 4).
-- vLLM engine респаунил OOM-killed воркеров бесконечно (контейнер "Up"),
-  ноды сами не восстановились → **HA power-cycle** (`switch.dgx1-4`
-  off→on через HA API, токен из config) — все 4 восстановились чисто
-  (116-117 GiB free, 0 контейнеров).
-- **Вывод: 4 nodes не хватает** (даже с идеальным распределением weights 101.5
-  + overhead не помещаются в 121.63 с безопасным запасом). Нужно **6+ nodes**
-  (67.6 GiB/node → 54 GiB headroom) или 5 nodes (81.2 GiB/node → 40 GiB
-  headroom) при подключении dgx5 (192.168.88.43, сейчас off) к RoCE-фабрике.
-  Рецепт оставлен как reference; не запускать на 4x.
-
-## 09-09: DFlash2 variant для full model — не существует
-
-Оператор: "прогони variant с DFlash2, раз full не влезает". Проверено HF:
-`local-inference-lab` имеет **только** `GLM-5.3-NVFP4-Spark` (435.6 GB) и
-`GLM-5.3-NVFP4` (464.9 GB) — **нет DFlash2-варианта full model**
-(DFlash2-чеки = 1.3 GB draft-модели для Flash-версии:
-`GLM-5.3-Flash-DFlash2*`). DFlash2 — speculative drafter, он **добавляет**
-памяти (draft model + drafter KV), а не экономит — для full model это
-усугубило бы OOM. Ответ оператору: variant не существует; для full model
-нужно расширение кластера (см. выше).
-
-## 09-09: DFlash2 на r28 — всё ещё blocked (dtype mismatch, уточнение)
-
-Оператор: "проверь, работает ли DFlash сейчас". Пересобрал рецепт
-`glm53-flash-r28-dflash2-2x` из git-истории (b3babad), перецелил на r28
-image + build_profile. Запуск → **BLOCKED, новая точка отказа**:
+  HBM — and vLLM does a **double load** (first pass 294 s @ 1.5 GB/s, physical
+  92.6 GB; the engine restarts and loads a second time while the first copy is
+  still in memory) → 2×101.5 > 121.63 → **all 4 nodes went into OOM-thrash**
+  (0 GiB avail, sshd down on 3 of 4).
+- The vLLM engine respawned OOM-killed workers forever (container "Up"), the
+  nodes did not recover on their own → **HA power-cycle** (`switch.dgx1-4`
+  off→on via the HA API, token from config) — all 4 recovered clean (116-117
+  GiB free, 0 containers).
+- **Verdict: 4 nodes are not enough** (even with a perfect weight
+  distribution, 101.5 + overhead does not fit in 121.63 with a safe margin).
+  Need **6+ nodes** (67.6 GiB/node → 54 GiB headroom) or 5 nodes (81.2
+  GiB/node → 40 GiB headroom) with dgx5 (192.168.88.43, currently off) joined
+  to the RoCE fabric. Recipe kept as reference; do not run on 4x.
+
+## 09-09: DFlash2 variant of the full model — does not exist
+
+Operator: "run the DFlash2 variant since the full model doesn't fit". Checked
+HF: `local-inference-lab` has **only** `GLM-5.3-NVFP4-Spark` (435.6 GB) and
+`GLM-5.3-NVFP4` (464.9 GB) — **no DFlash2 variant of the full model** (the
+DFlash2 checkpoints are 1.3 GB draft models for the Flash version:
+`GLM-5.3-Flash-DFlash2*`). DFlash2 is a speculative drafter — it **adds**
+memory (draft model + drafter KV), it does not save any — for the full model
+it would worsen the OOM. Answer to the operator: the variant does not exist;
+the full model needs cluster expansion (see above).
+
+## 09-09: DFlash2 on r28 — still blocked (dtype mismatch, refinement)
+
+Operator: "check whether DFlash works now". Rebuilt the
+`glm53-flash-r28-dflash2-2x` recipe from git history (b3babad), re-pointed it
+at the r28 image + build_profile. Launch → **BLOCKED, a new failure point**:
 ```
 ValueError: No valid attention backend found for kv_cache_dtype=fp8_ds_mla, use_mla=False, block_size=256
 ```
-Диагностика по исходникам r28 (vLLM 62f6385c):
-- DFlash2 drafter (`qwen3_dflash.py`) получает KV-dtype от общего engine
-  config (`get_kv_quant_mode(self.kv_cache_dtype)`), а engine GLM-5.3 работает
-  в `fp8_ds_mla`.
-- DFlash-аттеншн-бэкенд (`vllm/v1/attention/backends/dflash_attn/__init__.py:219,227`)
-  требует **bf16**: `TypeError: q and out must be bfloat16` /
-  `K and V caches must be bfloat16`.
-- r28 commit `fa4720ea` ("restore GLM boundaries across DCP and DFlash
-  groups") починил prefix-cache/границы, но **не** dtype-констрейн.
-- В Aiden lane DFlash2 работает в их вилке — у них есть override на bf16 KV
-  для drafter. В нашей r27/r28 вилке per-drafter dtype override нет.
-
-**Статус: blocked в форке** — нужен либо DFlashAttention с fp8_ds_mla, либо
-per-drafter bf16-KV override. Отчёт для владельца форка (voipmonitor) в
-очереди. r27 DFlash2 (прошлый сессии-блокер) — тот же корень, не регресс.
-
-## 09-09: Прод переключён на full r28 (решение оператора) + upstream-описания
-
-Оператор: (1) поднимать прод на r28; (2) dgx5 — пока только 10G, в кластер не
-подключаем (386B остаётся ждёт решения об инфраструктуре); (3) подготовить
-описания проблем, но "странно, они говорят что всё живое" — смотреть в сторону
+Diagnosis against the r28 sources (vLLM 62f6385c):
+- The DFlash2 drafter (`qwen3_dflash.py`) gets the KV dtype from the shared
+  engine config (`get_kv_quant_mode(self.kv_cache_dtype)`), and the GLM-5.3
+  engine runs `fp8_ds_mla`.
+- The DFlash attention backend
+  (`vllm/v1/attention/backends/dflash_attn/__init__.py:219,227`) requires
+  **bf16**: `TypeError: q and out must be bfloat16` / `K and V caches must be
+  bfloat16`.
+- The r28 commit `fa4720ea` ("restore GLM boundaries across DCP and DFlash
+  groups") fixed prefix-cache/boundaries, but **not** the dtype constraint.
+- In the Aiden lane DFlash2 works in their fork — they have a bf16-KV override
+  for the drafter. Our r27/r28 fork has no per-drafter dtype override.
+
+**Status: blocked in the fork** — needs either DFlashAttention with
+fp8_ds_mla or a per-drafter bf16-KV override. A report for the fork owner
+(voipmonitor) is queued. The r27 DFlash2 blocker (previous session) has the
+same root cause — not a regression.
+
+## 09-09: Prod switched to full r28 (operator decision) + upstream issue writeups
+
+Operator: (1) switch prod to r28; (2) dgx5 is 10G-only for now, not joining
+the cluster (386B keeps waiting on an infra decision); (3) prepare issue
+descriptions, but "it's strange they say everything is alive" — look at
 sparkring.
 
-- **Прод = `glm53-flash-r28-4x-dcp4` на 4nodes-switch** (заменён aiden-4x).
-  Запуск: weights 184G + MTP 12.3G (instanttensor), RoCEnante live,
-  KV 8,060,928 tokens, API подтверждён completion-запросом. ~09:00 UTC.
-- **Sparkring-разведка** (FujitsuPolycom/sparkring):
-  - SparkRing = SIRCL + RoCEnante (наш RoCEnante — из их `third_party/b12x_roce`)
-    + patched NCCL; switchless rings 2x/4x/6x.
-  - **DFlash2 в sparkring — RETIRED profile** (JJ R8/R10: FP8 KV + BF16
+- **Prod = `glm53-flash-r28-4x-dcp4` on 4nodes-switch** (replaced aiden-4x).
+  Launch: weights 184G + MTP 12.3G (instanttensor), RoCEnante live, KV
+  8,060,928 tokens, API verified with a completion request. ~09:00 UTC.
+- **Sparkring recon** (FujitsuPolycom/sparkring):
+  - SparkRing = SIRCL + RoCEnante (our RoCEnante comes from their
+    `third_party/b12x_roce`) + patched NCCL; switchless rings 2x/4x/6x.
+  - **DFlash2 in sparkring is a RETIRED profile** (JJ R8/R10: FP8 KV + BF16
     DFlash2 depth 7, vLLM `dev/jovian-judgement` tag e02b1746 + b12x 9ae41c5c)
-    — замещён **native MTP3**, который и есть наш текущий прод-путь
-    (MTP k=5, head in-checkpoint). Т.е. "всё живое" — потому что их текущий
-    путь не трогает ни b12x-loader, ни DFlash2.
-  - 386B full model: DFlash2-варианта не существует (1.3 GB draft-модели
-    только для 184G Flash); drafter добавляет память, не экономит.
-- **Upstream-описания** (для voipmonitor/local-inference-lab):
+    — superseded by **native MTP3**, which is exactly our current prod path
+    (MTP k=5, head in-checkpoint). I.e. "everything is alive" because their
+    current path touches neither the b12x loader nor DFlash2.
+- 386B full model: no DFlash2 variant exists (1.3 GB draft models only for
+  the 184G Flash); a drafter adds memory, it does not save any.
+- **Upstream issue writeups** (for voipmonitor/local-inference-lab):
   `specs/tech-architecture/upstream-issues-2026-09-09.md` —
-  (1) b12x O_DIRECT loader: `eh_proj.weight` missing from pool allocation
-  plan (b12x/loader/_checkpoint.py:370) — small fix if plan is config-driven;
-  (2) DFlash2: bf16-KV drafter wiring exists in JJ R8 line, absent in r27/r28
-  base — вопрос владельцу: native MTP — long-term path? (тогда не чиним),
-  либо port из JJ R8; (3) 386B = capacity note, не баг.
-- Cluster: прод 4x r28 serving; 2nodes-dac свободен; dgx5 off (10G only).
-
-## 15:00 UTC+2 — SparkRing shared image: заготовлено, ждём окно
-
-Оператор: «есть имидж — давай попробуем, минимальные усилия». Публичный
-digest-only имидж `ghcr.io/fujitsupolycom/sparkring@f25cdb6...` (22.5 GB),
-pull на dgx1+dgx2 (первый pull по `:latest` — manifest unknown, имидж
-публикуется по digest; второй pull — ок; на dgx2 диск 95% полон, осталось
-49G — впритык, но влезло).
-
-Рецепт `glm53-flash-sparkring-2x`: профиль `glm53-flash-spark-tp2-mtp3`
+  (1) b12x O_DIRECT loader: `eh_proj.weight` missing from the pool allocation
+  plan (b12x/loader/_checkpoint.py:370) — small fix if the plan is
+  config-driven; (2) DFlash2: the bf16-KV drafter wiring exists in the JJ R8
+  line, absent in the r27/r28 base — question to the owner: is native MTP the
+  long-term path? (then nothing to fix) or port from JJ R8; (3) 386B = a
+  capacity note, not a bug.
+- Cluster: prod 4x r28 serving; 2nodes-dac free; dgx5 off (10G only).
+
+## 15:00 UTC+2 — SparkRing shared image: prepared, waiting for a window
+
+Operator: "there is an image — let's try it, minimal effort". Public
+digest-only image `ghcr.io/fujitsupolycom/sparkring@f25cdb6...` (22.5 GB),
+pulled to dgx1+dgx2 (first pull by `:latest` — manifest unknown, the image is
+published by digest only; second pull — ok; dgx2 disk was 95% full, 49G left
+— tight, but it fit).
+
+Recipe `glm53-flash-sparkring-2x`: profile `glm53-flash-spark-tp2-mtp3`
 (TP2/DCP1, RoCEnante 2-path, KV 8.75 GiB fp8, max-len 262K, GMU 0.87, MTP3
-humming, block-size 256, b12x loader, patched NCCL LD_PRELOAD, 43 env —
-всё из profile.json их quickstart). Модель — снапшот df116c4f из NFS через
-extra_mount в /models/target (read_only); entrypoint = default имиджа
-(verify_sources.py --serve wrapper).
-
-Ошибки в ходе подготовки (все поправлено): (1) TOML: bare `args = [...]`
-после `[[extra_mounts]]` молча влетел в mount (тот же класс ловушки) —
-перенёс args/env до table headers; (2) рецепт без `name` — required field;
-(3) dry-run показал `--pp 2` — launcher по дефолту pp=nnodes, надо явное
-`pipeline_parallel_size = 1` в рецепте. Dry-run теперь чистый.
-
-**Почему serve упал с «0 nodes» (4 попытки)**: не баг, не SSH — пара 2x
-(dgx1+dgx2, ssh 88.14/88.10) — те же физические машины, что в 4nodes-switch,
-и на них сейчас стоит **production r28 4x** (sm121_glm53-flash-r28-4x-dcp4_0/_1).
-Probe по sm121-лейблам их видит → occupied → 0 free. Health/web показывают
-per-cluster view (у 2nodes-dac "no containers" — там свои лейблы), что
-маскирует занятость физикой. Вывод: любой sparkring-профиль (2x и 4x)
-требует сначала `sm121 stop --recipe glm53-flash-r28-4x-dcp4` (downtime
-production) — на согласованное окно. Оператору сообщено, ждём решения.
-
-## 16:00–17:00 UTC+2 — SparkRing shared image: 2x прогнан (с деградацией), 4x в работе
-
-Оператор: чистка dgx2 (459.5G docker + 9 осиротевших имиджей + LTX/kk3 93G →
-диск 30%) и «пара часов на тесты на всём кластере». Prod r28 4x остановлен.
-
-**Image-обвязка** (по инициативе оператора): `docker-images/sparkring-shared/`
-— тонкий `FROM <digest>` + обвязка. Проблема digest-only pull решена
-(имидж потеган, prune-safe, стандартный build/sync лаунчера). Попутно два
-реальных бага их имиджа на нашей инфраструктуре:
-1. **b12x loader JIT**: компилит `_storage.c` с `-L/opt/cuda-13.3/lib64 -lcudart`,
-   но в CUDA 13 имидже `/opt/cuda-13.3/lib64` **не существует**, а libcudart лежит
-   только как pip `nvidia/cu13/lib/libcudart.so.13` (без dev-сиmlink) → первый
-   старт умирал на `ld: cannot find -lcudart`. Фикс: symlink + pre-build в
-   нашем Dockerfile (native-кэш испечён в имидж).
-2. **wrapper warmup deadlock**: `serve-with-warmup.py` ждёт API на
-   `127.0.0.1:$PORT` (дефолт 8015) и шлёт warmup-запросы с
-   `model=$SERVED_MODEL_NAME` (дефолт `glm-5.3-flash`). Без `PORT=8000` и
-   `SERVED_MODEL_NAME` в env wrapper бесконечно ретраит мёртвый порт (1800s
-   deadline) — API стоит, health 200, но warmup не стартует. Оба env добавлены
-   в рецепты; таймаут warmup 600→1800 (холодный JIT-компиляб).
-
-**SparkRing 2x (2nodes-dac, профиль tp2-mtp3, RoCEnante)** — bench d0 hardmode:
-- **89/100** (101/114), 12/14 категорий 100%; 2 safety-срыва (TC-43, TC-59)
-- **НО: completion 64.8% — сервер умер на TC-57** (31 сценарий TC-57–88
-  исключён: connection_error «All connection attempts failed»). TC-55/56
-  прошли (21.7s/28.9s), TC-57 завис на 129.7s — и всё. Контейнеры исчезли
-  (--rm), в journal/dmesg ни OOM, ни docker-событий → вероятно их собственный
-  `sparkring-memory-guard` (floor 4GB, из profile lifecycle) убил стек.
-- Perf d0 (до падения): **739 pp / 31.9 tg / TTFT 3,741 ms** (c1) — prefill в 2
-  раза медленнее нашего r28 2x (1,475), TTFT в 2.7 раза хуже (1,362),
-  decode tg +17% (31.9 vs 27.2). c2: 1,195 pp / 50.0 tg / 3,165 TTFT.
-- Вердикт 2x: качество на месте, скорость хуже (особенно prefill/TTFT),
-  **надёжность — провал** (30 мин под нагрузкой не выдержал; наш r28 2x
-  проходит полный hardmode стабильно).
-
-4x switched (профиль tp4-switched-mtp3, plain NCCL, KV 2.28M токенов, 1M ctx):
-запущен, bench идёт.
-
-## 16:50 UTC+2 — SparkRing 4x: почти полный прогон, та же деградация; прод возвращён
-
-**SparkRing 4x switched** (профиль tp4-switched-mtp3, plain NCCL — RoCEnante
-OFF, KV 2.28M токенов, 1M ctx, GMU 0.80, fastsafetensors):
-- **89/100** (152/170), **completion 96.6%** — умер только на TC-85/86/88
-  (последние 3-4 сценария): TC-84 pass 33s → TC-85 timeout 140s → TC-86 timeout
-  → TC-87 degraded → TC-88 500 → контейнеры исчезли со всех 4 нод.
-- Perf d0 (до деградации): **1,230 pp / 59.2 tg / TTFT 2,630 ms** (c1),
-  c2: 1,735 / 62.9 / 1,581. Decode заметно быстрее нашего r28 4x
-  (~46.8 tg) — +26%; prefill 1,230 vs r28 ~1,500-1,600.
-
-**Общий вердикт sparkring shared image (2x + 4x):**
-- Качество на месте: 89/100 на обоих ланах (наш r28 2x = 90, 4x = 90-91).
-- Скорость: 2x — prefill в 2 раза медленнее нашего, TTFT 2.7× хуже, decode +17%;
-  4x — decode +26% (59.2 vs 46.8 tg), prefill/TTFT хуже.
-- **Надёжность — главный провал**: оба профиля умирают под sustained hardmode
-  нагрузкой (2x на ~57/88, 4x на ~85/88). Kernel OOM в dmesg/journal на момент
-  смерти НЕТ → убивает их собственный `sparkring-memory-guard`
-  (memory_guard_minimum_floor_bytes=4GB из profile lifecycle) — штатная
-  self-defense, но наш r28 проходит 88/88 стабильно. Гипотеза оператора
-  («просто более тяжёлый вармап») не подтверждается: смерть — посреди/в конце
-  бенча под нагрузкой, не на warmup-фазе.
-
-**Рецепты по требованию оператора (чистота):**
-- `PORT` убран из [env] обоих рецептов — теперь лаунчер сам инжектит
-  `PORT=<recipe.port>` в docker env (command.rs, generic: нужен wrapper-стилю
-  entrypoint'ам типа sparkring serve-with-warmup, которые поллят
-  127.0.0.1:$PORT; для ванильных vllm/sglang инертен — у них --port CLI-флаг).
-- `model` = стандартный алиас `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark`
-  (репо-ид, как в r28-spark production) — vllm резолвит по HF-кешу
-  refs/main = последний снапшот (df116c4f на сегодня).
-
-**Прод возвращён**: r28 4x поднят на 4nodes-switch (16:53 UTC). Доживём до
-подтверждения API, затем — решение оператора по повторным прогонам
-(«два раза прогнать» — повтор пока не гоняли, ждём окном).
+humming, block-size 256, b12x loader, patched NCCL LD_PRELOAD, 43 env vars —
+all from their quickstart profile.json). Model — the df116c4f snapshot from
+NFS via an extra_mount to /models/target (read_only); entrypoint = the image
+default (verify_sources.py --serve wrapper).
+
+Mistakes during prep (all fixed): (1) TOML: a bare `args = [...]` after
+`[[extra_mounts]]` silently landed inside the mount (same trap class) — moved
+args/env before the table headers; (2) recipe without `name` — required
+field; (3) dry-run showed `--pp 2` — the launcher defaults to pp=nnodes, the
+recipe needs an explicit `pipeline_parallel_size = 1`. Dry-run is clean now.
+
+**Why serve failed with "0 nodes" (4 attempts)**: not a bug, not SSH — the 2x
+pair (dgx1+dgx2, ssh 88.14/88.10) are the **same physical machines** as in
+4nodes-switch, and they were running **production r28 4x**
+(sm121_glm53-flash-r28-4x-dcp4_0/_1). The sm121-label probe sees them →
+occupied → 0 free. Health/web show a per-cluster view (2nodes-dac says "no
+containers" — its own labels), which masks the physical occupancy. Conclusion:
+any sparkring profile (2x and 4x) needs `sm121 stop --recipe
+glm53-flash-r28-4x-dcp4` first (production downtime) — in an agreed window.
+Operator informed, waiting for the decision.
+
+## 16:00–17:00 UTC+2 — SparkRing shared image: 2x run (with degradation), 4x in progress
+
+Operator: clean dgx2 (459.5G docker + 9 orphaned images + 93G LTX/kk3 → disk
+30%) and "we have a couple of hours for tests on the whole cluster". Prod r28
+4x stopped.
+
+**Image wrapper** (operator's idea): `docker-images/sparkring-shared/` — a
+thin `FROM <digest>` + wrapper. Solves the digest-only pull problem (image
+tagged, prune-safe, standard launcher build/sync flow). Along the way, two
+real bugs in their image on our infra:
+1. **b12x loader JIT**: compiles `_storage.c` with `-L/opt/cuda-13.3/lib64
+   -lcudart`, but in the CUDA 13 image `/opt/cuda-13.3/lib64` **does not
+   exist** and libcudart lives only as pip `nvidia/cu13/lib/libcudart.so.13`
+   (no dev symlink) → the first start died with `ld: cannot find -lcudart`.
+   Fix: symlink + pre-build in our Dockerfile (native cache baked into the
+   image).
+2. **wrapper warmup deadlock**: `serve-with-warmup.py` waits for the API on
+   `127.0.0.1:$PORT` (default 8015) and sends warmup requests with
+   `model=$SERVED_MODEL_NAME` (default `glm-5.3-flash`). Without `PORT=8000`
+   and `SERVED_MODEL_NAME` in env, the wrapper retries a dead port forever
+   (1800 s deadline) — the API is up, health 200, but the warmup never starts.
+   Both env vars added to the recipes; warmup timeout 600→1800 (cold JIT
+   compile).
+
+**SparkRing 2x (2nodes-dac, profile tp2-mtp3, RoCEnante)** — bench d0 hardmode:
+- **89/100** (101/114), 12/14 categories at 100%; 2 safety failures (TC-43,
+  TC-59)
+- **BUT: completion 64.8% — the server died at TC-57** (31 scenarios TC-57–88
+  excluded: connection_error "All connection attempts failed"). TC-55/56
+  passed (21.7s/28.9s), TC-57 hung for 129.7s — and that was it. The
+  containers vanished (--rm); no OOM or docker events in journal/dmesg →
+  probably their own `sparkring-memory-guard` (4 GB floor, from the profile
+  lifecycle) killed the stack.
+- Perf d0 (before the death): **739 pp / 31.9 tg / TTFT 3,741 ms** (c1) —
+  prefill 2× slower than our r28 2x (1,475), TTFT 2.7× worse (1,362), decode
+  tg +17% (31.9 vs 27.2). c2: 1,195 pp / 50.0 tg / 3,165 TTFT.
+- 2x verdict: quality intact, speed worse (especially prefill/TTFT),
+  **reliability — failure** (did not survive 30 min under load; our r28 2x
+  passes the full hardmode stably).
+
+4x switched (profile tp4-switched-mtp3, plain NCCL, KV 2.28M tokens, 1M ctx):
+launched, bench running.
+
+## 16:50 UTC+2 — SparkRing 4x: near-complete run, same degradation; prod restored
+
+**SparkRing 4x switched** (profile tp4-switched-mtp3, plain NCCL — RoCEnante
+OFF, KV 2.28M tokens, 1M ctx, GMU 0.80, fastsafetensors):
+- **89/100** (152/170), **completion 96.6%** — died only on TC-85/86/88 (the
+  last 3-4 scenarios): TC-84 pass 33s → TC-85 timeout 140s → TC-86 timeout →
+  TC-87 degraded → TC-88 500 → containers vanished from all 4 nodes.
+- Perf d0 (before degradation): **1,230 pp / 59.2 tg / TTFT 2,630 ms** (c1),
+  c2: 1,735 / 62.9 / 1,581. Decode clearly faster than our r28 4x (~46.8 tg)
+  — +26%; prefill 1,230 vs r28 ~1,500-1,600.
+
+**Overall sparkring shared-image verdict (2x + 4x):**
+- Quality intact: 89/100 on both lanes (our r28 2x = 90, 4x = 90-91).
+- Speed: 2x — prefill 2× slower than ours, TTFT 2.7× worse, decode +17%; 4x —
+  decode +26% (59.2 vs 46.8 tg), prefill/TTFT behind.
+- **Reliability is the main failure**: both profiles die under sustained
+  hardmode load (2x at ~57/88, 4x at ~85/88). No kernel OOM in dmesg/journal
+  at the moment of death → their own `sparkring-memory-guard`
+  (memory_guard_minimum_floor_bytes=4GB from the profile lifecycle) does the
+  killing — a stock self-defense, but our r28 passes 88/88 stably. The
+  operator's hypothesis ("just a heavier warmup") is not confirmed: the death
+  is mid/end of the bench under load, not in the warmup phase.
+
+**Recipe cleanup per operator request:**
+- `PORT` removed from both recipes' [env] — the launcher now injects
+  `PORT=<recipe.port>` into the docker env itself (command.rs, generic:
+  needed by wrapper-style entrypoints like sparkring's serve-with-warmup that
+  poll 127.0.0.1:$PORT; inert for stock vllm/sglang — they take --port as a
+  CLI flag).
+- `model` = the standard alias `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark`
+  (repo id, like the r28-spark production) — vllm resolves it from the HF
+  cache refs/main = latest snapshot (df116c4f as of today).
+
+**Prod restored**: r28 4x brought up on 4nodes-switch (16:53 UTC), API
+verified. Awaiting the operator's decision on repeat runs ("run it twice" —
+the rerun has not been done yet, waiting for a window).

recipes/glm53-flash-r28-dflash2-2x/recipe.toml

@@ -7,8 +7,8 @@
 # fp8_ds_mla, use_mla=False, block_size=256". Fork-side work: teach
 # DFlashAttention the fp8_ds_mla KV, or add a drafter bf16-KV override.
 #
-# DFlash2 re-test on the r28 full stack (2026-09-08, operator: "может он
-# заработал"). The 2026-09-06 blocker (C4 parent-page stride assert + FP8-KV
+# DFlash2 re-test on the r28 full stack (2026-09-08, operator: "maybe it
+# works now"). The 2026-09-06 blocker (C4 parent-page stride assert + FP8-KV
 # drafter backend rejection) targeted the r27 tree; r28's vLLM carries
 # fa4720ea "fix(cache): restore GLM boundaries across DCP and DFlash groups"
 # + the #646 2048/256 page split, so the geometry may now line up.

specs/tech-architecture/glm53-recipe-state.md

@@ -1107,22 +1107,41 @@ the 4x verdict.
 
 ## SparkRing shared image (09-09)
 
-Оператор дал доступ к публичному imaged sparkring-стекa:
+The operator gave access to the public SparkRing stack image:
 `ghcr.io/fujitsupolycom/sparkring@sha256:f25cdb6bf7df85754ea5c62445b139dac3143bc6e910a0442d681b6e63c31c4b`
-(digest-only, тегов нет). Внутри: их vLLM-форк `0.1.dev1+gd377796e8`
+(digest-only, no tags). Inside: their vLLM fork `0.1.dev1+gd377796e8`
 (revision 17bd258075f4), b12x 2883a5df, sparkcache d0cf7296, patched NCCL
-2.30.7, transport `tp2-rocenante-adaptive` (b12x.comm.roce — наш RoCEnante).
-Модель `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` @ df116c4f уже в
-NFS-кеше. Профили: **TP2/DCP1** (RoCEnante 2-path, KV 8.75 GiB, max-len
-262K, GMU 0.87, MTP3, block 256) и **TP4 switched** (plain NCCL, RoCEnante
-OFF, KV 24 GiB, max-len 1M, GMU 0.80, fastsafetensors).
-
-Рецепт `glm53-flash-sparkring-2x` (TP2-профиль, модель через extra_mount
-снапшота df116c4f в /models/target, entrypoint имиджа). Dry-run: чистый
-(ловля лауночера: `pipeline_parallel_size = 1` обязателен в рецепте, иначе
-pp=nnodes по дефолту; bare-ключи TOML — только до table headers).
-
-**Запуск требует окна downtime**: физическая пара 2x (dgx1+dgx2) = те же
-машинки, что держат production r28 4x (primes совпадают: 88.x/100.x/200.x —
-разные fabrics одной коробки). Probe по sm121-лейблам видит prod-контейнеры
-→ 0 free. И 2x, и 4x-профили требуют сначала `stop` prod.
+2.30.7, transport `tp2-rocenante-adaptive` (b12x.comm.roce — the origin of
+our RoCEnante). Model `local-inference-lab/GLM-5.3-Flash-NVFP4-Spark` @
+df116c4f is already in the NFS cache. Profiles: **TP2/DCP1** (RoCEnante
+2-path, KV 8.75 GiB, max-len 262K, GMU 0.87, MTP3, block 256) and **TP4
+switched** (plain NCCL, RoCEnante OFF, KV 24 GiB, max-len 1M, GMU 0.80,
+fastsafetensors).
+
+`docker-images/sparkring-shared/` — a thin `FROM <digest>` wrapper under the
+standard launcher build/sync flow (a digest-only raw pull leaves an untagged
+image that `docker image prune` reaps; the wrapper tags it). The Dockerfile
+also bridges the b12x loader JIT (`-L/opt/cuda-13.3/lib64 -lcudart`, but the
+CUDA 13 image ships libcudart only as pip `nvidia/cu13/lib/libcudart.so.13`
+and no lib64 dir at all) and pre-builds the loader extension into the image.
+
+Recipes `glm53-flash-sparkring-2x` / `-4x` — the profiles verbatim; `model`
+= the standard repo alias (vllm resolves it from the HF cache refs/main =
+latest snapshot); the launcher injects `PORT=<recipe.port>` (the
+serve-with-warmup wrapper polls 127.0.0.1:$PORT); `SERVED_MODEL_NAME` env
+must match `--served-model-name` (the wrapper's warmup model name); cold
+first-run JIT needs `DFLASH_WARMUP_TIMEOUT_SECONDS=1800`.
+
+Measured (09-09, hardmode d0, seed 42): 2x 89/100 (server died at TC-57,
+739 pp / 31.9 tg / TTFT 3,741 ms), 4x 89/100, 96.6% completion (died at
+TC-85/86/88, 1,230 pp / 59.2 tg / TTFT 2,630 ms). Quality on par, decode up
+to +26% (4x), prefill/TTFT behind; **reliability is the failure** — both
+profiles die under sustained load (their sparkring-memory-guard 4GB floor,
+no kernel OOM), while r28 passes 88/88 stably. Details:
+`history/night-shifts/2026-09-08.md`.
+
+**Running either profile requires a prod-downtime window**: the physical 2x
+pair (dgx1+dgx2) is the same hardware as the production r28 4x (88.x/100.x/
+200.x are different fabrics of the same boxes); the sm121-label probe sees
+the prod containers → 0 free. `sm121 stop --recipe glm53-flash-r28-4x-dcp4`
+first, always.