Acasă Explorează Ajutor
Autentificare
sm121
/
launcher
2
0
Bifurcare 0
Fisiere Probleme 0 Trageți solicitările 0 Wiki
Arbore: 06bf0e84da
Ramuri Etichete
launcher
/
history
/
night-shifts
/
2026-09-08.md

2026-09-08.md 9.8 KB
Istoric Crud

Night shift 2026-09-08

RoCEnante 2x A/B (r27-spark, 4nodes-switch) — DONE

Goal: operator asked to (1) make VLLM_ENABLE_ROCE_ALLREDUCE=1 the default in the r27 recipes and (2) test the 2x variant (the 4x DCP4 A/B from 09-07 showed decode +31…+90%).

Timeline (UTC; node clocks are +2 h ahead — keep that in mind when reading node-side log timestamps):

  • ~09:19 production 4x aiden stopped (operator OK) for the 2x work.
  • Arm A (NCCL control, glm53-flash-r27-spark as-is): 2x on dgx1+dgx2, bench c1,2 seed 42 → 158/176 (90).
  • Recipe rollout: VLLM_ENABLE_ROCE_ALLREDUCE=1 added + --disable-custom-all-reduce removed in both r27 recipes; the interim glm53-flash-r27-4x-dcp4-roce clone deleted, its reports consolidated into glm53-flash-r27-4x-dcp4/reports/ (commit fb3fbf5).
  • Arm B attempt 1 (10:13): picked dgx1+dgx3; the dgx3 worker container died within ~2 min, the rank-0 head hung in NCCL init (orphan). First sighting of the stop-timing race: sm121 stop returned before the previous (production 4x aiden) container had fully released the GPU on dgx3.
  • Operator directive: stop must actually wait for the containers to be gone; --no-wait to opt out. Implemented: stop now runs a per-node verification poll (30 × 2 s) after docker stop and errors if labeled containers are still running (commit a33cfa8). The web dashboard stop job uses the waiting variant.
  • Arm B attempt 2 (10:24, dgx2+dgx3): same dgx3 worker death.
  • Arm B attempt 3 (10:54, dgx1+dgx2): both workers survived; RoCEnante activated (world=2, hcas=rocep1s0f0,roceP2p1s0f0, dispatch ['B12X_ROCENANTE','PYNCCL']). Bench ran to ~43% and the engine died: TileLang trace crash in glm5next/nvidia/ops/glm_kpool.py ('int' object has no attribute 'to' at block.to(tl.int64)) → EngineDeadError, clean shutdown. NOT RoCEnante-specific on its face (the NCCL arm completed the same scenarios fine — but see below), one-off.
  • Arm B re-run (11:23, dgx1+dgx2, RoCEnante): clean full run → 162/176 (92), 100% completion, zero 500s. Crash did not reproduce.

2x A/B result (dev46, seed 42, c1,2):

cell NCCL tg RoCE tg Δ tg
d0 c1 23.4 27.4 +17%
d0 c2 33.5 33.3 −0.6%
d4096 c1 25.3 24.4 −3.6%
d4096 c2 31.8 30.6 −3.8%
d8192 c1 25.2 24.7 −2.0%
d8192 c2 29.1 29.4 +1.0%

Quality: 158 → 162 (+4). Scenario flips: up TC-03/33/49/61/62/68/75 (TC-33 fail→pass); down TC-40/50 (pass→partial), TC-88 (pass→fail). Conclusion: at 2x RoCEnante is a quality win and a small d0-c1 decode win; the 4x DCP4 decode win (+31…+90%) is the headline. RoCEnante is now the default in both r27 recipes.

sm121 bench (operator request)

"Given how we run our benches, should we add a command to the launcher that does all the dirty work?" — yes. Implemented (commit 1ef3eeb): sm121 bench --recipe R --cluster C --label L [--concurrency 1,2] [--seed 42] — waits for API readiness (900 s), runs tool-eval-bench (PATH) in a scratch cwd under ~/.cache/sm121/bench/, lands <UTC-date>_<label>.{md,json,log} into recipes/<lane>/reports/, prints the final score + perf rows. -d dry-run prints the exact command. Served name/port come from the recipe (--served-model-name in args / port). The same commit moves the stop handler from main.rs to docker/stop.rs (line-budget).

r28 full-stack evaluation + version profiles (evening shift)

Operator request: a build_profile entity — a named pin set referenced from recipes instead of copy-pasting the whole [build.versions] table. The half-cut r28 recipe (vLLM advanced, b12x/LMCache left at r27) was exactly the failure mode this kills.

  • Feature (15e9b62): build_profile = "<name>" → docker-images/<dir>/ profiles/<name>.env (same KEY=VAL format); merge base < profile < recipe [build.versions]; merged file written to the node before build.sh and folded into the sm121.checksum label. profiles/glm53-r28.env carries the full r28 lock (vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a — all three forks cut simultaneously 09-06, strictly ahead of r27 pins).
  • TOML context trap (b5f0662): the first build_profile edit landed after the [[extra_mounts]] block → the bare key parsed into the LAST mount table and serde silently ignored it; the build ran with base pins while the recipe "had" the override. Bare keys must stay top-level (tables like [build.versions] reset the context and are safe anywhere). Verified end-to-end with a live load_recipe + build_checksum probe before re-running the build; pitfall documented in both skills.
  • Build: ~6 min (buildx cache), label 14271940…; image verified in-container (vllm r28, lmcache 934c287a, b12x 1.3.0 — the r28 b12x moved to static versioning, so the scm pretend no longer applies; pin still enforced by the HEAD+tree gates).
  • Memory gate flake: first serve died at worker init — Free memory 107.86 < 0.9×121.63 = 109.46 GiB (the gpu_memory_utilization admission gate, vLLM #48140): docker load of the 16 GB image + ~12 GiB of starting vLLM processes pushed the check under the line. The launcher's existing drop_caches (post-sync, pre-container, default on) ran but the margin is still razor-thin; a retry with settled cache booted fine. Follow-up considered: GMU 0.88 in the 4x recipes (gate → 107.0 GiB; KV is explicitly pinned to 16 GiB so capacity is unchanged).
  • Bench (full r28, seed 42, hardmode, c1,2): 155/176 (88/100) vs r27 RoCE 160/176 and mixed-stack r28 159/176; median turn 3.3 s (r27 3.9 s — the r28 scheduler win holds). 8 hard scenarios lost points vs r27 (TC-21/ 49/51/61/67/80/82/88); TC-80+TC-88 failed only on full r28. But: r27 itself spread 155–162 across its A/B runs, and the r28 b12x M8/M1 changes are layout-only ("no model math, scale precision, or output reduction is changed"), so 155 is inside the observed run-to-run band. Re-run queued to pin the variance before judging the stack.
  • Re-run: 159/176 — the 155 was variance, not a regression: 8 scenarios flipped between the two identical-stack runs (incl. TC-80/TC-88, which now PASS). Full r28 (155–159) and r27 (155–162) are statistically tied on quality; r28 wins on speed (median turn 3.0–3.3 s vs 3.9 s; TTFT ~3.6× better per the mixed run). Full r28 stack = production candidate. Also hardened startup: GMU 0.90 → 0.88 in all r27/r28 4x/2x recipes (ff1f727) so the gpu_memory_utilization admission gate (107.0 GiB, not 109.5) survives the ~12 GiB of starting vLLM processes on top of drop_caches (which already ran post-sync, pre-container, default on).

Pending: (1) b12x O_DIRECT loader test — DONE, blocked upstream: --load-format b12x boots the loader (O_DIRECT, 44 shards, 92.6 GB physical, 404.1 s execution, 40.07/44.59 GiB into write-combined shared storage) but dies at b12x/loader/_checkpoint.py:370: ValueError: checkpoint destination was not allocated as a weight: model.language_model.layers.45.eh_proj.weight — the GLM/Step-3.5-MTP eh_proj linear is missing from the pool allocation plan. Report to the fork owner; keep --load-format instanttensor for GLM-5.3 NVFP4 meanwhile. Test recipe kept: glm53-flash-r28-4x-dcp4-b12xload (re-test once the fork fix lands, then delete). (2) Discord env vars — TESTED: --mamba-cache-mode align + VLLM_GLM53_SPLIT_TARGET_BLOCK_SIZE=2048 / _MAMBA_BLOCK_SIZE=256 boots clean (split2048 recipe): KV 8,269,149 tokens (+2.6% vs 8,060,928 standard at the same 16 GiB pin — better page packing), bench 156/176 / 3.1 s — inside the standard-r28 noise band (155/159), so neutral for sequential serving; the win (256-token recurrent checkpoints → finer prefix reuse) only shows under high prefix-cache hit workloads. Keep OFF by default; it is the #646 page geometry the DFlash2 path expects.

GLM-5.3-NVFP4-Spark 386B boot test (operator request) — NOT 4x-capable

local-inference-lab/GLM-5.3-NVFP4-Spark (386B params, GlmMoeDsaForCausalLM — registered in the fork under vllm/models/deepseek_v32, 78 layers, hidden 6144, 8 experts/tok, 1M context): 406 GiB of safetensors on disk (84 shards), ~101.5 GiB per node on 4x. First load pass completed (294 s at 1.5 GB/s), the engine then restarted and began a SECOND full load while the first copy was still resident → 2×101.5 > 121.63 GiB → all four nodes went OOM-thrashing (0 GiB available, sshd unreachable on 3 of 4). Kill + wait for the OOM killer; nodes recovering slowly (swap thrash).

Answer to the operator: 4 nodes is not enough — a single copy already leaves ~20 GiB/node, which the runtime (vLLM processes ~12 GiB + KV + activations + JIT) consumes to zero. 6+ nodes is the safe shape (67.6 GiB/node weights → 54 GiB headroom); 5 nodes (dgx5, currently off, 88.43 in config) would fit at ~81 GiB/node with ~40 GiB headroom if it joins the RoCE cluster. Keep the glm53-spark-nvfp4-4x recipe as a reference only (6 GiB KV pin, r28 image) — do not run on 4x.

Ops notes / lessons

  • Node clocks on the Sparks run +2 h ahead of UTC; the launcher's own INFO lines are UTC. Compare timestamps across both before drawing conclusions.
  • The 12:0x stop && serve restore chain broke silently: time stop | tail in a backgrounded list swallowed the stop's output, and the serve half never ran — production sat down 07:18→12:21 without a notice. Rule: background chains that chain state-changing commands must log to a file both halves read, or be run foreground; verify the second half actually started. The 12:21 direct serve started fine (4/4, loading at 12:23).
  • Per operator: the 4x node is not needed right now — the 12:21 4x aiden serve was stopped again at 12:24 and left down. Cluster is idle/clean; restore with sm121 serve --recipe glm53-flash-aiden-4x --cluster 4nodes-switch when the operator wants the agent back.