Inicio Explorar Ayuda
Iniciar sesión
sm121
/
launcher
2
0
Fork 0
Archivos Incidencias 0 Pull Requests 0 Wiki
Árbol: e00a59ec49
Ramas Etiquetas
launcher
/
history
/
night-shifts
/
2026-09-13.md

2026-09-13.md 30 KB
Histórico Raw

Night shift 2026-09-13 → 2026-09-14

Prepared 2026-09-13 ~21:30 UTC by the assistant. Operator brief: "посмотри обновления, поищи по форумам, готовь план для ночной смены".

State at handoff
4nodes-switch ON, serving the eugr DS4.1 prod (dsv41-flash-4x-b12x, started 13:31 UTC, 512K, DSpark k=5, Engram-on-disk + per-rank local sparse). This is the agent's own brain — confirm before stopping.
1node-5 (dgx5) ON, idle — the build node (no RoCE / 10 GbE only, so it cannot join the 4x fabric: 5-node lanes are out).
Checkpoint RedHatAI/GLM-5.3-MXFP4 downloading to the NAS: ~110 GB / 403.4 GB at 114 MB/s → ETA ~40 min. Verify before P0.
Kernel all nodes on 6.17.0-1031-nvidia — do NOT upgrade (see Updates #3).
git git.kilork.org still unreachable → commit locally; ~10 unpushed commits.

Updates found tonight (external)

  1. R37 lock published 2026-09-13 (lab, DS4.1): image localinferencelab/vllm:jovian-judgement-community-20260913-r37. The headline change is a kernel-dispatch overhead removal (repeated CPU arg-list expansion, mutable-schema inspection, scalar dtype allocations, tensor-equality calls) → uncached 32K prefill +43.65% (14,896 → 21,399 tok/s), C1 output +17.7%, verifier +11.9%, Sieve +8.2% (their RTX quartet; integrated, not per-PR). Also corrects the named high reasoning budget 50 → 75. Pins: vLLM c687594b8a8 (tree 3fb88fdf…), b12x f1c4e9dd5b1 (tree 480ea232…), LMCache 29bc5a2e unchanged, DeepGEMM 8b1392b9, torch 2.13.0 / flashinfer 0.6.17 — no dependency resync needed. → GLM stays at R35 (still the latest qualified GLM lock); R37 is the DS4.1 lock. Action: build profile written → docker-images/glm53-jovian-r27/profiles/jovian-r37-ds41.env (P1).
  2. eugr nightly-20260913 image built (13:09 UTC, supersedes nightly-20260910 = our DS4.1 prod image). Changelog not at the usual raw path (404) — read the repo/thread during the shift. Cheap A/B (P3).
  3. ⚠ Kernel regression (forum #383023): 7.0.0-1019-nvidia breaks NCCL/RoCE — ibv_reg_mr_iova2 failed with error Cannot allocate memory during TP communication. Reproducible kernel-only A/B; root cause traced (2026-09-13) to KHO's default scratch reservation marking ~9.3 GiB as unaccounted MIGRATE_CMA; fix PR NVIDIA/NV-Kernels#590. Working: 6.17.0-1032 (and our 6.17.0-1031). Action: do not upgrade the kernel on any node; if a node ever shows RDMA ENOMEM, check uname -r first.
  4. Forum #382897 ("DS4.1-Flash 552B on 4× Spark, 77.2 tok/s c1"): our upstream author (tonyd2wild — the repo our dsv41-flash image is built from). The 77 tok/s is a warmed, streaming, counting-prompt peak with up-to-5.95/6 draft acceptance; the same post lists prose 23, reasoning 39, math 47, code 52 tok/s — i.e. our measured 47–49 tg on agentic traffic is consistent; peak-on-repetition ≠ agent workload (clickbait, as the operator suspected). Two useful nuggets: 203 GB of Engram stays on disk so 552B fits 4x, and "2 GPUs were latched below 1 GHz — unplug the Spark to clear it, a reboot won't" → add a clock check before benches.
  5. New lab GLM docs: models/glm-5.3-flash-spark-tp2.md (09-12, Spark **TP2
    • vision** serving & measured capacity — GLM-5.3-Flash is natively multimodal and all our GLM lanes are text-only --language-model-only; a possible capability lane later). Also models/glm-5.1-mxfp4.md and the kld/glm-5.3-flash-* quant-fidelity reports.
  6. Other recent threads worth a skim only: #382939 (optimizing NVIDIA's official nvidia/GLM-5.3-Flash-NVFP4 on 2x), #382697 / #382927 (2x vs 4x comparisons), #382584 / #381911 (DS4-Flash-Vision-Exp on 1x).

P0 — GLM-5.3 (full) MXFP4 on 4× — the headline

Recipe recipes/glm53-full-mxfp4-4x (r35 stack, DCP4, 512K, KV 8 GiB pin, GMU 0.85, no spec on the first boot). Read its header for the full memory plan; the arithmetic in one line: 93.9 GiB weights/node + ~12 GiB engine + 8 GiB KV + ~4 GiB activations ≈ 118 of 121.63 GiB.

P0.0 prerequisites (~10 min)

  1. Free the cluster — only after confirming with the operator (it may be serving the agent): sm121 stop --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch, wait ~12 s before any serve.
  2. Download complete: du -sh $NAS/models--RedHatAI--GLM-5.3-MXFP4 ≈ 403 GB and 141 shards present (an .incomplete file = resume with huggingface-cli download RedHatAI/GLM-5.3-MXFP4). HF_HUB_OFFLINE=1 makes a missing shard a hard load failure.
  3. Clock check (nugget #4): per node nvidia-smi --query-gpu=clocks.sm,clocks_event_reasons.active --format=csv,noheader — a SM clock latched below ~1 GHz invalidates every bench number.
  4. sm121 health --cluster 4nodes-switch.

P0.1 boot probe — conservative (~60–90 min)

Edit the recipe args: remove --decode-context-parallel-size 4 and --cp-kv-cache-interleave-size 4; set --kv-cache-memory-bytes 4294967296 (4 GiB) and --max-model-len 131072. Then:

sm121 serve --keep --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch

Poll with docker logs --tail 2 sm121_glm53-full-mxfp4-4x_0 every 240–280 s.

  • Expect ~94 GiB/rank of weights (~10–20 min via instanttensor).
  • ABORT SIGNAL — a second "Loading safetensors" pass: that is the 2026-09-08 double-load pattern that OOM-thrashed all four nodes (sshd down, HA power-cycle). sm121 stop immediately; do not wait to see if it settles.
  • Success markers to record verbatim: Model loading took X GiB, the GPU KV cache size: … line, Application startup complete.
  • The KV line is the most valuable number of the night — it gives the real bytes/token for the full model (the parked estimate was ~54 KB/token; everything about 512K depends on it).
  • If the boot dies on memory: record the numbers and go to P0.4.

P0.2 smoke + capability (~20 min)

  • Chat: chat_template_kwargs: {"reasoning_effort":"low"} (glm45 parser, glm47 tool parser). Check coherent content + a tool-call round-trip.
  • Token-loop check (the W4A4-on-new-arch risk): generate ~300 tokens and confirm no repeated-token degeneration.
  • Long-context sanity if the pool allows: filler + needle at the largest context that fits, temperature 0 (the operator's floor is 512K — this is the capability claim).

P0.3 the 512K attempt (DCP4) — (~90 min + bench)

Restore the DCP4 lines, --max-model-len 524288, KV pin 8 GiB. Boot again.

  • If it rejects DCP (the GLM C4 DCP requires cp_kv_cache_interleave_size to be divisible by 4 / page-stride class that blocks DFlash2 on 2x) → the full model is DCP-incompatible; decision point (P0.4).
  • Verify the KV line reports concurrency at 524288.
  • Bench: sm121 bench --recipe glm53-full-mxfp4-4x --cluster 4nodes-switch --label dev65-full-mxfp4-dcp4 (~2.5 h, under pty automatically).
  • Compare against the GLM-5.3-Flash r35 prod anchor: 163/176 (93), 1,554 pp / 49.6 tg / 1,544 ms TTFT @d0 c1, KV 8.06M @512K. A ~740B model beating that on quality would be the story of the week; matching prefill while holding 512K would already make it a capability lane.

P0.4 fallbacks if it does not fit / degrades

  • Bench the P0.1 shape (no DCP, 131072, 4 GiB KV) as the honest capability datapoint.
  • 5–6 nodes are not available (dgx5 is 10 GbE-only, off the RoCE fabric), so 4x is the ceiling for this model tonight.
  • If it thrashes: sm121 stop; if the nodes do not recover, HA power-cycle (sm121 power off/on --cluster 4nodes-switch, ~3–5 min boot).
  • Record "full GLM-5.3 on 4x = does not fit / fits with N ctx" — a negative is a valuable result here (it retires the question).

P0.5 teardown (10 min)

sm121 stop; restore prod if wanted (sm121 serve --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch); write the report + history entry + commit locally.

P1 — build the R37 image (background, dgx5, no GPU contention)

The profile is written and dep-checked: profiles/jovian-r37-ds41.env (vLLM c687594b8a8, b12x f1c4e9dd5b1, LMCache/FlashKDA/DeepGEMM unchanged, torch 2.13.0 / FI 0.6.17 same as our GB10 pins).

  1. Create the recipe pair (copy, then swap 3 lines each):
    • recipes/dsv41-flash-jovian-r36-4x → recipes/dsv41-flash-jovian-r37-4x (docker_image = "jovian-r37-ds41:local", build_profile = "jovian-r37-ds41"; keeps the composed_model local-Engram hardlinks + 512K + DSpark K7 args).
    • optionally recipes/glm53-flash-r35-4x-dcp4 → recipes/glm53-flash-r37-4x-dcp4 (same image, GLM args) — the image is two-layer, so one build serves both lanes.
  2. sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan (~14 min warm buildx cache) → transfer to dgx1–4 (~4 min, 16 GB, parallel docker save | ssh | docker load).
  3. Sanity checks in the image (read-only, cheap): GlmMoeDsaForCausalLM in registry.py, mxfp4-pack-quantized scheme, liburing.so.2, the b12x native loader present.
  4. Note: the r35/r36 images now read as STALE against the build dir (it gained profiles) — warn-only, but a rebuild makes labels match.

P2 — R37 A/B (needs the 4x cluster; only after P0)

  • dsv41-flash-jovian-r37-4x: boot 512K + DSpark K7 + local Engram, bench dev65 → compare with the r36 datapoint (154/176, 3,774 pp / 822 ms TTFT) and the eugr prod (167/176, 1,642 pp / 1,738 ms). Purpose: does R37's +43.65% prefill claim hold on GB10, and does it close the quality gap that kept the lab stack out of prod? This is the only route we have to a faster DS4.1 prod.
  • Optional GLM arm: same image with the r35 GLM args (MTP3/DCP4/512K) → does the dispatch fix lift GLM prefill too? Baseline 163/176, 1,554 pp.
  • Delete the r36 recipes once R37 is measured (keep both recipes until then).

P3 — eugr nightly-20260913 (cheap, only if ≥3 h remain)

Swap docker_image in dsv41-flash-4x-b12x (or a copy) to eugr/spark-vllm-b12x:nightly-20260913, boot, bench dev65 → A/B against the nightly-20260910 prod numbers (167/176, 1,642 pp). ~2.5–3 h total.

Guardrails

  • Never stop / power off a serving cluster without the operator's OK.
  • serve --keep for every first/unknown boot (default --rm wipes crash logs); the 09-08/09-13 posts-mortems depended on it.
  • Poll in short loops (240–280 s); macOS has no timeout binary.
  • Bench convention: dev65 = 88 scenarios, seed 42, --hardmode --perf --concurrency 1,2; reports auto-land in recipes/<lane>/reports/ (manifest + recipe copy included).
  • OOM-thrash drill: container stays "Up", workers respawn forever, sshd can die on 2–3 nodes → sm121 stop, then HA power-cycle if needed.
  • One serve at a time on the 4x cluster (TP4 uses all four nodes; a stray second serve is an instant OOM).
  • Commit locally after every milestone (git server down).

Open questions for the operator

  1. OK to take the 4× cluster off prod tonight (agent downtime), and for how long? P0 needs the 4 nodes exclusively.
  2. Priority if time is short: (a) the full-model capability datapoint, (b) R37 (build + A/B), (c) both?
  3. If the full model cannot hold 512K on 4×, is a smaller context (131K–262K) acceptable for this lane, or is 512K a hard requirement (= the lane is dead on this hardware)?

Amendments 2026-09-13 ~22:00 UTC (after reading the lab's GLM docs)

Operator: build R37 first, then read models/glm-5.3-flash-spark-tp2.md and models/glm-5.1-mxfp4.md properly ("GLM-5.3 did not fundamentally change the architecture") and fold the logical consequences into the plan.

R37 build started

sm121 build --recipe dsv41-flash-jovian-r37-4x --build-node dgx5.lan (recipe created as a clone of the r36 one with image/profile swapped; the build fetches c687594b8a8 from voipmonitor/vllm + f1c4e9dd5b1 from voipmonitor/b12x with the HEAD/tree gates — both present upstream). Log: /tmp/build-r37.log. Next after it lands: transfer to dgx1–4, then P2.

P0 amendment — the MXFP4 MoE backend must be explicit

The r35/r37 tree's default MoE backend is b12x (r35 changelog). The lab's GLM-5.1 MXFP4 runbook says for MXFP4 GLM experts: --moe-backend flashinfer_cutlass_afp8 → native FlashInfer Cutlass MXFP4×MXFP8; do not set B12X_MOE_FORCE_A16; GLM must get standard SwiGLU (gpt-oss alpha/beta/clamp defaults must not apply). The recipe now sets flashinfer_cutlass_afp8 explicitly.

  • Already in our Sept tree (verified by image inspection): oracle/mxfp4.py:297 maps that name to FLASHINFER_CUTLASS_MXFP4_MXFP8; quantization/mxfp4.py routes non-gpt_oss to the native Mxfp4MoEMethod; the gpt-oss SwiGLU gates in experts/flashinfer_cutlass_moe.py:270-280 abstain — i.e. the June GLM-5.1 MXFP4 vLLM changes are merged.
  • RedHatAI ships quant_method=compressed-tensors / mxfp4-pack-quantized (CompressedTensorsW4A4Mxfp4), a different ingestion path from the lab's quant_method=mxfp4 — hence the fallback order: flashinfer_cutlass_afp8 → b12x (proven on our DS4.1 MXFP4 experts) → drop the flag and read the engine's selected backend from the log.
  • Numerics gate either way: a missing SwiGLU clamp (r33/r34 lesson) and W4A4 token-loop corruption both present as degenerate repetition. Run the P0.2 smoke with ~300 tokens and look at it, before any bench.

P4 — capability lanes worth a logical slot (from the lab docs)

(a) LMCache RAM + filesystem tiers → long context on a tiny GPU KV pool. Their TP2 GLM-5.3-Flash contract is MAX_MODEL_LEN=1048576 with only KV_CACHE_MEMORY_BYTES=4294967296 (4 GiB/rank) and a 1,048,320-token cold text run in 139.7 s, RAM restore 0.808 s, filesystem restore 1.279 s, zero recompute. Options (before the image name): -e CACHE_MODE=lmcache -e LMCACHE_L1_SIZE_GB=64 -e LMCACHE_L2_ROOT=/cache/lmcache-l2. Caveats: the 64 GiB L1 arena is preallocated pinned host shm, so host /dev/shm must be big enough (Docker --shm-size does NOT resize host shm); LMCACHE_L2_ENABLED=0 = RAM-only; the /cache volume persists FS objects across container replacement; objects cover 4,096 global tokens, attention pages 2,048 tokens per DCP rank; checkpoint files are hashed at startup (first log delayed while weights are read — keep them immutable while serving). Why it matters for us: it decouples "≥512K context" from "GPU KV pool size" — exactly the wall the full MXFP4 model hits on 4×. Worth a lane on the r35 GLM Flash profile (cheap: env only, same image).

(b) GLM vision on 2 GPUs. The lab's TP2/DCP2 GLM-5.3-Flash contract includes vision: one image per API request (across all turns), native ~8,000-feature ceiling, video disabled, full-image FlashAttention for the image pass, target KV FP8 + FP32 recurrent state, MTP3 probabilistic. Cold 1,048,320-token image input: correct answer in 143.0 s with min free VRAM 232.31 MiB/GPU. All our GLM lanes are text-only (--language-model-only) — this is a capability upgrade (screenshots/diagrams in agent traffic), and it is a documented, qualified contract rather than guesswork.

(c) Two conventions to copy into our recipes.

  • Explicit MAX_MODEL_LEN + explicit KV_CACHE_MEMORY_BYTES (they never let a lane silently shorten context; KV_CACHE_MEMORY_BYTES=auto is for auto-profiling and MAX_MODEL_LEN=-1 is the opt-in for accepting a shorter context; GMU does not resize an explicit KV allocation). We already pin both — keep it that way, and quote the KV line in every run report.
  • CUBLAS_WORKSPACE_CONFIG=:4096:1 bounds the cuBLAS workspace to 4 MiB per handle → 140 MiB less live allocation per rank (they measured it; not a confirmed cure for every community cuBLAS crash). Cheap, consider for the full-model lane where every 100 MiB counts.

Important caveat from their doc: "The memory mechanisms can also apply to TP4, but their magnitude depends on sharding and workload. No TP4 memory or performance improvement is qualified by the TP2 measurements. The TP2 NCCL configuration is not a TP4 recommendation." (They use 2 NCCL channels / 1 MiB buffer on TP2; we run 16 channels / 2 MiB on TP4 — do not copy the TP2 values.)

Reference numbers from that doc (2× GB10, TP2/DCP2, MTP3, 4 GiB KV/rank, 1M ctx): 32K prefill 10,648.9 tok/s; C1 MTP3 median output 170.77 tok/s; verifier 67.5 steps/s; R2 vs R1 = +0.41% prefill / +4.52% C1.

Progress 2026-09-13 ~22:20 UTC — both lanes are READY

P1 DONE — R37 image built and distributed. jovian-r37-ds41:local, checksum label 9350429b719b85df3009ad4c8ac86d58f16bd0ede942b7ad67a0f4baa2d1aeba, built on dgx5 in ~37 min (the deps layer was cold this time; the r36 build was 13m47s warm), transferred to all four nodes in ~4 min (4 parallel docker save | ssh | docker load), verified on dgx1:

  • vLLM 0.26.1rc0+glm53.r37.vllmc687594b (the r37 composed pin)
  • b12x 1.3.0, liburing.so.2 → 2.5
  • GlmMoeDsaForCausalLM registered; vllm/models/deepseek_v32/nvidia/ ships model.py, mtp.py, b12x.py, glm52_low_latency_gemm.py — i.e. the full-model arch, its MTP head AND a b12x path are all in the image.

P0 prerequisite DONE — the checkpoint is complete on the NAS. models--RedHatAI--GLM-5.3-MXFP4: 141/141 shards, 0 .incomplete, 376 GiB (= 403.4 GB decimal ✓ matches the HF index exactly).

Switch is the only thing left, and it needs a human decision. sm121 health --cluster 4nodes-switch → all four nodes online, 0 free, the eugr DS4.1 prod occupying them (sm121_dsv41-flash-4x-b12x_0..3, up 9 h). Both night lanes (P0 and P2) need the whole quartet, and stopping the prod interrupts whatever it is serving — possibly this agent's own model. So: confirm, then

sm121 stop --recipe dsv41-flash-4x-b12x --cluster 4nodes-switch   # ~10 s
# wait ~12 s
sm121 serve --keep --recipe dsv41-flash-jovian-r37-4x --cluster 4nodes-switch   # P2
# or the P0 ladder on glm53-full-mxfp4-4x

Suggested order now that both are ready: P2 (R37) first — it is a known-good configuration class (a boot is low-risk, ~50 min) and answers the +43% prefill claim on GB10; then P0 (MXFP4) with the whole remaining night, whose probe boot decides the capability question. Teardown/restore is P0.5.

P0 in flight (interim, ~23:55 UTC) — full MXFP4 on 4x: PROBE PASSED, DCP4 ladder

Boot ladder for glm53-full-mxfp4-4x-probe (no DCP, 131K, r35 img):

  1. --moe-backend flashinfer_cutlass_afp8 → argparse rejects it (not a CLI choice on r35; the CLI family is flashinfer_cutlass, the _afp8 key is an oracle-level thing).
  2. --moe-backend flashinfer_cutlass → accepted, but the flag is INERT for this checkpoint: the compressed-tensors W4A4-MXFP4 MoE scheme picks MarlinExperts on its own. The boot then died in the mxfp4 LINEAR kernel: the default priority [FlashInfer, Marlin, Humming, B12x] starts with the FlashInfer kernel, whose is_supported() wrongly accepts capability 121 (numeric has_device_capability(100) passes on GB10) while the cute-dsl mm_fp4 GEMM rejects it at runtime: mm_fp4 does not support backend 'cute-dsl' with capability 121 (kv_b_proj dequant, mla_attention.py).
  3. --linear-backend b12x → correct selection ("Using B12xMxFp4LinearKernel for MXFP4 GEMM"; verified per-component: family(120) True, b12x blockscaled OK, cute-dsl input-quantize works on 121) — but the UNCONDITIONAL b12x warmup crashes on this checkpoint's shapes: "Expected source and destination tensors to have the same rank, but got 3 and 4" (b12x dense_gemm via synthetic zero source). Filed-worthy for the lab.
  4. --linear-backend marlin + 8 GiB KV → BOOTED. (A 4 GiB pin was rejected: the engine needs 6.58 GiB/rank for ONE 131072-token request → the full model's KV = 51.6 KB/token/rank, fp8_ds_mla.) Pool: 159,232 tokens per 8 GiB pin (52.7 KB/token), max concurrency 1.21x at 131K. Smoke: 27×43=1161 ✓, tool call exact in 4.2 s ✓, 35,157-token needle answered "73451" in 5.1 s (≈7.8K tok/s prefill) ✓, no degenerate repetition. The 740B model works on 4× GB10.
  5. DCP4 first boot: DCP sparse indexer currently supports only cp_kv_cache_interleave_size=1 (got 4) — the x4 interleave is a Flash dense-MLA rule, not the DSA top-k path. Recipe fixed to interleave 1, rebooting.

DCP4 512K pool estimate: 8 GiB/rank ÷ (52.7 KB/token ÷ 4) ≈ 650K tokens ≥ 524288 ✓. Then smoke + dev65 bench dev65-full-mxfp4-dcp4, then P2.

Night shift results (final, 2026-09-14 ~02:45 UTC)

P0 — full GLM-5.3 (RedHatAI MXFP4, ~740B) on 4× GB10: WORKS (131K lane)

Booted, smoke-passed, benched (label dev65-full-mxfp4-131k, report in recipes/glm53-full-mxfp4-4x-probe/reports/):

  • Quality 158/176 (90, their 100-scale "91 Excellent", 1 safety warning TC-51, 87/88 scored).
  • Perf pp2048/tg128: d0 c1 934 pp / 19.0 tg / 2,124 ms, c2 867/26.5/ 3,185; d4096 c1 896/22.6/6,285; d8192 c1 877/22.8/10,406.
  • KV: 52.7 KB/token/rank (fp8_ds_mla), pool 159,232 tokens per 8 GiB pin, max concurrency 1.21× at 131K. Load ~10 min at ~600 MB/s from NAS.
  • MoE = MarlinExperts (scheme's own pick; --moe-backend is inert for the compressed-tensors path), linear = Marlin via --linear-backend marlin (the b12x warmup crashes on this checkpoint's shapes; the flashinfer kernel wrongly claims sm_12x support — both documented in the recipe).
  • 512K is BLOCKED on 4×: DCP4 is the only way 512K fits (26.3 GiB aggregate KV), and the DCP>1 sparse-decode path crashes identically on BOTH r35 and r37: after "DCP sparse indexer currently supports only cp_kv_cache_interleave_size=1 (got 4)" → interleave=1 → graph-capture dummy run dies in deepseek_v32/attention.py:596 with "shape '[16, 16, 512]' is invalid for input of size 524288" (the ×4 DCP factor; the dcp_manager.combine path is gated on use_pcp, which we do not set). Upstream bug to file with the lab. The 512K route for this model is the LMCache RAM/FS tiers (P4a below) or the DCP fix.

P2 — R37 DS4.1 on 4×: prefill champion, quality regressed

(Report recipes/dsv41-flash-jovian-r37-4x/reports/, label dev65-r37-ds41-4x; image label 9350429b, built on dgx5, 37 min.)

  • Perf: d0 c1 3,830 pp / 46.4 tg / 832 ms vs prod 1,642/47.0/1,738 → prefill +133% (2.3×), TTFT 2.1× better, decode flat (the dispatch removal is a prefill win; the lab measured it on RTX, GB10 matches and beats). c2: 2,688 pp / 71.8 tg.
  • Quality 150/176 (85), safety-capped (4 warnings: TC-51/74/82 + one more — the same long-horizon cluster as the r36 K7 run).
  • Verdict: eugr DS4.1 (nightly-20260910) stays prod of record for agent quality; R37 is the prefill/batch champion (long-context, low-interactivity workloads, or until the quality regression is fixed).

P3 skipped (eugr nightly-20260913 A/B)

The marginal nightly diff is not worth pulling an unverified external image in the last hours before teardown; R37 already answers the "newer tree" question decisively.

Teardown

  • dgx5 powered off ~22:33 UTC (operator authorized, at shift start).
  • 4nodes-switch: R37 lane stopped, then sm121 power off --cluster 4nodes-switch (operator's instruction for end of shift). No prod restore — the operator re-serves manually in the morning.

Morning recommendations (priority order)

  1. File the DCP>1 sparse-decode bug (repro: full GLM-5.3 MXFP4, DCP4, interleave 1, both r35 and r37 images; exact messages above) — this is the only thing standing between the full model and 512K on 4×.
  2. 512K via LMCache tiers on the full model (the lab's own pattern: small GPU KV + 64 GiB L1 pinned shm + /cache FS; check host /dev/shm capacity first — Docker --shm-size does not resize host shm).
  3. R37 as the prefill lane for batch/long-context; keep eugr for agent traffic. A/B an eugr nightly that inherits the r37 dispatch work if/when the lab merges it.
  4. The b12x-mxfp4-linear warmup rank bug + the flashinfer is_supported sm_12x false-positive (both workarounds are in the recipe; file both).
  5. Push all local commits when git.kilork.org recovers.

Morning shift (2026-09-14, ~05:50-07:00 UTC)

Operator request: bring prod up; R37 makes suspicious typos

"как будто оно не попадает в лексику" — check that nothing was exported/quantized/ serialized wrong.

Finding 1: R37 tree runtime-quantizes the LM head by default (NOT the cause, but real)

  • envs.py: VLLM_MXFP8_LM_HEAD default "1" — "Quantize eligible unquantized LM heads on b12x by default; =0 opts out." Implementation: model_executor/layers/vocab_parallel_embedding.py (sets runtime_lm_head_quantization = "mxfp8" for unquantized, untied LM heads).
  • The eugr/tonyd2wild prod image (nightly-20260910 base) has NO such feature — prod runs a BF16 head. Added VLLM_MXFP8_LM_HEAD = "0" to the R37 recipe (commit 902e989). Boot without the "Quantizing LM head shards to MXFP8" line = opt-out confirmed; KV pool shrank 10.39M -> 10.19M tokens (head now BF16 = more memory) — indirect confirmation.
  • But the typos persist with the BF16 head (verbatim/greedy probes below).

Typo evidence (R37 lane, temp=0 greedy, ~06:20 UTC)

Long 300-word Russian generation: "данних. Етот цыфровой оазис среди вечной мерзлоти... навикации... иследований... критically важной точкой опори... видлели колосальное количество енергии" (10+ mangled words in ~150 words). Code sample: "возврощает", "валідный" (Ukrainian і), "должин быть словарём". Identifiers check: all 5 near-identical config keys emitted correctly (code structure intact; it's the vocab surface layer that wobbles).

Bisect (in progress at 06:30)

Flag diff vs every clean lane: R37 contract uniquely adds --linear-backend b12x (prod: b12x MoE only, default linear; GLM r35 clean lane: B12X attention

  • instanttensor, default linear). Boot A = r37 image WITHOUT --linear-backend b12x. If typos persist -> Boot B: prod's spec config (K5 probabilistic/block, adaptive off) instead of K7 greedy/standard/adaptive-on; Boot C: full prod arg set on the r37 image (tree vs contract).

Naming correction (operator question)

Our prod 4x image is NOT eugr: dsv41-flash-b12x:local = thin wrapper over dsv41-flash:local = digest-pinned vllm/vllm-openai arm64 nightly + tonyd2wild overlay chain (dsv41-feat@e47aa780 + 7 boot-10 patches + FlashInfer 0.7.0rc1, all built in this repo). "eugr" was my sloppy label inherited from the deepseek-v4-flash-eugr*/glm53-flash-*eugr* lane naming; from now on: "DS4.1 prod" for the tonyd2wild lane, "eugr" only for the eugr-image lanes.

Boot D interrupted: cluster-wide outage (~07:27 UTC)

During Boot D (r37 + prod contract, engram off), all 4 nodes went unreachable mid-load (~3 min after container start): "Host is down" / "No route to host" from the live gateway (192.168.88.1 up), HA (192.168.88.232) alive but returning 401 to the known token. Earlier in the window one ping to dgx1 got through (1/2, 2 ms) - the nodes died abruptly, not degraded. All 4 boxes at once, in the middle of a 4x parallel NFS load. Suspects: PDU/segment power trip, or a physical event; nothing sm121 did can take a node's management NIC down (the serve had already passed container start and was at the API-wait stage). Polling for recovery; Boot D test is the only remaining bisect step. HA check (~07:38): switch.dgx1-4 = ON, sw6_mt/sw7_fan ON (dgx5-8 off as expected). So the nodes are powered but the 88.x management path is dead - either the nodes hung/crashed under load (powered but kernel dead) or the management segment/switch failed. Cannot distinguish from the Mac (no route to 200.x). Physical check needed: power-cycle the nodes or check the management switch.

GLM-5.3 eugr 2x re-bench (2026-09-14 ~09:20 UTC, dev66-glm53-eugr-2x-0910)

Operator redeployed glm53-flash-eugr-2x on dgx1+dgx2 (2nodes-dac) after the outage (he powered on only 1+2; dgx3/4/5 stay off by choice). Same image as the 09-11 run: eugr/spark-vllm-b12x:nightly-20260910, model local-inference-lab/GLM-5.3-Flash-NVFP4-Spark, 1M ctx, KV pool 1,162,277 tokens.

Result: 161/176 (91, ★★★★★), 1 safety warning (TC-51, same as the other lanes). Perf d0 c1: 1,326 pp / 28.1 tg / 1,458 ms TTFT (c2: 1,308/33.9/ 2,157); d4096 c1 1,663/30.5/3,369; d8192 c1 1,762/31.8/5,297.

vs anchors: eugr 2x 09-11 166/176 (same stack, -5 = variance); r28 2x (our image) 158/176, 1,475 pp / 27.2 tg / 1,362 ms — eugr wins quality, r28 wins prefill by ~11%, decode/TTFT near-tie. Verdict unchanged: eugr 2x = quality lane, r28 = prefill lane (2x).

Also this morning: report/web "running" status was stale for the r37 lane — the run-ledger (.lock) running record from the killed Boot D serve was never closed (sm121 stop can't verify with dgx3/4 offline). Closed the record per mark_stop semantics + docker rm'd the exited r37 containers on dgx1/2. Systemic options on the table: (1) report cross-checks ledger against the live probe, (2) sm121 runlock close <recipe> offline command.

Report stale-status + sm121 runlock (2026-09-14, ~11:40 UTC)

Operator asked for both fixes after the r37 false-"up":

  1. sm121 report --serve now cross-checks the run ledger against a live fleet probe every refresh tick (30 s, only when an open run exists): an open run whose recipe has no running container on any reachable node is rendered orange "stale" with a sm121 runlock close <recipe> tooltip instead of green "up". Inconclusive (unknown cluster, unreachable node) stays green. tmux deploys are never probed. src/report/stale.rs (mark_stale + 4 unit tests), serve.rs refresh hook, page.html state cell.
  2. sm121 runlock list|close <recipe> — offline ledger management (the mark_stop file op without node verification). RunlockCommands lives in runlock.rs (cli.rs at the 500-line cap). All gates green (310 tests). Live-verified: eugr 2x shows up (probe sees the containers), r37 shows stopped.