Acasă Explorează Ajutor
Autentificare
sm121
/
launcher
2
0
Bifurcare 0
Fisiere Probleme 0 Trageți solicitările 0 Wiki
Arbore: 95e45ae40d
Ramuri Etichete
launcher
/
history
/
night-shifts
/
2026-09-08.md

2026-09-08.md 25 KB
Istoric Crud

Night shift 2026-09-08

RoCEnante 2x A/B (r27-spark, 4nodes-switch) — DONE

Goal: operator asked to (1) make VLLM_ENABLE_ROCE_ALLREDUCE=1 the default in the r27 recipes and (2) test the 2x variant (the 4x DCP4 A/B from 09-07 showed decode +31…+90%).

Timeline (UTC; node clocks are +2 h ahead — keep that in mind when reading node-side log timestamps):

  • ~09:19 production 4x aiden stopped (operator OK) for the 2x work.
  • Arm A (NCCL control, glm53-flash-r27-spark as-is): 2x on dgx1+dgx2, bench c1,2 seed 42 → 158/176 (90).
  • Recipe rollout: VLLM_ENABLE_ROCE_ALLREDUCE=1 added + --disable-custom-all-reduce removed in both r27 recipes; the interim glm53-flash-r27-4x-dcp4-roce clone deleted, its reports consolidated into glm53-flash-r27-4x-dcp4/reports/ (commit fb3fbf5).
  • Arm B attempt 1 (10:13): picked dgx1+dgx3; the dgx3 worker container died within ~2 min, the rank-0 head hung in NCCL init (orphan). First sighting of the stop-timing race: sm121 stop returned before the previous (production 4x aiden) container had fully released the GPU on dgx3.
  • Operator directive: stop must actually wait for the containers to be gone; --no-wait to opt out. Implemented: stop now runs a per-node verification poll (30 × 2 s) after docker stop and errors if labeled containers are still running (commit a33cfa8). The web dashboard stop job uses the waiting variant.
  • Arm B attempt 2 (10:24, dgx2+dgx3): same dgx3 worker death.
  • Arm B attempt 3 (10:54, dgx1+dgx2): both workers survived; RoCEnante activated (world=2, hcas=rocep1s0f0,roceP2p1s0f0, dispatch ['B12X_ROCENANTE','PYNCCL']). Bench ran to ~43% and the engine died: TileLang trace crash in glm5next/nvidia/ops/glm_kpool.py ('int' object has no attribute 'to' at block.to(tl.int64)) → EngineDeadError, clean shutdown. NOT RoCEnante-specific on its face (the NCCL arm completed the same scenarios fine — but see below), one-off.
  • Arm B re-run (11:23, dgx1+dgx2, RoCEnante): clean full run → 162/176 (92), 100% completion, zero 500s. Crash did not reproduce.

2x A/B result (dev46, seed 42, c1,2):

cell NCCL tg RoCE tg Δ tg
d0 c1 23.4 27.4 +17%
d0 c2 33.5 33.3 −0.6%
d4096 c1 25.3 24.4 −3.6%
d4096 c2 31.8 30.6 −3.8%
d8192 c1 25.2 24.7 −2.0%
d8192 c2 29.1 29.4 +1.0%

Quality: 158 → 162 (+4). Scenario flips: up TC-03/33/49/61/62/68/75 (TC-33 fail→pass); down TC-40/50 (pass→partial), TC-88 (pass→fail). Conclusion: at 2x RoCEnante is a quality win and a small d0-c1 decode win; the 4x DCP4 decode win (+31…+90%) is the headline. RoCEnante is now the default in both r27 recipes.

sm121 bench (operator request)

"Given how we run our benches, should we add a command to the launcher that does all the dirty work?" — yes. Implemented (commit 1ef3eeb): sm121 bench --recipe R --cluster C --label L [--concurrency 1,2] [--seed 42] — waits for API readiness (900 s), runs tool-eval-bench (PATH) in a scratch cwd under ~/.cache/sm121/bench/, lands <UTC-date>_<label>.{md,json,log} into recipes/<lane>/reports/, prints the final score + perf rows. -d dry-run prints the exact command. Served name/port come from the recipe (--served-model-name in args / port). The same commit moves the stop handler from main.rs to docker/stop.rs (line-budget).

r28 full-stack evaluation + version profiles (evening shift)

Operator request: a build_profile entity — a named pin set referenced from recipes instead of copy-pasting the whole [build.versions] table. The half-cut r28 recipe (vLLM advanced, b12x/LMCache left at r27) was exactly the failure mode this kills.

  • Feature (15e9b62): build_profile = "<name>" → docker-images/<dir>/ profiles/<name>.env (same KEY=VAL format); merge base < profile < recipe [build.versions]; merged file written to the node before build.sh and folded into the sm121.checksum label. profiles/glm53-r28.env carries the full r28 lock (vLLM 62f6385c / b12x 7216f6b9 / LMCache 934c287a — all three forks cut simultaneously 09-06, strictly ahead of r27 pins).
  • TOML context trap (b5f0662): the first build_profile edit landed after the [[extra_mounts]] block → the bare key parsed into the LAST mount table and serde silently ignored it; the build ran with base pins while the recipe "had" the override. Bare keys must stay top-level (tables like [build.versions] reset the context and are safe anywhere). Verified end-to-end with a live load_recipe + build_checksum probe before re-running the build; pitfall documented in both skills.
  • Build: ~6 min (buildx cache), label 14271940…; image verified in-container (vllm r28, lmcache 934c287a, b12x 1.3.0 — the r28 b12x moved to static versioning, so the scm pretend no longer applies; pin still enforced by the HEAD+tree gates).
  • Memory gate flake: first serve died at worker init — Free memory 107.86 < 0.9×121.63 = 109.46 GiB (the gpu_memory_utilization admission gate, vLLM #48140): docker load of the 16 GB image + ~12 GiB of starting vLLM processes pushed the check under the line. The launcher's existing drop_caches (post-sync, pre-container, default on) ran but the margin is still razor-thin; a retry with settled cache booted fine. Follow-up considered: GMU 0.88 in the 4x recipes (gate → 107.0 GiB; KV is explicitly pinned to 16 GiB so capacity is unchanged).
  • Bench (full r28, seed 42, hardmode, c1,2): 155/176 (88/100) vs r27 RoCE 160/176 and mixed-stack r28 159/176; median turn 3.3 s (r27 3.9 s — the r28 scheduler win holds). 8 hard scenarios lost points vs r27 (TC-21/ 49/51/61/67/80/82/88); TC-80+TC-88 failed only on full r28. But: r27 itself spread 155–162 across its A/B runs, and the r28 b12x M8/M1 changes are layout-only ("no model math, scale precision, or output reduction is changed"), so 155 is inside the observed run-to-run band. Re-run queued to pin the variance before judging the stack.
  • Re-run: 159/176 — the 155 was variance, not a regression: 8 scenarios flipped between the two identical-stack runs (incl. TC-80/TC-88, which now PASS). Full r28 (155–159) and r27 (155–162) are statistically tied on quality; r28 wins on speed (median turn 3.0–3.3 s vs 3.9 s; TTFT ~3.6× better per the mixed run). Full r28 stack = production candidate. Also hardened startup: GMU 0.90 → 0.88 in all r27/r28 4x/2x recipes (ff1f727) so the gpu_memory_utilization admission gate (107.0 GiB, not 109.5) survives the ~12 GiB of starting vLLM processes on top of drop_caches (which already ran post-sync, pre-container, default on).

Pending: (1) b12x O_DIRECT loader test — DONE, blocked upstream: --load-format b12x boots the loader (O_DIRECT, 44 shards, 92.6 GB physical, 404.1 s execution, 40.07/44.59 GiB into write-combined shared storage) but dies at b12x/loader/_checkpoint.py:370: ValueError: checkpoint destination was not allocated as a weight: model.language_model.layers.45.eh_proj.weight — the GLM/Step-3.5-MTP eh_proj linear is missing from the pool allocation plan. Report to the fork owner; keep --load-format instanttensor for GLM-5.3 NVFP4 meanwhile. Test recipe kept: glm53-flash-r28-4x-dcp4-b12xload (re-test once the fork fix lands, then delete). (2) Discord env vars — TESTED: --mamba-cache-mode align + VLLM_GLM53_SPLIT_TARGET_BLOCK_SIZE=2048 / _MAMBA_BLOCK_SIZE=256 boots clean (split2048 recipe): KV 8,269,149 tokens (+2.6% vs 8,060,928 standard at the same 16 GiB pin — better page packing), bench 156/176 / 3.1 s — inside the standard-r28 noise band (155/159), so neutral for sequential serving; the win (256-token recurrent checkpoints → finer prefix reuse) only shows under high prefix-cache hit workloads. Keep OFF by default; it is the #646 page geometry the DFlash2 path expects.

GLM-5.3-NVFP4-Spark 386B boot test (operator request) — NOT 4x-capable

local-inference-lab/GLM-5.3-NVFP4-Spark (386B params, GlmMoeDsaForCausalLM — registered in the fork under vllm/models/deepseek_v32, 78 layers, hidden 6144, 8 experts/tok, 1M context): 406 GiB of safetensors on disk (84 shards), ~101.5 GiB per node on 4x. First load pass completed (294 s at 1.5 GB/s), the engine then restarted and began a SECOND full load while the first copy was still resident → 2×101.5 > 121.63 GiB → all four nodes went OOM-thrashing (0 GiB available, sshd unreachable on 3 of 4). Kill issued; the engine kept RESPAWNING OOM-killed workers (the container stayed up), so the nodes never recovered on their own — operator approved a power cycle via HA (switch.dgx1-4 off → on, 2026-09-09 ~05:30 UTC). All four back clean: 116–117 GiB available, 0 sm121 containers.

Answer to the operator: 4 nodes is not enough — a single copy already leaves ~20 GiB/node, which the runtime (vLLM processes ~12 GiB + KV + activations + JIT) consumes to zero. 6+ nodes is the safe shape (67.6 GiB/node weights → 54 GiB headroom); 5 nodes (dgx5, currently off, 88.43 in config) would fit at ~81 GiB/node with ~40 GiB headroom if it joins the RoCE cluster. Keep the glm53-spark-nvfp4-4x recipe as a reference only (6 GiB KV pin, r28 image) — do not run on 4x.

Ops notes / lessons

  • Node clocks on the Sparks run +2 h ahead of UTC; the launcher's own INFO lines are UTC. Compare timestamps across both before drawing conclusions.
  • The 12:0x stop && serve restore chain broke silently: time stop | tail in a backgrounded list swallowed the stop's output, and the serve half never ran — production sat down 07:18→12:21 without a notice. Rule: background chains that chain state-changing commands must log to a file both halves read, or be run foreground; verify the second half actually started. The 12:21 direct serve started fine (4/4, loading at 12:23).
  • Per operator: the 4x node is not needed right now — the 12:21 4x aiden serve was stopped again at 12:24 and left down. Cluster is idle/clean; restore with sm121 serve --recipe glm53-flash-aiden-4x --cluster 4nodes-switch when the operator wants the agent back.

09-09: 2x on 2nodes-dac (DAC A/B, operator request) + bench pty fix

Operator: run the 2x tests on 2nodes-dac too — a direct DAC link may give speed.

  • glm53-flash-r28-spark (r28 image + RoCE-ON) on 2nodes-dac (100.x fabric, dgx1+dgx2). RoCEnante is live on DAC as well (b12x_roce_all_reduce.py: "all-reduce is live", "all-gather is live") — CX-7 RoCE does not need a switch. Load 304.5 s (91.4 GB/s), KV 2,206,618 tokens (10 GiB pin).
  • Bench 2x DAC: 158/176 (90/100), 1 safety fail (TC-43), median turn 5.5 s. Perf d0: c1 1,475 pp / 27.2 tg / TTFT 1,362 ms, c2 1,396 / 35.2 / 1,949.
  • Verdict: direct DAC gives no advantage — numbers match the 09-08 r27 2x RoCE A/B on the same DAC fabric to within ±2% (armB: 1,500 / 27.4 / 1,398). For TP2 the inter-node allreduce share is small; switch vs direct pair is indistinguishable. RoCE-ON stays the default (NCCL tg is lower: 23.4 vs 27.4).
  • Server stopped, cluster clean (0 sm121 containers).

09-09: sm121 bench hung after the perf phase (2x) — pty fix

Symptom: 3/3 sm121 bench runs on the 2x server hung right after the llama-benchy perf phase (child tool-eval-bench: 0% CPU, socket ESTABLISHED, rich output frozen; the server itself healthy — /v1/completions answers). 4x runs through the same launcher worked (4/4) — looked "2x-specific".

Isolation (A/B on the live 2x server):

  • A: manual run with redirect to file → works (94/100 c1, up to TC-88).
  • B: through | tee → "dies" with `[Errno 32] Broken pipe — but that is a test artifact (tee wrote to the closed capture → tee died → pipe broke), not a bug.
  • C: script -q /dev/null (pty) → works (TC-14/15 in 3 min, passed the perf→scenarios transition where the pipe variant hung).

Diagnosis: tool-eval-bench's rich Live display deadlocks on a non-tty stdout pipe at the perf→scenarios transition (a race in the render loop); on a tty (pty) it is clean. 4x "worked" by luck (timing-dependent race).

Fix (bench.rs): the child now runs under /usr/bin/script -q /dev/null (BSD form, macOS; fallback to direct exec if script is absent). Gates green (271 tests). Confirmed by a live run: the 2x DAC bench ran to the end (158/176) — the transition passed without hanging.

09-09: 386B full model (GLM-5.3-NVFP4-Spark) — does not fit on 4 nodes

Operator: run the full model local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (406 GiB, 78 layers, hidden 6144, 8 experts/tok, 1M ctx, GlmMoeDsaForCausalLM = fork registry) on 4 nodes.

  • Recipe glm53-spark-nvfp4-4x (r28 image, KV 6 GiB, GMU 0.88) created.
  • OOM, proven empirically: weights 406.1 GB / 4 = 101.5 GiB/node + engine overhead (vLLM workers + docker daemon ~12 GiB) ≈ 113.5+ GiB > 121.63 GiB HBM — and vLLM does a double load (first pass 294 s @ 1.5 GB/s, physical 92.6 GB; the engine restarts and loads a second time while the first copy is still in memory) → 2×101.5 > 121.63 → all 4 nodes went into OOM-thrash (0 GiB avail, sshd down on 3 of 4).
  • The vLLM engine respawned OOM-killed workers forever (container "Up"), the nodes did not recover on their own → HA power-cycle (switch.dgx1-4 off→on via the HA API, token from config) — all 4 recovered clean (116-117 GiB free, 0 containers).
  • Verdict: 4 nodes are not enough (even with a perfect weight distribution, 101.5 + overhead does not fit in 121.63 with a safe margin). Need 6+ nodes (67.6 GiB/node → 54 GiB headroom) or 5 nodes (81.2 GiB/node → 40 GiB headroom) with dgx5 (192.168.88.43, currently off) joined to the RoCE fabric. Recipe kept as reference; do not run on 4x.

09-09: DFlash2 variant of the full model — does not exist

Operator: "run the DFlash2 variant since the full model doesn't fit". Checked HF: local-inference-lab has only GLM-5.3-NVFP4-Spark (435.6 GB) and GLM-5.3-NVFP4 (464.9 GB) — no DFlash2 variant of the full model (the DFlash2 checkpoints are 1.3 GB draft models for the Flash version: GLM-5.3-Flash-DFlash2*). DFlash2 is a speculative drafter — it adds memory (draft model + drafter KV), it does not save any — for the full model it would worsen the OOM. Answer to the operator: the variant does not exist; the full model needs cluster expansion (see above).

09-09: DFlash2 on r28 — still blocked (dtype mismatch, refinement)

Operator: "check whether DFlash works now". Rebuilt the glm53-flash-r28-dflash2-2x recipe from git history (b3babad), re-pointed it at the r28 image + build_profile. Launch → BLOCKED, a new failure point:

ValueError: No valid attention backend found for kv_cache_dtype=fp8_ds_mla, use_mla=False, block_size=256

Diagnosis against the r28 sources (vLLM 62f6385c):

  • The DFlash2 drafter (qwen3_dflash.py) gets the KV dtype from the shared engine config (get_kv_quant_mode(self.kv_cache_dtype)), and the GLM-5.3 engine runs fp8_ds_mla.
  • The DFlash attention backend (vllm/v1/attention/backends/dflash_attn/__init__.py:219,227) requires bf16: TypeError: q and out must be bfloat16 / K and V caches must be bfloat16.
  • The r28 commit fa4720ea ("restore GLM boundaries across DCP and DFlash groups") fixed prefix-cache/boundaries, but not the dtype constraint.
  • In the Aiden lane DFlash2 works in their fork — they have a bf16-KV override for the drafter. Our r27/r28 fork has no per-drafter dtype override.

Status: blocked in the fork — needs either DFlashAttention with fp8_ds_mla or a per-drafter bf16-KV override. A report for the fork owner (voipmonitor) is queued. The r27 DFlash2 blocker (previous session) has the same root cause — not a regression.

09-09: Prod switched to full r28 (operator decision) + upstream issue writeups

Operator: (1) switch prod to r28; (2) dgx5 is 10G-only for now, not joining the cluster (386B keeps waiting on an infra decision); (3) prepare issue descriptions, but "it's strange they say everything is alive" — look at sparkring.

  • Prod = glm53-flash-r28-4x-dcp4 on 4nodes-switch (replaced aiden-4x). Launch: weights 184G + MTP 12.3G (instanttensor), RoCEnante live, KV 8,060,928 tokens, API verified with a completion request. ~09:00 UTC.
  • Sparkring recon (FujitsuPolycom/sparkring):
    • SparkRing = SIRCL + RoCEnante (our RoCEnante comes from their third_party/b12x_roce) + patched NCCL; switchless rings 2x/4x/6x.
    • DFlash2 in sparkring is a RETIRED profile (JJ R8/R10: FP8 KV + BF16 DFlash2 depth 7, vLLM dev/jovian-judgement tag e02b1746 + b12x 9ae41c5c) — superseded by native MTP3, which is exactly our current prod path (MTP k=5, head in-checkpoint). I.e. "everything is alive" because their current path touches neither the b12x loader nor DFlash2.
  • 386B full model: no DFlash2 variant exists (1.3 GB draft models only for the 184G Flash); a drafter adds memory, it does not save any.
  • Upstream issue writeups (for voipmonitor/local-inference-lab): specs/tech-architecture/upstream-issues-2026-09-09.md — (1) b12x O_DIRECT loader: eh_proj.weight missing from the pool allocation plan (b12x/loader/_checkpoint.py:370) — small fix if the plan is config-driven; (2) DFlash2: the bf16-KV drafter wiring exists in the JJ R8 line, absent in the r27/r28 base — question to the owner: is native MTP the long-term path? (then nothing to fix) or port from JJ R8; (3) 386B = a capacity note, not a bug.
  • Cluster: prod 4x r28 serving; 2nodes-dac free; dgx5 off (10G only).

15:00 UTC+2 — SparkRing shared image: prepared, waiting for a window

Operator: "there is an image — let's try it, minimal effort". Public digest-only image ghcr.io/fujitsupolycom/sparkring@f25cdb6... (22.5 GB), pulled to dgx1+dgx2 (first pull by :latest — manifest unknown, the image is published by digest only; second pull — ok; dgx2 disk was 95% full, 49G left — tight, but it fit).

Recipe glm53-flash-sparkring-2x: profile glm53-flash-spark-tp2-mtp3 (TP2/DCP1, RoCEnante 2-path, KV 8.75 GiB fp8, max-len 262K, GMU 0.87, MTP3 humming, block-size 256, b12x loader, patched NCCL LD_PRELOAD, 43 env vars — all from their quickstart profile.json). Model — the df116c4f snapshot from NFS via an extra_mount to /models/target (read_only); entrypoint = the image default (verify_sources.py --serve wrapper).

Mistakes during prep (all fixed): (1) TOML: a bare args = [...] after [[extra_mounts]] silently landed inside the mount (same trap class) — moved args/env before the table headers; (2) recipe without name — required field; (3) dry-run showed --pp 2 — the launcher defaults to pp=nnodes, the recipe needs an explicit pipeline_parallel_size = 1. Dry-run is clean now.

Why serve failed with "0 nodes" (4 attempts): not a bug, not SSH — the 2x pair (dgx1+dgx2, ssh 88.14/88.10) are the same physical machines as in 4nodes-switch, and they were running production r28 4x (sm121_glm53-flash-r28-4x-dcp4_0/_1). The sm121-label probe sees them → occupied → 0 free. Health/web show a per-cluster view (2nodes-dac says "no containers" — its own labels), which masks the physical occupancy. Conclusion: any sparkring profile (2x and 4x) needs sm121 stop --recipe glm53-flash-r28-4x-dcp4 first (production downtime) — in an agreed window. Operator informed, waiting for the decision.

16:00–17:00 UTC+2 — SparkRing shared image: 2x run (with degradation), 4x in progress

Operator: clean dgx2 (459.5G docker + 9 orphaned images + 93G LTX/kk3 → disk 30%) and "we have a couple of hours for tests on the whole cluster". Prod r28 4x stopped.

Image wrapper (operator's idea): docker-images/sparkring-shared/ — a thin FROM <digest> + wrapper. Solves the digest-only pull problem (image tagged, prune-safe, standard launcher build/sync flow). Along the way, two real bugs in their image on our infra:

  1. b12x loader JIT: compiles _storage.c with -L/opt/cuda-13.3/lib64 -lcudart, but in the CUDA 13 image /opt/cuda-13.3/lib64 does not exist and libcudart lives only as pip nvidia/cu13/lib/libcudart.so.13 (no dev symlink) → the first start died with ld: cannot find -lcudart. Fix: symlink + pre-build in our Dockerfile (native cache baked into the image).
  2. wrapper warmup deadlock: serve-with-warmup.py waits for the API on 127.0.0.1:$PORT (default 8015) and sends warmup requests with model=$SERVED_MODEL_NAME (default glm-5.3-flash). Without PORT=8000 and SERVED_MODEL_NAME in env, the wrapper retries a dead port forever (1800 s deadline) — the API is up, health 200, but the warmup never starts. Both env vars added to the recipes; warmup timeout 600→1800 (cold JIT compile).

SparkRing 2x (2nodes-dac, profile tp2-mtp3, RoCEnante) — bench d0 hardmode:

  • 89/100 (101/114), 12/14 categories at 100%; 2 safety failures (TC-43, TC-59)
  • BUT: completion 64.8% — the server died at TC-57 (31 scenarios TC-57–88 excluded: connection_error "All connection attempts failed"). TC-55/56 passed (21.7s/28.9s), TC-57 hung for 129.7s — and that was it. The containers vanished (--rm); no OOM or docker events in journal/dmesg → probably their own sparkring-memory-guard (4 GB floor, from the profile lifecycle) killed the stack.
  • Perf d0 (before the death): 739 pp / 31.9 tg / TTFT 3,741 ms (c1) — prefill 2× slower than our r28 2x (1,475), TTFT 2.7× worse (1,362), decode tg +17% (31.9 vs 27.2). c2: 1,195 pp / 50.0 tg / 3,165 TTFT.
  • 2x verdict: quality intact, speed worse (especially prefill/TTFT), reliability — failure (did not survive 30 min under load; our r28 2x passes the full hardmode stably).

4x switched (profile tp4-switched-mtp3, plain NCCL, KV 2.28M tokens, 1M ctx): launched, bench running.

16:50 UTC+2 — SparkRing 4x: near-complete run, same degradation; prod restored

SparkRing 4x switched (profile tp4-switched-mtp3, plain NCCL — RoCEnante OFF, KV 2.28M tokens, 1M ctx, GMU 0.80, fastsafetensors):

  • 89/100 (152/170), completion 96.6% — died only on TC-85/86/88 (the last 3-4 scenarios): TC-84 pass 33s → TC-85 timeout 140s → TC-86 timeout → TC-87 degraded → TC-88 500 → containers vanished from all 4 nodes.
  • Perf d0 (before degradation): 1,230 pp / 59.2 tg / TTFT 2,630 ms (c1), c2: 1,735 / 62.9 / 1,581. Decode clearly faster than our r28 4x (~46.8 tg) — +26%; prefill 1,230 vs r28 ~1,500-1,600.

Overall sparkring shared-image verdict (2x + 4x):

  • Quality intact: 89/100 on both lanes (our r28 2x = 90, 4x = 90-91).
  • Speed: 2x — prefill 2× slower than ours, TTFT 2.7× worse, decode +17%; 4x — decode +26% (59.2 vs 46.8 tg), prefill/TTFT behind.
  • Reliability is the main failure: both profiles die under sustained hardmode load (2x at ~57/88, 4x at ~85/88). No kernel OOM in dmesg/journal at the moment of death → their own sparkring-memory-guard (memory_guard_minimum_floor_bytes=4GB from the profile lifecycle) does the killing — a stock self-defense, but our r28 passes 88/88 stably. The operator's hypothesis ("just a heavier warmup") is not confirmed: the death is mid/end of the bench under load, not in the warmup phase.

Recipe cleanup per operator request:

  • PORT removed from both recipes' [env] — the launcher now injects PORT=<recipe.port> into the docker env itself (command.rs, generic: needed by wrapper-style entrypoints like sparkring's serve-with-warmup that poll 127.0.0.1:$PORT; inert for stock vllm/sglang — they take --port as a CLI flag).
  • model = the standard alias local-inference-lab/GLM-5.3-Flash-NVFP4-Spark (repo id, like the r28-spark production) — vllm resolves it from the HF cache refs/main = latest snapshot (df116c4f as of today).

Prod restored: r28 4x brought up on 4nodes-switch (16:53 UTC), API verified. Awaiting the operator's decision on repeat runs ("run it twice" — the rerun has not been done yet, waiting for a window).

17:30 UTC+2 — PORT injection reverted to explicit recipe env

Operator: injecting PORT into every container is an implicit/magical thing. The recipe [env] table already is the explicit mechanism (renders -e KEY=VAL). Reverted the launcher-wide PORT=<recipe.port> pair (command.rs) and put PORT = "8000" back into both sparkring recipes' env. Stock recipes get no PORT env. Gates green (275 tests), dry-run verified: sparkring -e PORT=8000, r28-spark 0.

18:10 UTC+2 — [env] values can resolve from recipe fields ({ from = "port" })

Operator: a hardcoded PORT = "8000" in [env] next to port = 8000 is fragile (two places for one value); the launcher-wide PORT injection was rejected as implicit. Solution: [env] values are now an untagged EnvValue enum — plain string (unchanged) or a reference into a recipe field: PORT = { from = "port" }. Recipe::resolved_env() resolves at serve time (docker -e, tmux export, publish run.sh); unknown from targets fail fast listing the supported ones (currently: port). No string parsing — the TOML structure carries the intent. Both sparkring recipes use it. 279 tests green (4 new: resolve, error, mixed TOML parse, docker render).

Refactor follow-up (same commit batch): Recipe + EnvValue moved from config/types.rs to config/recipe.rs (types.rs hit the 500-line gate at 554; recipe.rs is 183 lines now). The pre_serve facade re-exports moved from types.rs to config.rs, trimmed to the four names non-test code imports through crate::config:: (test modules now import RecipeArgs/VenvSetup directly from crate::pre_serve). Gates: 279 tests, lines ok.