launcher/AGENTS.md

AGENTS.md — sm121

Rust CLI (edition 2024) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (russh). Docker default, tmux alternative.

Memory discipline (the web process runs 24/7 on a 4 GB Pi)

The add-on image builds the launcher with --features mimalloc (the CI Dockerfile does this; a plain cargo build stays on the default allocator). The 2026-10-13/14 daily Pi OOMs (dmesg oom-kill … task=sm121, anon-rss ~2.7 GiB) were rooted in the report panel's file watcher: recipe_dirs() always includes the cwd (.), the container cwd is /, and notify's recursive walk followed the /sys subsystem symlink loops — an endless walk (~4.6 GB/h allocation churn) the allocator ratcheted into RSS until OOM. Fixes: watchable_recipe_dirs() (watch only real recipe workspaces — never /), one shared HA REST client (a reqwest+rustls build per probe tick was another ~2 GB/h), the lazy report refresh (full re-collect only on debounced file events; the 30 s tick only re-runs the stale cross-check, riding the dashboard's fleet bus — no extra SSH), the SSH key parsed once and cached (ssh.rs), and mimalloc as the backstop. Post-fix RSS is ~40 MiB flat (was 270 MiB/hour growth). Heap hunting: cargo build --release --features dhat + clean SIGTERM exit dumps a per-callsite retained-heap dhat-heap.json (summarize with /tmp/dhat-sum.py); the profiler must be created in main before anything allocates. Note: axum graceful shutdown hangs on live SSE — web closes connections on the signal instead.

Dev quick-ref

Dev quick-ref

Action	Command
Build	cargo build
All gates	make check (fmt → clippy → test → ≤500 lines)
Dry-run	cargo run -- -d serve --recipe deepseek-v4-flash
Tests	cargo test (65, all passing)
l10n	locales/{en,ru,es,it,fr,de}/main.ftl — Fluent FTL files

See DEVELOPMENT.md for architecture, module layout, and dev workflow.

Commands

serve/stop/bench/logs/health/doctor/exec/power/tmux/init-host-keys/config/web/report/runlock/validate
repos/promote/archive/unarchive

validate — offline recipe-contract gate (no cluster, no SSH): checks every recipes/<name>/recipe.toml against the launcher contract — docker_image_dir must exist under docker-images/, port must be the cluster default 30000, no floating :latest on engine images (alpine test images exempt), models must float (no @revision), required_nodes within 1..=8. Active recipes: violations are errors (exit 1). Archived lanes are grandfathered — their errors downgrade to warnings (advisory, always exit 0). Also runs as a cargo test over the repo's own recipes/, so make check fails if a new recipe breaks the contract. Rules live in src/validate.rs.

serve records the launch in the recipe's run ledger (recipes/<name>/.lock, gitignored) and then waits for the engine to actually come up — polls GET /v1/models on the head node up to 30 min (--no-wait skips); a run that never reaches readiness is closed with its uptime and the serve fails. Follow mode watches liveness (30 s probes; 2 misses = died): a dead engine — or an ended log stream with a dead engine — fails loudly and closes the run.

stop stops ALL nodes in parallel, every node runs the verifying command (polls up to 60 s that no labeled container remains) and failures are aggregated: one straggler cannot leave the other nodes un-stopped, and the error names the nodes that did not verify. The verified stop closes the run in the ledger. --no-wait opts out of the per-node verification.

report — the table of tested recipes: markdown (default) or an interactive HTML table via --serve/--open. It shows each recipe's run-ledger state (● running + uptime — live in the HTML, launch count) and supports archive curation: --archive <name> / --unarchive <name> flip the recipe's archived flag (comment-preserving), archived recipes leave the output, --archived shows them instead. stop waits until the containers are actually stopped and removed (polls up 60 s per node) — an immediate serve on the same nodes would otherwise race the dying containers' GPU release.

runlock — offline run-ledger management (the per-recipe .lock): list shows every recipe with an open (running) entry, close <recipe> closes the open run without any node verification. stop closes the ledger only after verifying every node of the cluster, so a node that died mid-run (or a killed serve) leaves the run open — sm121 report --serve cross-checks the ledger against a live fleet probe and renders such rows as orange "stale" (tooltip: sm121 runlock close <recipe>) instead of green "up" when all nodes of the run's cluster are reachable but no running container of the recipe exists; an unreachable node is inconclusive and keeps the green (the fix for a confirmed stale row is runlock close).

bench runs tool-eval-bench (from PATH) against a live serve and lands <UTC-date>_<label>.{md,json,log} in recipes/<lane>/reports/ — it does the scratch-cwd/absolute-json/log juggling, polls the API for readiness first, and prints the final score + perf rows. Defaults: seed 42, hardmode, perf, concurrency 1,2; served name/port from the recipe.

Depth suite: every campaign also appends a perf-only depth sweep (src/bench_depth.rs): depths 16384,65536,131072,262144 at c1 and c2, 2 runs per point, pp2048/tg128, same log file → the report's Throughput table gets the long-context rows; a depth failure only warns. Both perf paths pass --benchy-args=--enable-prefix-caching — **without that flag llama-benchy's defaults measure two serialized cold 67K prefills and the c2 depth row collapses to 13.7/5.7/2.1 t/s (a harness artifact, NOT the engine — case + evidence:specs/tech-architecture/sparkring-mimo-lanes.md)`. Never quote depth c2 numbers from a run older than 2026-09-29.

Run package (versioning per run): every bench run also lands two more files under the same stem — <stem>.recipe.toml (a byte-exact copy of the recipe file as it stood at bench time) and <stem>.manifest.json (recipe + config SHA-256, launcher git commit, tool-eval-bench version, and the served image's identity per node — docker id + sm121.checksum build label, captured over SSH at bench time). sm121 report reads the manifests back: each run shows a provenance chip (⚙ img@<buildsum12>·rcp@<sha12>, ⚠ when the recipe has since drifted from the copy) and the markdown table gets a Provenance column. Old runs without a package show — / no chip.

sm121 web — local axum + SSE dashboard: recipes panel window (the sm121 report --serve page mounted at /recipes — one implementation, its own SSE stream and /r/ viewer prefix-rewritten server-side), topology map (physical layer: 8 GX10 boxes on a 2×4 grid, 200G fabric switch + passive 10G switch + DAC direct links derived from config ib_ip/extra_entities, HA glyphs with live states, cluster frames — click a frame chip to activate the cluster), LED board (cluster-level health, non-blocking extra_entities as ⚙ badges), themes, serve/stop/power jobs with type-the-name power gate, live log-less jobs window fed by docker labels + SSE. UI checks: cargo test ui_playwright_checks (deno + Playwright; one-time browser: deno run -A src/web/pw-install.mjs). Design/progress: specs/tech-architecture/web-dashboard.md.

Cluster: exact name, prefix (4→4nodes-switch), or auto (smallest with NCCL + matching required_nodes).

serve --solo overrides cluster selection: finds the first free node across all clusters and runs one single-node container there (no --nnodes/--tp/ --master-addr flags — the engine defaults to single-node). --no-solo forces multi-node even if the recipe sets solo = true. A recipe can also set solo = true as the default.

serve --force skips the occupancy check during cluster selection — occupied nodes are kept in the available set (reachable is still required). Use to run a lightweight test recipe alongside a live serve, or to force onto a busy node. Dangerous for GPU recipes (OOM, port conflicts); the operator is responsible for coexistence.

serve --fit (optional value) scales a recipe to a node count and lets tensor_parallel_size track it (overrides the recipe's required_nodes and tensor_parallel_size; pipeline_parallel_size is left untouched). --fit=N scales to N nodes; bare --fit uses the whole selected cluster (TP = cluster node count). With --cluster auto, --fit=N auto-picks the smallest cluster with ≥N free nodes; bare --fit + --cluster auto picks the smallest cluster, so pass --cluster <name> --fit to scale onto a specific larger cluster. Incompatible with --solo.

serve --mirror (optional value) mounts a per-node local mirror of the HF

cache instead of the shared one at /root/.cache/huggingface. Bare --mirror uses ~/.cache/hf-models; --mirror <path> uses that exact path (on each node). The mirror keeps the HF hub layout (<mirror>/hub/models--org--model/...), so refs/main path refs, {{repo}} mounts, and composed-model symlinks work unchanged. Before launch each node checks its mirror and, if a required repo (main model + repo-based path refs + composed sources) is missing, copies it from the shared cache (cp -a, one-time per-node cost); a repo absent from BOTH locations aborts with a huggingface-cli download hint. Later serves read the model locally — no streaming weights over slow shared storage at load time.

serve --hf-cache <path> overrides the host-side HF cache dir for that serve (config hf_cache_dir, else ~/.cache/huggingface); ~ expands to the node's SSH-user home (quote it — the shell would expand it locally first). The mount target stays /root/.cache/huggingface, so model path refs and repo mounts resolve as usual. Mutually exclusive with --mirror (both pick the HF mount source — the error says so).

Declarative staging ([[staging]], spec specs/tech-architecture/staging-feature.md): a recipe declares [[staging]] kind = "hub-mirror" blocks and the operator runs plain serve — the launcher materializes the recipe's model data on every launch node before containers start, then mounts the node-local mirror instead of the shared HF cache (dst replaces the --hf-cache / config source; --mirror is rejected when a block declares a mirror). Verification is content-based: a per-root ledger (<dst>/hub/.sm121- staging-ledger.json) stores sampled content hashes — never mtime/size (the stale-NFS failure class) — so an unchanged repo costs zero copies, a same-size mutation in the mirror is detected and re-copied, and the second serve is verification-only. repos defaults to the recipe's model + repo path-refs + composed sources; skip leaves external/managed repos alone; min_free gates the whole pass per node (G/T/%). --stage=skip assumes staged (CI/known-good), --stage=force re-verifies everything. kind = "sparse-rows" (spec P2) packs each launch rank's slice of the Engram tables onto node-local NVMe (dst, ledger of sampled row hashes in the dst dir); rank/tp derive from the cluster and recipe — row ranges are never hardcoded. The runtime-visible dir must be bind-mounted by the recipe's extra_mounts (dst is host-side). The first staging serve packs (~15 min over the NAS source); combine with --no-wait if a cold weight load also pushes launch past the readiness budget. format = "sparse-copy" switches the output from the packed-bin shards to safetensors-pread layout — sparse node-local copies of the original shards containing exactly the rank's row ranges (holes are free) + engram-local.json row-range manifest: the aiden16/DSV41_ENGRAM_DIR contract. Sample-hash verification also adopts a manually staged tree without re-copying.

Cluster health is cluster-level (health, web dashboard): a cluster is UP only when all its nodes are reachable AND its blocking extra_entities report on. HA entities are probed via HA REST; a [mikrotik] device referenced by name is probed through the RouterOS API login (logged in = on; unreachable or rejected login = off, like a dead switch — health warns with the reason). One offline node — or a blocking switch off — takes the whole cluster down: containers on surviving nodes are not reported (serve/stop/jobs views all ignore them; power on stays available). extra_entities entries can opt out of gating with blocks = false (e.g. a fan that may be off while the cluster serves) — non-blocking entities are still toggled by power and shown as a soft badge, but never take the cluster down. Shared rule: cluster_state::verdict + home_assistant::extra_entities_states.

serve --clear deletes the host-side extra_mounts folders (alias/host — repo-based path refs are never touched) on the launch nodes with sudo rm -rf before containers start. The containers run as root, so the cache files there are usually root-owned. Use to reset JIT/kernel caches between serves. -d --clear prints the exact sudo rm -rf '...' command per node instead of running it.

⚠️ Cluster may be serving live traffic. A model (often the one serving this very agent) may be running on the cluster at any time. Before running stop, power off, or power switch, ALWAYS first run health and confirm with the user that the cluster is not serving before proceeding. Do NOT assume the cluster is idle — a stray stop/power off has killed running agents mid-task before. logs, health, doctor, and dry-runs (-d) are always safe.

Recipe format (recipes/<name>/recipe.toml)

Port convention: every recipe serves on the cluster default port 30000 — leave the port field unset (or 30000). When adapting a foreign recipe (forum/upstream), always drop or rewrite its port to the default — the foreign port (8000/8888/…) is not used on this cluster and breaks the bench/smoke defaults that read the recipe port.

Recipes and their image build contexts (docker-images/) live in the shared recipe repo — https://git.kilork.org/sm121/recipes (checkout at ../recipes) — the single working set; the launcher repo no longer carries a local recipes/ (removed 2026-10-13). The dev config points at the checkout via recipe_sources + docker_image_sources. They must move together — the served image must match the recipe revision. The add-on (and [[recipe_repos]]) sync the same repo: a curated credential-free config.toml (the add-on's base config — the repo may be shared with other users, so no ssh key / HA token / mikrotik password in it). It retired the old publish-bundle registry (flat sm121 publish bundles as the registry's storage; sm121recipes.onlinedb.org is gone — the add-on recipe-repo flow supersedes it).

engine = "vllm"                     # or "sglang"
docker_image = "img:tag"
model = "org/model"
parent_model = "org/base-model"     # optional; report category (medals/tree) ranks under the base model
archived = false                    # optional; true hides the recipe from `sm121 report` (set via `--archive`)
port = 30000                        # default — the only port convention on this cluster
required_nodes = 2                  # optional; also caps the launch set (N nodes, not all)
solo = false                       # optional; single-node on first free node, no dist flags
deploy = "docker"                   # optional; "docker" (default) or "tmux"
venv = "~/venvs/vllm025"            # optional; Python venv for tmux (activates before entrypoint)
venv_setup = [...]                 # optional; commands to create venv if it doesn't exist (tmux only)
venv_setup_script = "venv.sh"      # optional; script file (relative to recipe dir) to create venv (tmux only)
tensor_parallel_size = 2
pipeline_parallel_size = 1          # vLLM only, 0 = auto
distributed_backend = "mp"          # or "ray"
shm_size = "64gb"                   # optional, only set when non-empty
entrypoint = ["vllm", "serve"]      # optional, after image; used by both docker and tmux
docker_image_dir = "glm52-nf3-hybrid"  # optional; build dir under docker-images/ (default: recipe name)
before_serve = [{ command = "echo", on_failure = "abort" }]
after_stop = [{ script = "restore_dgx_services" }]   # runs after `sm121 stop`
args = ["--kv-cache-dtype", "fp8"]

[[extra_mounts]]                    # extra bind-mounts for `docker run -v`
alias = "jit_cache"                # resolves via [mount_aliases] in config.toml; wins over `host`
host = "~/.cache/jit"              # fallback default if alias absent; `~` expanded to ssh_user home
container = "/root/.cache/jit"     # container-side target (required)
read_only = false                 # optional, default false (adds :ro)
env = "XDG_CACHE_HOME"            # optional: injects -e <name>=<container path>

# Model path reference (mutually exclusive with alias/host): resolves at
# serve time, per node, from the HF cache — refs/main → latest snapshot, so
# the path survives `huggingface-cli download` updates with no fragile
# symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the
# recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes
# the resolved **container-side** HF cache path per-node — the host cache-dir
# prefix (e.g. `/home/<user>/.cache/huggingface`) is rewritten to the
# container mount target `/root/.cache/huggingface` (no bind-mount — the HF
# cache is already mounted there, so symlinks resolve correctly).
# Absent subfolder → serve aborts with a hint to `huggingface-cli download <repo>`.
[[extra_mounts]]
repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4"
subfolder = "mtp-draft"           # optional; omit = whole snapshot dir
container = "mtp_draft"

[mods]
commands = [                         # run *inside* container before engine
    "pip install flash-attn",
    { command = "./patch.sh", on_failure = "warn" },
]

[env]
KEY = "val"                         # overrides launcher defaults
PORT = { from = "port" }            # or: value resolves from a recipe field at
                                     # serve time (supported target: port) —
                                     # the value is declared once, can't drift

build_profile = "r28"             # optional; named pin set from docker-images/<dir>/profiles/<name>.env

[build.versions]                    # optional; overrides pins in docker-images/<dir>/versions.env
VLLM_COMMIT = "00907fc..."          # recipe wins over profile > image pin; folded into build checksum

Deployment target (deploy)

The deploy field selects how the recipe is launched on each node:

• docker (default): docker run with the recipe's entrypoint. Supports all features (mods, composed_model, image sync, extra_mounts).
• tmux: a detached tmux session running the engine directly (no Docker). The recipe's entrypoint is used when present (e.g. ["vllm", "serve"] → vllm serve); otherwise the launcher falls back to the engine's Python module path. Docker-only features are rejected: [mods], [composed_model], image sync, and --sync-image/--build-node/ --force-build CLI opts are ignored with a warning.

The serve command dispatches based on recipe.deploy:

engine = "vllm"
deploy = "tmux"                     # or "docker" (default)
entrypoint = ["vllm", "serve"]

Engine-specific args (args) are interpreted by the engine itself — the launcher doesn't validate them, so the same args list works for both vllm and sglang recipes. The entrypoint field is shared: both deployment paths use it, so a single recipe can be served via either docker or tmux without changes.

Venv creation (venv + venv_setup)

When deploy = "tmux", the venv field specifies a Python virtualenv path that is activated before the entrypoint (source <venv>/bin/activate && ...). The venv_setup field is an optional set of shell commands to create and initialize the venv if it does not already exist on a node. Before serve, the launcher checks test -f <venv>/bin/activate on each node; if absent, it runs the setup commands in order, then re-checks. A failure aborts the serve.

venv_setup accepts either an inline multi-line string (one command per line; blank lines and # comments are skipped) or a Vec<String> list:

deploy = "tmux"
venv = "~/venvs/vllm025"
venv_setup = """
    uv venv ~/venvs/vllm025 -p 3.12
    uv pip install vllm==0.25.1 --torch-backend=cu130
    uv pip install flashinfer-python==0.6.15.dev20260712 \
      flashinfer-cubin==0.6.15.dev20260712 \
      flashinfer-jit-cache==0.6.15.dev20260712 \
      --extra-index-url https://flashinfer.ai/whl/nightly/ \
      --extra-index-url https://flashinfer.ai/whl/nightly/cu130/
    hf download poolside/Laguna-S-2.1-NVFP4
    hf download poolside/Laguna-S-2.1-DFlash-NVFP4
"""

Or as a list:

venv_setup = [
    "uv venv ~/venvs/vllm025 -p 3.12",
    "uv pip install vllm==0.25.1 --torch-backend=cu130",
]

Alternatively, venv_setup_script references a shell script file (relative to the recipe directory) that is read and run on each node if the venv doesn't exist. Script file takes precedence over venv_setup if both are set:

venv_setup_script = "venv_setup.sh"

If the venv already exists on a node, the setup commands are skipped. venv_setup and venv_setup_script are ignored in Docker mode (the container has its own Python).

Env vars (auto-set)

Cluster-driven: NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX, GLOO_/TP_SOCKET_IFNAME, MN_IF_NAME, UCX_NET_DEVICES, OMPI_MCA_btl_tcp_if_include Defaults: OMP_NUM_THREADS=4, NCCL_IGNORE_CPU_AFFINITY=1, NCCL_NET=IB, NCCL_DEBUG=WARN Engine IP: VLLM_HOST_IP / SGLANG_WORKER_IP Head: VLLM_LOGGING_COLOR=1, TERM=xterm-256color, -p <port>:<port> Ray: RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS Recipe [env] wins (last wins in -e).

Mods

Mods run inside the container before the engine process via entrypoint wrapping. When [mods] is present, the recipe must also set entrypoint (the known process to exec at the end). Same on_failure semantics as before_serve (default: abort). On failure the container exits; configure docker run --restart externally to restart on crash.

Implementation: --entrypoint sh image -c 'mods && exec <original entrypoint>'

Built-in scripts (baked into the binary)

Host scripts under host-scripts/ are embedded at compile time (include_str! in src/builtin_scripts.rs) and can be referenced by alias from any before_serve / after_stop list. The launcher uploads

• runs them via SSH per node — no scp from the operator's machine.

Entry shape (untagged enum, three forms):

before_serve = [
  "sudo nvidia-smi -lgc",                       # bare string = raw shell command
  { command = "echo hi", on_failure = "warn" },  # raw command + on_failure
  { script = "stop_dgx_services" },              # built-in alias
  { script = "stop_dgx_services", on_failure = "warn" },
]

A bare string is always a raw command (today's behavior, unchanged); a built-in is referenced only via { script = "<alias>" }. Unknown aliases error at resolution time with the list of known ones.

Built-in aliases:

Alias	File	Purpose
stop_dgx_services	host-scripts/stop-dgx-services.sh	Stop non-essential services on a DGX node (frees unified memory)
restore_dgx_services	host-scripts/restore-dgx-services.sh	Restore services stopped by stop_dgx_services

To add a built-in: drop the .sh into host-scripts/, add an include_str! + BuiltinScript entry in src/builtin_scripts.rs, document the alias here. Scripts stay diffable / shellcheck-able as real files.

Flags auto-generated

• vLLM: --tensor-parallel-size, --pipeline-parallel-size, --nnodes, --node-rank, --master-addr, --master-port, --host, --port, --headless (workers)
• SGLang: --tp, --nnodes, --node-rank, --dist-init-addr, --host, --port
• Both: --distributed-executor-backend (ray only, mp is auto-detected)

Docker image builds

All node operations go through sm121 commands — serve (with --build-node), stop, logs, health, bench. Do NOT scp + bash build.sh or docker run/stop over raw SSH: the launcher owns the upload, the tmux build session, the checksum label, the label-based container management, and the stop-wait verification. Raw SSH/docker is allowed only for one-off diagnostics the launcher cannot express (hung-worker docker rm -f by name, log forensics) — and those get noted in the shift history.

Image distribution (sync_image_to_nodes) is node-to-node, never through the Mac: the source node (newest build, or --build-node/--sync-image-source) runs docker save | ssh <peer> docker load per peer, with the inner hop host-key verified against the pinned fingerprint, and routed over the peer's ib_ip (200G fabric) when it has one — with a logged fallback to ssh_ip (10G mgmt) if the fabric hop fails, so a stale node-to-node authorized_keys warns instead of blocking the serve.

A recipe's docker_image is built on demand from docker-images/<docker_image_dir>/ (default <dir> = recipe name). sm121 serve --build-node <n> (or the standalone sm121 build) uploads the dir to a node, runs bash build.sh in a tmux session, and stamps a sm121.checksum label; later serves skip the build when the label matches a recursive SHA-256 of the dir.

Pinned builds (docker-images/<dir>/versions.env): every build input (CUDA base, torch/NCCL/flashinfer/vLLM fork+commit, GPU arch, …) is a pinned KEY=VAL. build.sh is a generic driver — it forwards every pin in versions.env as a --build-arg to the Dockerfile (no hardcoded variable list). Add a pin to versions.env and it is picked up automatically. locks/ holds --require-hashes lockfiles + deps.lock (the staged dep-subtree fingerprints fetch-deps.sh verifies every build against — dep commits, vendored-cutlass gitlink SHAs, triton_kernels content hash; regenerate via --regen-lock / --build-arg REGEN_LOCK=1 after a pin cut); checksums/ holds verified out-of-band downloads (e.g. tiktoken encodings). See docker-images/glm52-nf3-hybrid/ for the reference cu130 / aarch64 / sm_121a stack; docker-images/solar-open2-nvfp4/ is the same stack minus the b12x suite (stock vLLM serve).

Recipe pin overrides (build_profile + [build.versions]): a recipe may override any pin without editing the image dir. A build_profile = "<name>" references docker-images/<dir>/profiles/<name>.env (same KEY=VAL format) — the way to ship a full pin set by name (an engine generation cut across all forks: vLLM + b12x + LMCache together; see docker-images/jovian-judgement/profiles/r28.env). Precedence: image versions.env < profile < recipe [build.versions] (one-off experiments). The merged file is written to the node and folded into the checksum, so any pin change triggers a rebuild. Keys must match versions.env names the Dockerfile declares as ARG.

When cutting a new profile, bump every fork of the stack together (the source lock is cut as one set — see the profile header comment); a half-cut (vLLM advanced, b12x/LMCache left behind) silently skips loader/perf fixes.

build_profile = "r28"             # full r28 lock: vLLM + b12x + LMCache
# or, for a one-off experiment on top of a profile:
[build.versions]
TORCH_VERSION = "2.11.0"

Structured output under speculative decoding (glm52)

The glm52 image patches vLLM's structured-output reasoning-boundary logic (upstream PRs #44297 + #44993) to fix guided decoding (response_format) under MTP + reasoning parser. Without the patch, when the reasoning parser emits the reasoning-end marker (</think>) under speculative decoding, the grammar FSM fails to advance, so the content channel is unconstrained — models emit prose or markdown-fence prefixes before JSON (tool-eval-bench Structured Output: 50% instead of 100%). The patch is applied at image build time by docker-images/glm52/patch-structured-output.py, which modifies three files in the installed vLLM package:

1. v1/structured_output/request.py — adds reasoning_end_token_index field
2. v1/structured_output/__init__.py — should_advance() accepts new_token_ids (authoritative step delta), returns True for all constraint types at the boundary, records the index; grammar_bitmask() detects mid-window reasoning end and constrains subsequent drafts; adds _find_reasoning_end_index() and trim_reasoning_for_advance()
3. v1/core/sched/scheduler.py — update_from_output() passes new_token_ids to should_advance() and trims reasoning content before accept_tokens()

Fixes: vLLM #34650, #43388, #48228. The b12x fork predates the upstream merge (Jul 23, 2026); this patch is a forward-port until the fork rebases.

Composed model directories

A recipe may declare [composed_model] to combine multiple HF repos into a single model directory at serve time — e.g. a base checkpoint + an expert overlay — without mutating the HF cache. The launcher builds a /tmp dir per-node with symlinks into the HF cache (container-side absolute paths) + generated files, bind-mounts it at mount, and rewrites model to that path.

Symlinks point to /root/.cache/huggingface/... (the container-side HF cache mount), not the host-side path. They are dangling on the host but resolve correctly inside the container where the HF cache is mounted at that exact path — avoiding the host↔container mount-root mismatch. Later sources win (overwrite earlier symlinks), so order matters for overlays.

[composed_model]
mount = "/root/model"               # container-side target; `model` is rewritten to this

[[composed_model.sources]]          # symlinked into the composed dir (later wins)
repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ"      # base: config, weights, tokenizer
# (no glob → whole hierarchy symlinked recursively, incl. subdirs)

[[composed_model.sources]]
repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ"  # overlay: Lloyd NF4 experts
glob = "L*.safetensors"                             # only expert weight files (glob, may include / for subdirs)
exclude = ["L5.safetensors"]                       # optional: skip these paths (keep earlier source's version)

[[composed_model.files]]            # generated real files (not symlinks)
name = "nf4_layers.json"
content = '{"layers":[3,4,...,77]}'

[[composed_model.local_files]]      # hardlinked from a per-node local host path
host = "/var/tmp/engram-local/DeepSeek-V4.1-Flash/model-00047-of-00048.safetensors"
name = "model-00047-of-00048.safetensors"  # optional; defaults to the host basename

local_files hardlinks real files from node-local storage (NVMe) into the composed dir — run AFTER the repo symlink pass, so a local file replaces the repo's symlink at the same name (local wins). Use to serve a shard from local disk while the rest of the model stays on the shared cache (e.g. Engram table shards from a sparse per-rank NVMe copy instead of the NAS). Hardlinked, NOT copied — a sparse source stays sparse and costs zero extra disk; the host path must be on the same filesystem as /tmp and exist on every launch node (missing → serve aborts).

With no glob, the entire snapshot hierarchy is symlinked recursively (every file in every subdir, as individual symlinks with real parent dirs) — so a later source can overlay individual files at any path. With a glob, only matching entries are symlinked (one level; / allowed for a specific subdir, e.g. grids/*.json) and the directory structure is preserved. exclude (optional list of path patterns) skips matched files so an earlier source's version survives — e.g. exclude = ["L5.safetensors"] keeps the base checkpoint's L5.safetensors instead of the overlay's. Allowed characters (glob + exclude): alphanumeric, * ? . _ - /.

The composed dir is cleaned up automatically on sm121 stop.

Serving notes (DeepSeek-V4-Flash / DSpark on GB10)

Operational findings from the SvangenStudios agent-serving benchmark (v0.1.1, measured on this exact stack: DS4-Flash-0731, DSpark k=5, 2× DGX Spark, TP=2). See specs/tech-architecture/dgx-spark-agent-serving-findings.md for full detail.

repetition_penalty crashes the DSpark path — an illegal-memory-access that kills the serving container. Any gateway/proxy in front of an sm121-served DSpark model must filter repetition_penalty from requests before forwarding. presence_penalty and frequency_penalty are verified safe. This is a DSpark-stack bug, not model behavior; the benchmark explicitly did not repeat the destructive test.

UMA memory gate (vLLM #48140) — on GB10 (unified memory), vLLM's startup KV-cache budget check effectively reads Linux MemFree, so ~20 GB of reclaimable page cache (created by reading the weights) is booked as unavailable. The effect: large --max-model-len profiles (128K–262K+) may deterministically refuse to boot even when the memory should fit, while 32K–64K boots normally. Two candidate mitigations:

• run echo 3 > /proc/sys/vm/drop_caches (or the built-in drop_caches mod from eugr's repo) as a before_serve hook before the engine starts;
• a local UMA-aware patch of gpu_worker.py to use MemAvailable instead of MemFree. The drop_caches workaround remains unverified on this deployment — test before relying on it for a large-context recipe.

--gpu-memory-utilization is a KV lever, not a safety lever on GB10: weights take ~77.7 GiB of a ~119.6 GiB node, so the GMU ceiling directly squeezes the KV-cache budget. The benchmark found 0.80 boots but dies under traffic; 0.78 is the safe ceiling. Do not raise GMU to "be safe" — that reduces usable KV capacity. The deepseek-v4-flash-eugr / -src / -eugr-agent recipes all ship 0.78.

Three engines — an LLM server is three nearly-independent resources: prefill (compute-bound, serializes; one 32K prefill saturates the box), decode (bandwidth-bound, scales to ~N=4 concurrency), and admission (scheduler-bound, blocked during long prefill — a short request submitted mid-prefill waits until prefill completes, ~221× normal TTFT). --max-num-batched-tokens governs how often decode gets a scheduling slot between prefill chunks; 2048 gives 2.9× the decode share of 8192 under a concurrent prefill, for 7.7% prefill throughput. Use the deepseek-v4-flash-eugr-agent recipe (2048) for interactive agent traffic; use deepseek-v4-flash-eugr (8192) for exclusive long-context batch jobs.

Qwen3.8-27B recipes (measured)

qwen38-* recipes serve the Qwen3.8-27B family on a single DGX Spark (dgx1, qwen38 cluster). The tested winner is qwen38-27b-nvfp4-unsloth (vLLM v0.27.1 + MTP k=5 on unsloth NVFP4): ~21.7 tok/s decode, 100/100 agentic tool-eval, and the MTP head is in-checkpoint. SGLang (DSpark and the DFlash2 fork) measured ~2.3× slower on this box despite the forum's 52-87 tok/s claims (an ndec.py cached-context artifact). See specs/tech-architecture/qwen38-recipe-state.md for the full measured matrix, why each decision was made, and how to re-benchmark.

GLM-5.3-Flash NVFP4 re-cut (measured)

local-inference-lab/GLM-5.3-Flash-NVFP4 was re-cut in place on 2026-09-16 (the old ~186 GB upload is gone): 199.4 GB = 36 expert shards + 4 nonexpert + 4 MTP shards. Structure is identical to the Spark checkpoint (verified from model.safetensors.index.json): model.language_model.layers.* with SPLIT GDN naming, a model.visual.* tower (skip via --language-model-only), MTP head embedded at layer 45 — so the JJ-fork engine requirements are the same as the Spark lanes (r35 + VLLM_DISABLED_KERNELS; r38 pool-deadlock on 2x still applies). Lanes: glm53-flash-nvfp4-lab (2x, same dir as the old pre-re-cut lane — it had no reports/lock) and glm53-flash-nvfp4-lab-4x (4x).

Measured 2026-09-20 (r35, harness dev65, seed 42, hardmode):

• 2x: 164/176 (93/100) — the family best (old 186 GB cut: 89/100; Spark r35 2x: 92). The operator's "they improved it" hunch was right. The re-cut is heavier: ~93 GiB weights/rank, so auto-sizing leaves only ~2.4 GiB KV at GMU 0.88. With --max-num-seqs 4 + --max-num-batched-tokens 2048 (frees the profile-run peak) an explicit 10 GiB KV pin fits: pool 1,406,229 tokens = 2.68× a 524K request → the 2x lane runs 512K context (bench 2026-09-20_090824_512k-kv10g: 160/176, 91/100; perf ≈ 256K config, pp ~1,500, tg 33/41 c1/c2). Host margin after boot: ~2-3 GiB available/node — 12 GiB is the realistic pin ceiling.
• 4x: 159/176 (90/100), decode 54-68 t/s (2× the 2x), KV 49.3 GiB → 7.35M-token pool @524K. The 4x loses 5 pts to the 2x (TC-43/45/49/68/80/88 flakes); TC-51 fails on both (known harness recurring).
• Weights live on all 4 nodes' local NVMe (hf-local, snapshot 175ae8ce); dgx1 is the tightest host (~55 GB free after the model).

SparkRing MiMo lanes (MOPD — operator's fast line, 2026-09-29)

MOPD only from now on (RL lanes retired — RL keeps training artifacts). Three lanes on the installer image ghcr.io/fujitsupolycom/sparkring@sha256:c977a2d2… + their toolchain.py serve entrypoint — full contract, deviations and diagnosis are in specs/tech-architecture/sparkring-mimo-lanes.md:

• mimo26-flash-mopd-4x-sparkring (TP4, dgx58-switch) — benched honest 165/184, c2 ≥ c1 at every depth up to 256K.
• mimo26-flash-mopd-2x-sparkring (TP2, 2nodes-dac) — their qualified direct-pair-2 shape, B12X_ROCE_PAIR_PATHS=2, cluster block routes f1 (DAC) with GID 3.
• mimo26-pro-mopd-8x-sparkring (TP8, 8nodes-switch) — Pro-MOPD adea8e2c5373 (573.5 GB NAS), no upstream profile → our tp4 contract adapted; needs fleet + --sync-image.

Dry-run any of them with sm121 -d serve --recipe R --cluster C before touching nodes; serve timing for 2x/8x is operator-coordinated (prod cluster must be free).

Config fields

Config resolution: --config <path> > ./config.toml (cwd) > the global config (~/.config/sm121/config.toml on Linux, ~/Library/Application Support/sm121/config.toml on macOS — see sm121 config path). sm121 config init-global creates a template if missing. The global config exists so sm121 runs from any cwd: it holds the two source lists below (and may omit ssh_user/ssh_key_path — a cluster_sources file provides them when the main config leaves them empty); it may carry no [[clusters]] of its own — clusters may come entirely from cluster_sources.

ssh_user = "u"                   # may be empty — a cluster source can provide it
ssh_key_path = "/path"            # PathBuf; same

recipe_sources = ["/path/to/recipes"]       # extra recipe dirs (each `<name>/recipe.toml`); searched after ./recipes
cluster_sources = ["/path/clusters.toml"]   # extra cluster def files: `ssh_user`/`ssh_key_path`,
                                            # `[[clusters]]`, `[nccl.<name>]`, `[before_serve.<name>]`;
                                            # merged by name, this file wins

[[clusters]]
name = "c"
master_ib_ip = "10.0.0.1"            # optional; omitted → rank-0 node's ib_ip (else ssh_ip)
nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }]
extra_entities = [                  # cluster-level HA entities
    "switch.network",                # bare string = blocking (off → cluster down)
    { entity = "switch.fan", blocks = false },  # non-blocking: toggled + shown, never gates up/down
]

[nccl."c"]
ib_hca = "..."
socket_ifname = "..."
ib_gid_index = 3        # optional: omit to auto-detect (RoCE-v2 IPv4 GID, per node)
ib_disable = 0

[home_assistant]
url = "..."
token = "..."
shutdown_delay_seconds = 30

[exec_aliases]
name = "command"

[mount_aliases]                     # host paths referenced by name from recipe [[extra_mounts]]
jit_cache = "/mnt/nas/ai/jit"      # alias wins over recipe `host`; lets operators repoint mounts site-wide
hf_cache = "/mnt/nas/ai/huggingface"

[before_serve."*"]                  # global: runs on every cluster; "*" is a literal key, not a glob
commands = ["sudo nvidia-smi -lgc 0,2000"]

[before_serve."c"]                  # per-cluster: runs after the global list, can supplement/override
commands = ["cmd", { command = "cmd", on_failure = "warn" }]

Image registry push ([registry])

[registry]
base = "ghcr.io/kilork"   # push refs: {base}/{image-name}:{version} + :latest

Versioned tags, never :local. The version is the image's build checksum (recursive SHA-256 of the image dir — the same value stamped as the sm121.checksum label), so identical content always maps to the identical tag: re-publishing "the same thing" cannot create a duplicate, changed content is a new version tag. Each successful local build (sm121 build / serve --build-node) then pushes {base}/{image-name}:<checksum:12> + :latest from the build node. sm121 publish --push publishes without a build: it finds the image on --node (or probes all cluster nodes), takes the version from the sm121.checksum label (image ID fallback for unlabeled prebuilt images), and pushes from that node.

Duplicate guard: a publication ledger (published.json next to the global config) records {repo → {version, image digest, time}}; a re-push with an unchanged image digest is skipped (with a note) unless --force (build/serve/publish). The push runs in a tmux session on the node (sm121-push-<label>, log /tmp/<label>-push.log) with the same 30 s poll / 90 min cap as the build — a 30 GB push must not ride on one SSH session. The launcher stores no registry credentials: the node must be logged in once (ssh <node> && docker login ghcr.io); a failed push fails with the log tail + a docker login hint. --no-push (build + serve) skips the push even when [registry] is set; without the section, builds behave as before.

HA add-on + recipe repos

Full design: specs/tech-architecture/ha-addon-recipe-repos.md.

Recipe repos — [[recipe_repos]] in config (ordered list; first entry wins on a recipe-name collision). Each entry is a git repo holding recipes/ (+ docker-images/ build contexts); the launcher clones/pulls it into ~/.cache/sm121/recipe-repos/<sha256(url)> at every command start (best-effort — an unreachable repo warns and is skipped, never blocks a serve of a recipe from a reachable repo; skipped in dry-run), then appends its recipes/ dir to recipe_sources and its root to docker_image_sources. Archive state is a separate JSON registry (repos.json next to the global config), not a config flag:

sm121 repos                  # list: cloned? archived? last-pull ts? recipe count?
sm121 repos sync [--force]   # clone/pull now (--force re-clones from scratch)
                             # `sm121 web` also re-syncs in the background
                             # (--repo-sync-interval N, default 10, 0 disables),
                             # so a running dashboard never goes stale
sm121 archive <url>          # skip the repo in sync (state in repos.json)
sm121 unarchive <url>
sm121 promote <recipe> --from <url> --to <url> [--apply]
                             # cross-repo git mv: target add+commit+push FIRST,
                             # then source remove+commit+push (a failure
                             # duplicates, never deletes; collision refused;
                             # default/-d prints the exact git commands)

Image source — serve/build gain --image-source {repo,registry} (default repo = on-demand build from docker-images/). registry pulls the prebuilt {base}/{name}:{checksum12} from the [registry] base on every launch node, tags it as the recipe's docker_image, and verifies the sm121.checksum label — a missing tag names the tag + hints to build/push first (or fall back to --image-source repo). Mutually exclusive with --build-node.

The add-on image — the HA add-on store lives in the github.com/kilork/sm121-ha-integration repo (repository.json + sm121/ add-on dir + .github/workflows/, same pattern as kilork/cups-addon). Multi-stage build (sm121/Dockerfile): a rust:alpine stage compiles sm121 from the launcher repo (CI checks it out as launcher-src/ in the build context) with a trivial cargo build --release (pinned rust:1.98.1-alpine3.22, native arch per runner, dynamic musl); the alpine:3.22 final image ships it + git/jq/curl + the entrypoint (~25 MB, no toolchain, no compile-on-boot — a 4 GB Pi must not OOM on a release build, so the Pi is not the build host). The CI matrix builds each arch natively (aarch64 on an arm runner, amd64 on x86) and composes the multi-arch manifest. GitHub Actions builds + pushes ghcr.io/kilork/sm121-addon (aarch64 + amd64 + :latest/:1.1.0 manifest) on push to main via OIDC — no docker login anywhere (the GHCR package is created private: set it public once so HA pulls anonymously). A launcher change does not trigger the add-on CI (cross-repo) — re-run it (workflow_dispatch) to ship a new binary. Local dev: make addon-image (rsyncs this repo into the add-on store repo's launcher-src/ + docker build) + scripts/smoke-addon.sh (entrypoint + file:// recipe repo + curl /api/state). The entrypoint writes the effective config from the add-on options (env), drops the SSH key (base64 option), sm121 repos sync, then sm121 web --port 3100 --bind 0.0.0.0 (a bearer token is required; sm121_insecure: true runs it unauthenticated) with a background re-sync loop (SM121_REPO_RESYNC_MINUTES). Its custom_components/ custom-integration prototype is superseded (entities are a phase-2 nicety over the add-on's REST API). The cluster definitions (private IPs — never in the public recipe repo) are supplied as the sm121_config_extra add-on option (raw TOML, appended to the generated config): [[clusters]]/nccl + [home_assistant] + [mikrotik] live in HA's own option storage — private, on the Pi, no second repo, no PAT. If that TOML carries [home_assistant]/[registry], the matching flat options must stay empty (duplicate TOML tables fail the parse).

Single-launcher rule — the per-recipe run ledger (.lock in the recipe repo) is the serialization point: a second launcher that serves the same recipe sees the open run and refuses. Run one launcher as the source of truth (the add-on); the Mac launcher is for dev (dry-runs, building, promote, bench campaigns). The live shared recipe repo is git.kilork.org/sm121/recipes (restructured 2026-10-11 to the spec layout; the publish-bundle registry retired with it) — extra repos (e.g. a playground) are seeded with scripts/seed-recipe-repos.sh <name> [git-url] (recipes/ + docker-images/ only; the config.toml stays operator-curated and credential-free).

Power

• off: SSH shutdown → wait → HA turn_off per-node entities + cluster extras
• on: HA turn_on with ensure-powered: a node the caller believes down (web: last fleet snapshot; CLI: 1.5 s TCP probe) whose plug reads on gets a 30 s plug power cycle (a shorter cut leaves the PSU rail energized — the machine stays in soft-off). Each node's outcome becomes the job message (web) / log lines (CLI)
• jobs: the web registry persists to web-jobs.json next to the global config; jobs that were running at shutdown are marked failed with a sm121 runlock list hint
• switch --from <src> --to <tgt>: disable nodes/extras unique to from, enable all in to

RouterOS devices ([mikrotik] config)

A cluster entity that names a [mikrotik] device is not an HA switch — it is reached over the RouterOS API instead (src/mikrotik.rs, raw TCP 8728, variable-length word framing):

[mikrotik]
[[mikrotik.devices]]
name = "switch.dgx1"            # matched against node/entity names exactly
address = "192.168.88.1"
username = "admin"
password = "secret"
api_port = 8728                  # optional, default 8728
on_entity = "switch.lab_relay"   # optional: HA relay that powers it back on

• power off → login + /system/shutdown per matched device (after the HA off phase), then the on_entity outlet relay is switched off via HA — the API shutdown alone leaves the device powered (fans keep spinning, it can boot back up; an already-energized relay makes power on a no-op). A failed API call or relay warns, never blocks the rest of the flow.
• power on → on_entity relay via HA if set, otherwise a “power on manually” reminder (the API cannot wake an off router).
• health / web probe / serve auto power-on: mikrotik names are dispatched to the RouterOS API — ping for state, on_entity relay for turn_on (manual reminder when unset); never HA.
• Without a [mikrotik] section everything behaves exactly as before.

NCCL_IB_GID_INDEX auto-detect

The RoCE-v2 IPv4 GID index re-numbers across reboots on DGX Spark, so a pinned gid can silently go stale. When a cluster's [nccl] block omits ib_gid_index, serve resolves it at launch per node by SSHing in and scanning the sysfs IB GID table for the first RoCE-v2 / IPv4-mapped GID (mirrors the Aiden serving stack: forum 372268/708). The resolved value is injected into the docker -e / tmux export, so both deploy targets benefit. Set ib_gid_index = <n> explicitly to opt out (pin the value). Dry-run prints the detect command per node and renders NCCL_IB_GID_INDEX=auto.

i18n

fluent-i18n crate, auto-detect via set_locale(None). FTL in locales/.

Constraints

• Files ≤500 lines (enforced by make check-lines); extract submodules when a file approaches the limit
• Use rg (ripgrep) for all code search — plain grep -rn across this tree aborts/hangs and is unusable; rg is fast and skips .git/gitignored paths
• #![deny(clippy::wildcard_imports)] — no use foo::*;
• Clippy warnings: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do not add #[allow(...)] attributes. If the only viable fix is an allow attribute, consult the user for guidance.
• tool-eval-bench runs happen in a separate working dir, never in this repo — its output (e.g. data/benchmarks.sqlite) is not for this repo; a stray run here pollutes the working tree. Campaign reports ARE committed, next to the lane they benchmark: recipes/<name>/reports/<UTC-stamp>_<label>.md (copied from the tool's runs dir) plus the raw JSON — pass --json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the .log); same base name for both. Quality-gate semantics are version-sensitive (--hardmode = 97 scenarios on v2.8.0, was 92 on 2.7.1.dev25, 88 on 2.7.0) — see the bench header in specs/tech-architecture/glm53-recipe-state.md; the tool is installed from SeraphimSerapis/tool-eval-bench — current pin: v2.8.0 (2026-10-14); re-baseline the anchor lane before comparing new lanes.
• Harness version — do not uv tool upgrade casually. tool-eval-bench is a uv tool (~/.local/share/uv/tools/tool-eval-bench) installed from SeraphimSerapis/tool-eval-bench.git; the receipt is pinned at tag v2.8.0 (2026-10-14; 97 scenarios; perf presets, session ids, shareable results). Prior pins: 2.7.1.dev7 (92 scen), 2.7.1.dev25 (the 8x 95/100 anchor, 2026-10-12), v2.7.0 544afb576b3 (2026-09-21, first stable of dev65). Reports stamp the harness version in the header — A/B runs must share a harness version; pools differ across versions (88→92→97), so cross-version deltas are not comparable. To move the pin: uv tool install --reinstall --force "tool-eval-bench[perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<tag-or-rev>" then re-baseline the anchor lane. Reasoning effort: engine defaults (recipe templates) — explicit low-effort runs are separate labeled datapoints, not comparable to the standard ones.
• Bulk data moves node-to-node, never through the Mac — the Mac→node link is ~32 MB/s (a 200 GB transfer via the Mac takes hours); node-to-node over the 200G fabric does ~547 MB/s per stream. Node-to-node SSH works (the shared key is on every node; ssh kilork@<200.x-ib-ip> from a node; dgx4 has two RoCE IPs — 200.19 and 200.20 — use the ib_ip from config). Pattern for a large HF download: one hf download on the roomiest node (dgx4, 1.9 TB), then each target PULLS: nohup bash -c "ssh <node> 'tar -hcf - -C <modeldir> snapshots trees refs' | tar xf - -C <modeldir>" & (-h dereferences so targets don't need the hub blob layout). hf download on a node = ~/.local/bin/hf (dgx2) or uv tool install huggingface_hub (others).
• huggingface_hub ≥0.34 (the hf CLI) moved blobs to the hub level — hub/blobs/<xx>/<sha> shared across models, per-model dirs are symlink trees (snapshot → per-model blob → hub blob). du/find -type f on a model dir only sees the small real files (~19 GB of the 199 GB) — judge completeness by df deltas or hub-level du. The 09-20 re-cut download migrated dgx4's cache layout this way (the Spark model there is hub-blob-backed now).
• pkill -f over ssh self-matches — the pattern also matches the remote shell's own -c command line, so pkill -f "hf download" kills the script itself mid-run (and a nohup … hf download … line in the same command makes the later pkill match again). Use the [b]racket trick for the pattern and run kill and launch in separate ssh calls.