AGENTS.md — sm121

Rust CLI (edition 2024) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (russh). Docker default, tmux alternative.

Dev quick-ref

Action	Command
Build	`cargo build`
All gates	`make check` (fmt → clippy → test → ≤500 lines)
Dry-run	`cargo run -- -d serve --recipe deepseek-v4-flash`
Tests	`cargo test` (65, all passing)
l10n	`locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent FTL files

See DEVELOPMENT.md for architecture, module layout, and dev workflow.

Commands

serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys/config/web

sm121 web — local axum + SSE dashboard: LED board (cluster-level health, non-blocking extra_entities as ⚙ badges), themes, serve/stop/power jobs with type-the-name power gate, live log-less jobs window fed by docker labels + SSE. UI checks: cargo test ui_playwright_checks (deno + Playwright; one-time browser: deno run -A src/web/pw-install.mjs). Design/progress: specs/tech-architecture/web-dashboard.md.

Cluster: exact name, prefix (4→4nodes-switch), or auto (smallest with NCCL + matching required_nodes).

serve --solo overrides cluster selection: finds the first free node across all clusters and runs one single-node container there (no --nnodes/--tp/ --master-addr flags — the engine defaults to single-node). --no-solo forces multi-node even if the recipe sets solo = true. A recipe can also set solo = true as the default.

serve --force skips the occupancy check during cluster selection — occupied nodes are kept in the available set (reachable is still required). Use to run a lightweight test recipe alongside a live serve, or to force onto a busy node. Dangerous for GPU recipes (OOM, port conflicts); the operator is responsible for coexistence.

serve --fit (optional value) scales a recipe to a node count and lets tensor_parallel_size track it (overrides the recipe's required_nodes and tensor_parallel_size; pipeline_parallel_size is left untouched). --fit=N scales to N nodes; bare --fit uses the whole selected cluster (TP = cluster node count). With --cluster auto, --fit=N auto-picks the smallest cluster with ≥N free nodes; bare --fit + --cluster auto picks the smallest cluster, so pass --cluster <name> --fit to scale onto a specific larger cluster. Incompatible with --solo.

serve --mirror (optional value) mounts a per-node local mirror of the HF

cache instead of the shared one at /root/.cache/huggingface. Bare --mirror uses ~/.cache/hf-models; --mirror <path> uses that exact path (on each node). The mirror keeps the HF hub layout (<mirror>/hub/models--org--model/...), so refs/main path refs, {{repo}} mounts, and composed-model symlinks work unchanged. Before launch each node checks its mirror and, if a required repo (main model + repo-based path refs + composed sources) is missing, copies it from the shared cache (cp -a, one-time per-node cost); a repo absent from BOTH locations aborts with a huggingface-cli download hint. Later serves read the model locally — no streaming weights over slow shared storage at load time.

Cluster health is cluster-level (health, web dashboard): a cluster is UP only when all its nodes are reachable AND its blocking extra_entities (HA switches) report on. One offline node — or a blocking switch off — takes the whole cluster down: containers on surviving nodes are not reported (serve/stop/jobs views all ignore them; power on stays available). extra_entities entries can opt out of gating with blocks = false (e.g. a fan that may be off while the cluster serves) — non-blocking entities are still toggled by power and shown as a soft badge, but never take the cluster down. Shared rule: cluster_state::verdict + home_assistant::extra_entities_states.

serve --clear deletes the host-side extra_mounts folders (alias/host — repo-based path refs are never touched) on the launch nodes with sudo rm -rf before containers start. The containers run as root, so the cache files there are usually root-owned. Use to reset JIT/kernel caches between serves. -d --clear prints the exact sudo rm -rf '...' command per node instead of running it.

⚠️ Cluster may be serving live traffic. A model (often the one serving this very agent) may be running on the cluster at any time. Before running stop, power off, or power switch, ALWAYS first run health and confirm with the user that the cluster is not serving before proceeding. Do NOT assume the cluster is idle — a stray stop/power off has killed running agents mid-task before. logs, health, doctor, and dry-runs (-d) are always safe.

Recipe format (`recipes/<name>/recipe.toml`)

engine = "vllm"                     # or "sglang"
docker_image = "img:tag"
model = "org/model"
port = 30000                        # default
required_nodes = 2                  # optional; also caps the launch set (N nodes, not all)
solo = false                       # optional; single-node on first free node, no dist flags
deploy = "docker"                   # optional; "docker" (default) or "tmux"
venv = "~/venvs/vllm025"            # optional; Python venv for tmux (activates before entrypoint)
venv_setup = [...]                 # optional; commands to create venv if it doesn't exist (tmux only)
venv_setup_script = "venv.sh"      # optional; script file (relative to recipe dir) to create venv (tmux only)
tensor_parallel_size = 2
pipeline_parallel_size = 1          # vLLM only, 0 = auto
distributed_backend = "mp"          # or "ray"
shm_size = "64gb"                   # optional, only set when non-empty
entrypoint = ["vllm", "serve"]      # optional, after image; used by both docker and tmux
docker_image_dir = "glm52-nf3-hybrid"  # optional; build dir under docker-images/ (default: recipe name)
before_serve = [{ command = "echo", on_failure = "abort" }]
after_stop = [{ script = "restore_dgx_services" }]   # runs after `sm121 stop`
args = ["--kv-cache-dtype", "fp8"]

[[extra_mounts]]                    # extra bind-mounts for `docker run -v`
alias = "jit_cache"                # resolves via [mount_aliases] in config.toml; wins over `host`
host = "~/.cache/jit"              # fallback default if alias absent; `~` expanded to ssh_user home
container = "/root/.cache/jit"     # container-side target (required)
read_only = false                 # optional, default false (adds :ro)
env = "XDG_CACHE_HOME"            # optional: injects -e <name>=<container path>

# Model path reference (mutually exclusive with alias/host): resolves at
# serve time, per node, from the HF cache — refs/main → latest snapshot, so
# the path survives `huggingface-cli download` updates with no fragile
# symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the
# recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes
# the resolved **container-side** HF cache path per-node — the host cache-dir
# prefix (e.g. `/home/<user>/.cache/huggingface`) is rewritten to the
# container mount target `/root/.cache/huggingface` (no bind-mount — the HF
# cache is already mounted there, so symlinks resolve correctly).
# Absent subfolder → serve aborts with a hint to `huggingface-cli download <repo>`.
[[extra_mounts]]
repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4"
subfolder = "mtp-draft"           # optional; omit = whole snapshot dir
container = "mtp_draft"

[mods]
commands = [                         # run *inside* container before engine
    "pip install flash-attn",
    { command = "./patch.sh", on_failure = "warn" },
]

[env]
KEY = "val"                         # overrides launcher defaults

[build.versions]                    # optional; overrides pins in docker-images/<dir>/versions.env
VLLM_COMMIT = "00907fc..."          # recipe wins over the image's pin; folded into build checksum

Deployment target (`deploy`)

The deploy field selects how the recipe is launched on each node:

docker (default): docker run with the recipe's entrypoint. Supports all features (mods, composed_model, image sync, extra_mounts).
tmux: a detached tmux session running the engine directly (no Docker). The recipe's entrypoint is used when present (e.g. ["vllm", "serve"] → vllm serve); otherwise the launcher falls back to the engine's Python module path. Docker-only features are rejected: [mods], [composed_model], image sync, and --sync-image/--build-node/ --force-build CLI opts are ignored with a warning.

The serve command dispatches based on recipe.deploy:

engine = "vllm"
deploy = "tmux"                     # or "docker" (default)
entrypoint = ["vllm", "serve"]

Engine-specific args (args) are interpreted by the engine itself — the launcher doesn't validate them, so the same args list works for both vllm and sglang recipes. The entrypoint field is shared: both deployment paths use it, so a single recipe can be served via either docker or tmux without changes.

Venv creation (`venv` + `venv_setup`)

When deploy = "tmux", the venv field specifies a Python virtualenv path that is activated before the entrypoint (source <venv>/bin/activate && ...). The venv_setup field is an optional set of shell commands to create and initialize the venv if it does not already exist on a node. Before serve, the launcher checks test -f <venv>/bin/activate on each node; if absent, it runs the setup commands in order, then re-checks. A failure aborts the serve.

venv_setup accepts either an inline multi-line string (one command per line; blank lines and # comments are skipped) or a Vec<String> list:

deploy = "tmux"
venv = "~/venvs/vllm025"
venv_setup = """
    uv venv ~/venvs/vllm025 -p 3.12
    uv pip install vllm==0.25.1 --torch-backend=cu130
    uv pip install flashinfer-python==0.6.15.dev20260712 \
      flashinfer-cubin==0.6.15.dev20260712 \
      flashinfer-jit-cache==0.6.15.dev20260712 \
      --extra-index-url https://flashinfer.ai/whl/nightly/ \
      --extra-index-url https://flashinfer.ai/whl/nightly/cu130/
    hf download poolside/Laguna-S-2.1-NVFP4
    hf download poolside/Laguna-S-2.1-DFlash-NVFP4
"""

Or as a list:

venv_setup = [
    "uv venv ~/venvs/vllm025 -p 3.12",
    "uv pip install vllm==0.25.1 --torch-backend=cu130",
]

Alternatively, venv_setup_script references a shell script file (relative to the recipe directory) that is read and run on each node if the venv doesn't exist. Script file takes precedence over venv_setup if both are set:

venv_setup_script = "venv_setup.sh"

If the venv already exists on a node, the setup commands are skipped. venv_setup and venv_setup_script are ignored in Docker mode (the container has its own Python).

Env vars (auto-set)

Cluster-driven: NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX, GLOO_/TP_SOCKET_IFNAME, MN_IF_NAME, UCX_NET_DEVICES, OMPI_MCA_btl_tcp_if_include Defaults: OMP_NUM_THREADS=4, NCCL_IGNORE_CPU_AFFINITY=1, NCCL_NET=IB, NCCL_DEBUG=WARN Engine IP: VLLM_HOST_IP / SGLANG_WORKER_IP Head: VLLM_LOGGING_COLOR=1, TERM=xterm-256color, -p <port>:<port> Ray: RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS Recipe [env] wins (last wins in -e).

Mods

Mods run inside the container before the engine process via entrypoint wrapping. When [mods] is present, the recipe must also set entrypoint (the known process to exec at the end). Same on_failure semantics as before_serve (default: abort). On failure the container exits; configure docker run --restart externally to restart on crash.

Implementation: --entrypoint sh image -c 'mods && exec <original entrypoint>'

Built-in scripts (baked into the binary)

Host scripts under host-scripts/ are embedded at compile time (include_str! in src/builtin_scripts.rs) and can be referenced by alias from any before_serve / after_stop list. The launcher uploads

runs them via SSH per node — no scp from the operator's machine.

Entry shape (untagged enum, three forms):

before_serve = [
  "sudo nvidia-smi -lgc",                       # bare string = raw shell command
  { command = "echo hi", on_failure = "warn" },  # raw command + on_failure
  { script = "stop_dgx_services" },              # built-in alias
  { script = "stop_dgx_services", on_failure = "warn" },
]

A bare string is always a raw command (today's behavior, unchanged); a built-in is referenced only via { script = "<alias>" }. Unknown aliases error at resolution time with the list of known ones.

Built-in aliases:

Alias	File	Purpose
`stop_dgx_services`	`host-scripts/stop-dgx-services.sh`	Stop non-essential services on a DGX node (frees unified memory)
`restore_dgx_services`	`host-scripts/restore-dgx-services.sh`	Restore services stopped by `stop_dgx_services`

To add a built-in: drop the .sh into host-scripts/, add an include_str! + BuiltinScript entry in src/builtin_scripts.rs, document the alias here. Scripts stay diffable / shellcheck-able as real files.

Flags auto-generated

vLLM: --tensor-parallel-size, --pipeline-parallel-size, --nnodes, --node-rank, --master-addr, --master-port, --host, --port, --headless (workers)
SGLang: --tp, --nnodes, --node-rank, --dist-init-addr, --host, --port
Both: --distributed-executor-backend (ray only, mp is auto-detected)

Docker image builds

A recipe's docker_image is built on demand from docker-images/<docker_image_dir>/ (default <dir> = recipe name). sm121 serve --build-node <n> (or the standalone sm121 build) uploads the dir to a node, runs bash build.sh in a tmux session, and stamps a sm121.checksum label; later serves skip the build when the label matches a recursive SHA-256 of the dir.

Pinned builds (docker-images/<dir>/versions.env): every build input (CUDA base, torch/NCCL/flashinfer/vLLM fork+commit, GPU arch, …) is a pinned KEY=VAL. build.sh is a generic driver — it forwards every pin in versions.env as a --build-arg to the Dockerfile (no hardcoded variable list). Add a pin to versions.env and it is picked up automatically. locks/ holds --require-hashes lockfiles; checksums/ holds verified out-of-band downloads (e.g. tiktoken encodings). See docker-images/glm52-nf3-hybrid/ for the reference cu130 / aarch64 / sm_121a stack; docker-images/solar-open2-nvfp4/ is the same stack minus the b12x suite (stock vLLM serve).

Recipe overrides ([build.versions]): a recipe may override any pin without editing the image dir. The launcher merges recipe pins over the image's versions.env (recipe wins), writes the merged file to the node, and folds the overrides into the checksum so a pin change triggers a rebuild. Keys must match versions.env names the Dockerfile declares as ARG.

[build.versions]
VLLM_COMMIT = "00907fc9b982f973808a0193ad0f8c61fbb62854"  # bump fork commit for this recipe only
TORCH_VERSION = "2.11.0"

Structured output under speculative decoding (glm52)

The glm52 image patches vLLM's structured-output reasoning-boundary logic (upstream PRs #44297 + #44993) to fix guided decoding (response_format) under MTP + reasoning parser. Without the patch, when the reasoning parser emits the reasoning-end marker (</think>) under speculative decoding, the grammar FSM fails to advance, so the content channel is unconstrained — models emit prose or markdown-fence prefixes before JSON (tool-eval-bench Structured Output: 50% instead of 100%). The patch is applied at image build time by docker-images/glm52/patch-structured-output.py, which modifies three files in the installed vLLM package:

v1/structured_output/request.py — adds reasoning_end_token_index field
v1/structured_output/__init__.py — should_advance() accepts new_token_ids (authoritative step delta), returns True for all constraint types at the boundary, records the index; grammar_bitmask() detects mid-window reasoning end and constrains subsequent drafts; adds _find_reasoning_end_index() and trim_reasoning_for_advance()
v1/core/sched/scheduler.py — update_from_output() passes new_token_ids to should_advance() and trims reasoning content before accept_tokens()

Fixes: vLLM #34650, #43388, #48228. The b12x fork predates the upstream merge (Jul 23, 2026); this patch is a forward-port until the fork rebases.

Composed model directories

A recipe may declare [composed_model] to combine multiple HF repos into a single model directory at serve time — e.g. a base checkpoint + an expert overlay — without mutating the HF cache. The launcher builds a /tmp dir per-node with symlinks into the HF cache (container-side absolute paths) + generated files, bind-mounts it at mount, and rewrites model to that path.

Symlinks point to /root/.cache/huggingface/... (the container-side HF cache mount), not the host-side path. They are dangling on the host but resolve correctly inside the container where the HF cache is mounted at that exact path — avoiding the host↔container mount-root mismatch. Later sources win (overwrite earlier symlinks), so order matters for overlays.

[composed_model]
mount = "/root/model"               # container-side target; `model` is rewritten to this

[[composed_model.sources]]          # symlinked into the composed dir (later wins)
repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ"      # base: config, weights, tokenizer
# (no glob → whole hierarchy symlinked recursively, incl. subdirs)

[[composed_model.sources]]
repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ"  # overlay: Lloyd NF4 experts
glob = "L*.safetensors"                             # only expert weight files (glob, may include / for subdirs)
exclude = ["L5.safetensors"]                       # optional: skip these paths (keep earlier source's version)

[[composed_model.files]]            # generated real files (not symlinks)
name = "nf4_layers.json"
content = '{"layers":[3,4,...,77]}'

With no glob, the entire snapshot hierarchy is symlinked recursively (every file in every subdir, as individual symlinks with real parent dirs) — so a later source can overlay individual files at any path. With a glob, only matching entries are symlinked (one level; / allowed for a specific subdir, e.g. grids/*.json) and the directory structure is preserved. exclude (optional list of path patterns) skips matched files so an earlier source's version survives — e.g. exclude = ["L5.safetensors"] keeps the base checkpoint's L5.safetensors instead of the overlay's. Allowed characters (glob + exclude): alphanumeric, * ? . _ - /.

The composed dir is cleaned up automatically on sm121 stop.

Serving notes (DeepSeek-V4-Flash / DSpark on GB10)

Operational findings from the SvangenStudios agent-serving benchmark (v0.1.1, measured on this exact stack: DS4-Flash-0731, DSpark k=5, 2× DGX Spark, TP=2). See specs/tech-architecture/dgx-spark-agent-serving-findings.md for full detail.

repetition_penalty crashes the DSpark path — an illegal-memory-access that kills the serving container. Any gateway/proxy in front of an sm121-served DSpark model must filter repetition_penalty from requests before forwarding. presence_penalty and frequency_penalty are verified safe. This is a DSpark-stack bug, not model behavior; the benchmark explicitly did not repeat the destructive test.

UMA memory gate (vLLM #48140) — on GB10 (unified memory), vLLM's startup KV-cache budget check effectively reads Linux MemFree, so ~20 GB of reclaimable page cache (created by reading the weights) is booked as unavailable. The effect: large --max-model-len profiles (128K–262K+) may deterministically refuse to boot even when the memory should fit, while 32K–64K boots normally. Two candidate mitigations:

run echo 3 > /proc/sys/vm/drop_caches (or the built-in drop_caches mod from eugr's repo) as a before_serve hook before the engine starts;
a local UMA-aware patch of gpu_worker.py to use MemAvailable instead of MemFree. The drop_caches workaround remains unverified on this deployment — test before relying on it for a large-context recipe.

--gpu-memory-utilization is a KV lever, not a safety lever on GB10: weights take ~77.7 GiB of a ~119.6 GiB node, so the GMU ceiling directly squeezes the KV-cache budget. The benchmark found 0.80 boots but dies under traffic; 0.78 is the safe ceiling. Do not raise GMU to "be safe" — that reduces usable KV capacity. The deepseek-v4-flash-eugr / -src / -eugr-agent recipes all ship 0.78.

Three engines — an LLM server is three nearly-independent resources: prefill (compute-bound, serializes; one 32K prefill saturates the box), decode (bandwidth-bound, scales to ~N=4 concurrency), and admission (scheduler-bound, blocked during long prefill — a short request submitted mid-prefill waits until prefill completes, ~221× normal TTFT). --max-num-batched-tokens governs how often decode gets a scheduling slot between prefill chunks; 2048 gives 2.9× the decode share of 8192 under a concurrent prefill, for 7.7% prefill throughput. Use the deepseek-v4-flash-eugr-agent recipe (2048) for interactive agent traffic; use deepseek-v4-flash-eugr (8192) for exclusive long-context batch jobs.

Qwen3.8-27B recipes (measured)

qwen38-* recipes serve the Qwen3.8-27B family on a single DGX Spark (dgx1, qwen38 cluster). The tested winner is qwen38-27b-nvfp4-unsloth (vLLM v0.27.1 + MTP k=5 on unsloth NVFP4): ~21.7 tok/s decode, 100/100 agentic tool-eval, and the MTP head is in-checkpoint. SGLang (DSpark and the DFlash2 fork) measured ~2.3× slower on this box despite the forum's 52-87 tok/s claims (an ndec.py cached-context artifact). See specs/tech-architecture/qwen38-recipe-state.md for the full measured matrix, why each decision was made, and how to re-benchmark.

Config fields

Config resolution: --config <path> > ./config.toml (cwd) > the global config (~/.config/sm121/config.toml on Linux, ~/Library/Application Support/sm121/config.toml on macOS — see sm121 config path). sm121 config init-global creates a template if missing. The global config exists so sm121 runs from any cwd: it holds the two source lists below (and may omit ssh_user/ssh_key_path — a cluster_sources file provides them when the main config leaves them empty); it may carry no [[clusters]] of its own — clusters may come entirely from cluster_sources.

ssh_user = "u"                   # may be empty — a cluster source can provide it
ssh_key_path = "/path"            # PathBuf; same

recipe_sources = ["/path/to/recipes"]       # extra recipe dirs (each `<name>/recipe.toml`); searched after ./recipes
cluster_sources = ["/path/clusters.toml"]   # extra cluster def files: `ssh_user`/`ssh_key_path`,
                                            # `[[clusters]]`, `[nccl.<name>]`, `[before_serve.<name>]`;
                                            # merged by name, this file wins

[[clusters]]
name = "c"
master_ib_ip = "10.0.0.1"
nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }]
extra_entities = [                  # cluster-level HA entities
    "switch.network",                # bare string = blocking (off → cluster down)
    { entity = "switch.fan", blocks = false },  # non-blocking: toggled + shown, never gates up/down
]

[nccl."c"]
ib_hca = "..."
socket_ifname = "..."
ib_gid_index = 3        # optional: omit to auto-detect (RoCE-v2 IPv4 GID, per node)
ib_disable = 0

[home_assistant]
url = "..."
token = "..."
shutdown_delay_seconds = 30

[exec_aliases]
name = "command"

[mount_aliases]                     # host paths referenced by name from recipe [[extra_mounts]]
jit_cache = "/mnt/nas/ai/jit"      # alias wins over recipe `host`; lets operators repoint mounts site-wide
hf_cache = "/mnt/nas/ai/huggingface"

[before_serve."c"]
commands = ["cmd", { command = "cmd", on_failure = "warn" }]

Power

off: SSH shutdown → wait → HA turn_off per-node entities + cluster extras
on: HA turn_on (no SSH, requires HA config)
switch --from <src> --to <tgt>: disable nodes/extras unique to from, enable all in to

NCCL_IB_GID_INDEX auto-detect

The RoCE-v2 IPv4 GID index re-numbers across reboots on DGX Spark, so a pinned gid can silently go stale. When a cluster's [nccl] block omits ib_gid_index, serve resolves it at launch per node by SSHing in and scanning the sysfs IB GID table for the first RoCE-v2 / IPv4-mapped GID (mirrors the Aiden serving stack: forum 372268/708). The resolved value is injected into the docker -e / tmux export, so both deploy targets benefit. Set ib_gid_index = <n> explicitly to opt out (pin the value). Dry-run prints the detect command per node and renders NCCL_IB_GID_INDEX=auto.

i18n

fluent-i18n crate, auto-detect via set_locale(None). FTL in locales/.

Constraints

Files ≤500 lines (enforced by make check-lines); extract submodules when a file approaches the limit
#![deny(clippy::wildcard_imports)] — no use foo::*;
Clippy warnings: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do not add #[allow(...)] attributes. If the only viable fix is an allow attribute, consult the user for guidance.

AGENTS.md 25 KB Vēsture Neapstrādāts