# AGENTS.md — sm121

Rust CLI (`edition 2024`) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (`russh`). Docker default, tmux alternative.

## Dev quick-ref

| Action | Command |
|--------|---------|
| Build | `cargo build` |
| All gates | `make check` (fmt → clippy → test → ≤500 lines) |
| Dry-run | `cargo run -- -d serve --recipe deepseek-v4-flash` |
| Tests | `cargo test` (65, all passing) |
| l10n | `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent FTL files |

See [`DEVELOPMENT.md`](DEVELOPMENT.md) for architecture, module layout, and dev workflow.

## Commands

```
serve/stop/bench/logs/health/doctor/exec/power/tmux/init-host-keys/config/web/report/runlock/validate
```

`validate` — offline recipe-contract gate (no cluster, no SSH): checks every
`recipes/<name>/recipe.toml` against the launcher contract — `docker_image_dir`
must exist under `docker-images/`, `port` must be the cluster default 30000,
no floating `:latest` on engine images (`alpine` test images exempt), models
must float (no `@revision`), `required_nodes` within 1..=8. Active recipes:
violations are **errors** (exit 1). Archived lanes are grandfathered — their
errors downgrade to **warnings** (advisory, always exit 0). Also runs as a
`cargo test` over the repo's own `recipes/`, so `make check` fails if a new
recipe breaks the contract. Rules live in `src/validate.rs`.

`serve` records the launch in the recipe's run ledger (`recipes/<name>/.lock`,
gitignored) and then **waits for the engine to actually come up** — polls
`GET /v1/models` on the head node up to 30 min (`--no-wait` skips); a run that
never reaches readiness is closed with its uptime and the serve fails.
Follow mode watches liveness (30 s probes; 2 misses = died): a dead engine —
or an ended log stream with a dead engine — fails loudly and closes the run.

`stop` stops ALL nodes in parallel, every node runs the verifying command
(polls up to 60 s that no labeled container remains) and failures are
aggregated: one straggler cannot leave the other nodes un-stopped, and the
error names the nodes that did not verify. The verified stop closes the run
in the ledger. `--no-wait` opts out of the per-node verification.

`report` — the table of tested recipes: markdown (default) or an interactive
HTML table via `--serve`/`--open`. It shows each recipe's run-ledger state
(● running + uptime — live in the HTML, launch count) and supports archive
curation: `--archive <name>` / `--unarchive <name>` flip the recipe's
`archived` flag (comment-preserving), archived recipes leave the output,
`--archived` shows them instead. `stop` waits until the containers are
actually stopped and removed (polls up 60 s per node) — an immediate `serve`
on the same nodes would otherwise race the dying containers' GPU release.

`runlock` — offline run-ledger management (the per-recipe `.lock`):
`list` shows every recipe with an open (running) entry, `close <recipe>`
closes the open run without any node verification. `stop` closes the ledger
only after verifying every node of the cluster, so a node that died
mid-run (or a killed serve) leaves the run open — `sm121 report --serve`
cross-checks the ledger against a live fleet probe and renders such rows as
orange "stale" (tooltip: `sm121 runlock close <recipe>`) instead of green
"up" when all nodes of the run's cluster are reachable but no running
container of the recipe exists; an unreachable node is inconclusive and
keeps the green (the fix for a confirmed stale row is `runlock close`).

`bench` runs `tool-eval-bench` (from PATH) against a live serve and lands
`<UTC-date>_<label>.{md,json,log}` in `recipes/<lane>/reports/` — it does the
scratch-cwd/absolute-json/log juggling, polls the API for readiness first,
and prints the final score + perf rows. Defaults: seed 42, hardmode, perf,
concurrency 1,2; served name/port from the recipe.

**Depth suite:** every campaign also appends a perf-only depth sweep
(`src/bench_depth.rs`): depths `16384,65536,131072,262144` at **c1 and c2**,
2 runs per point, pp2048/tg128, same log file → the report's Throughput
table gets the long-context rows; a depth failure only warns. Both perf
paths pass `--benchy-args=--enable-prefix-caching` — **without that flag
llama-benchy's defaults measure two serialized cold 67K prefills and the c2
depth row collapses to 13.7/5.7/2.1 t/s (a harness artifact, NOT the
engine — case + evidence:`specs/tech-architecture/sparkring-mimo-lanes.md`)`.
Never quote depth c2 numbers from a run older than 2026-09-29.

**Run package (versioning per run):** every bench run also lands two more
files under the same stem — `<stem>.recipe.toml` (a byte-exact copy of the
recipe file as it stood at bench time) and `<stem>.manifest.json` (recipe +
config SHA-256, launcher git commit, tool-eval-bench version, and the served
image's identity per node — docker id + `sm121.checksum` build label,
captured over SSH at bench time). `sm121 report` reads the manifests back:
each run shows a provenance chip (`⚙ img@<buildsum12>·rcp@<sha12>`, ⚠ when
the recipe has since drifted from the copy) and the markdown table gets a
Provenance column. Old runs without a package show `—` / no chip.

`sm121 web` — local axum + SSE dashboard: **recipes panel** window (the
`sm121 report --serve` page mounted at `/recipes` — one implementation, its
own SSE stream and `/r/` viewer prefix-rewritten server-side), topology map
(physical layer:
8 GX10 boxes on a 2×4 grid, 200G fabric switch + passive 10G switch + DAC
direct links derived from config `ib_ip`/`extra_entities`, HA glyphs with
live states, cluster frames — click a frame chip to activate the cluster),
LED board (cluster-level health, non-blocking `extra_entities` as ⚙ badges),
themes, serve/stop/power jobs with type-the-name power gate, live log-less
jobs window fed by docker labels + SSE. UI checks: `cargo test ui_playwright_checks` (deno +
Playwright; one-time browser: `deno run -A src/web/pw-install.mjs`).
Design/progress: `specs/tech-architecture/web-dashboard.md`.

Cluster: exact name, prefix (`4`→`4nodes-switch`), or `auto` (smallest with NCCL + matching `required_nodes`).

`serve --solo` overrides cluster selection: finds the first free node across
all clusters and runs one single-node container there (no `--nnodes`/`--tp`/
`--master-addr` flags — the engine defaults to single-node). `--no-solo`
forces multi-node even if the recipe sets `solo = true`. A recipe can also
set `solo = true` as the default.

`serve --force` skips the occupancy check during cluster selection —
occupied nodes are kept in the available set (reachable is still required).
Use to run a lightweight test recipe alongside a live serve, or to force onto
a busy node. Dangerous for GPU recipes (OOM, port conflicts); the operator is
responsible for coexistence.

`serve --fit` (optional value) scales a recipe to a node count and lets
`tensor_parallel_size` track it (overrides the recipe's `required_nodes` and
`tensor_parallel_size`; `pipeline_parallel_size` is left untouched).
`--fit=N` scales to N nodes; bare `--fit` uses the whole selected cluster
(TP = cluster node count). With `--cluster auto`, `--fit=N` auto-picks the
smallest cluster with ≥N free nodes; bare `--fit` + `--cluster auto` picks
the smallest cluster, so pass `--cluster <name> --fit` to scale onto a
specific larger cluster. Incompatible with `--solo`.

`serve --mirror` (optional value) mounts a **per-node local mirror** of the HF

cache instead of the shared one at `/root/.cache/huggingface`. Bare `--mirror`
uses `~/.cache/hf-models`; `--mirror <path>` uses that exact path (on each
node). The mirror keeps the HF hub layout (`<mirror>/hub/models--org--model/...`),
so `refs/main` path refs, `{{repo}}` mounts, and composed-model symlinks work
unchanged. Before launch each node checks its mirror and, if a required repo
(main model + repo-based path refs + composed sources) is missing, copies it
from the shared cache (`cp -a`, one-time per-node cost); a repo absent from
BOTH locations aborts with a `huggingface-cli download` hint. Later serves read
the model locally — no streaming weights over slow shared storage at load time.

`serve --hf-cache <path>` overrides the **host-side HF cache dir** for that
serve (config `hf_cache_dir`, else `~/.cache/huggingface`); `~` expands to
the node's SSH-user home (quote it — the shell would expand it locally first).
The mount target stays `/root/.cache/huggingface`, so model path refs and
repo mounts resolve as usual. Mutually exclusive with `--mirror` (both pick
the HF mount source — the error says so).

**Declarative staging (`[[staging]]`, spec
[`specs/tech-architecture/staging-feature.md`](specs/tech-architecture/staging-feature.md)):**
a recipe declares `[[staging]] kind = "hub-mirror"` blocks and the operator
runs plain `serve` — the launcher materializes the recipe's model data on
every launch node **before** containers start, then mounts the node-local
mirror instead of the shared HF cache (`dst` replaces the `--hf-cache` /
config source; `--mirror` is rejected when a block declares a mirror).
Verification is **content-based**: a per-root ledger (`<dst>/hub/.sm121-
staging-ledger.json`) stores sampled content hashes — never mtime/size (the
stale-NFS failure class) — so an unchanged repo costs zero copies, a
same-size mutation in the mirror is detected and re-copied, and the second
serve is verification-only. `repos` defaults to the recipe's model + repo
path-refs + composed sources; `skip` leaves external/managed repos alone;
`min_free` gates the whole pass per node (`G`/`T`/`%`). `--stage=skip`
assumes staged (CI/known-good), `--stage=force` re-verifies everything.
`kind = "sparse-rows"` (spec P2) packs each launch rank's slice of the
Engram tables onto node-local NVMe (`dst`, ledger of sampled row hashes in
the dst dir); rank/tp derive from the cluster and recipe — row ranges are
never hardcoded. The runtime-visible dir must be bind-mounted by the
recipe's `extra_mounts` (`dst` is host-side). The first staging serve packs
(~15 min over the NAS source); combine with `--no-wait` if a cold weight
load also pushes launch past the readiness budget. `format = "sparse-copy"`
switches the output from the packed-bin shards to safetensors-pread layout
— sparse node-local copies of the original shards containing exactly the
rank's row ranges (holes are free) + `engram-local.json` row-range
manifest: the aiden16/DSV41_ENGRAM_DIR contract. Sample-hash verification
also **adopts** a manually staged tree without re-copying.

**Cluster health is cluster-level** (`health`, web dashboard): a cluster is
UP only when *all* its nodes are reachable AND its **blocking**
`extra_entities` report `on`. HA entities are probed via HA REST; a
`[mikrotik]` device referenced by name is probed through the **RouterOS API
login** (logged in = on; unreachable or rejected login = off, like a dead
switch — `health` warns with the reason). One offline node — or a blocking
switch off — takes the whole cluster down: containers on surviving nodes are
not reported (serve/stop/jobs views all ignore them; `power on` stays
available). `extra_entities` entries can opt out of gating with
`blocks = false` (e.g. a fan that may be off while the cluster serves) —
non-blocking entities are still toggled by `power` and shown as a soft badge,
but never take the cluster down. Shared rule:
`cluster_state::verdict` + `home_assistant::extra_entities_states`.

`serve --clear` deletes the **host-side `extra_mounts` folders** (alias/host
— repo-based path refs are never touched) on the launch nodes with
`sudo rm -rf` before containers start. The containers run as root, so the
cache files there are usually root-owned. Use to reset JIT/kernel caches
between serves. `-d --clear` prints the exact `sudo rm -rf '...'` command
per node instead of running it.

> ⚠️ **Cluster may be serving live traffic.** A model (often the one serving this
> very agent) may be running on the cluster at any time. Before running `stop`,
> `power off`, or `power switch`, ALWAYS first run `health` and confirm with the
> user that the cluster is not serving before proceeding. Do NOT assume the
> cluster is idle — a stray `stop`/`power off` has killed running agents mid-task
> before. `logs`, `health`, `doctor`, and dry-runs (`-d`) are always safe.

## Recipe format (`recipes/<name>/recipe.toml`)

**Port convention: every recipe serves on the cluster default port 30000 —
leave the `port` field unset (or `30000`). When adapting a foreign recipe
(forum/upstream), always drop or rewrite its `port` to the default — the
foreign port (8000/8888/…) is not used on this cluster and breaks the
bench/smoke defaults that read the recipe port.**

Recipes and their image build contexts (`docker-images/`) both live here
and must move together — the served image must match the recipe revision.
<https://git.kilork.org/sm121/recipes> holds only `sm121 publish` output
(self-contained bundles) as the registry's storage.

```toml
engine = "vllm"                     # or "sglang"
docker_image = "img:tag"
model = "org/model"
parent_model = "org/base-model"     # optional; report category (medals/tree) ranks under the base model
archived = false                    # optional; true hides the recipe from `sm121 report` (set via `--archive`)
port = 30000                        # default — the only port convention on this cluster
required_nodes = 2                  # optional; also caps the launch set (N nodes, not all)
solo = false                       # optional; single-node on first free node, no dist flags
deploy = "docker"                   # optional; "docker" (default) or "tmux"
venv = "~/venvs/vllm025"            # optional; Python venv for tmux (activates before entrypoint)
venv_setup = [...]                 # optional; commands to create venv if it doesn't exist (tmux only)
venv_setup_script = "venv.sh"      # optional; script file (relative to recipe dir) to create venv (tmux only)
tensor_parallel_size = 2
pipeline_parallel_size = 1          # vLLM only, 0 = auto
distributed_backend = "mp"          # or "ray"
shm_size = "64gb"                   # optional, only set when non-empty
entrypoint = ["vllm", "serve"]      # optional, after image; used by both docker and tmux
docker_image_dir = "glm52-nf3-hybrid"  # optional; build dir under docker-images/ (default: recipe name)
before_serve = [{ command = "echo", on_failure = "abort" }]
after_stop = [{ script = "restore_dgx_services" }]   # runs after `sm121 stop`
args = ["--kv-cache-dtype", "fp8"]

[[extra_mounts]]                    # extra bind-mounts for `docker run -v`
alias = "jit_cache"                # resolves via [mount_aliases] in config.toml; wins over `host`
host = "~/.cache/jit"              # fallback default if alias absent; `~` expanded to ssh_user home
container = "/root/.cache/jit"     # container-side target (required)
read_only = false                 # optional, default false (adds :ro)
env = "XDG_CACHE_HOME"            # optional: injects -e <name>=<container path>

# Model path reference (mutually exclusive with alias/host): resolves at
# serve time, per node, from the HF cache — refs/main → latest snapshot, so
# the path survives `huggingface-cli download` updates with no fragile
# symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the
# recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes
# the resolved **container-side** HF cache path per-node — the host cache-dir
# prefix (e.g. `/home/<user>/.cache/huggingface`) is rewritten to the
# container mount target `/root/.cache/huggingface` (no bind-mount — the HF
# cache is already mounted there, so symlinks resolve correctly).
# Absent subfolder → serve aborts with a hint to `huggingface-cli download <repo>`.
[[extra_mounts]]
repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4"
subfolder = "mtp-draft"           # optional; omit = whole snapshot dir
container = "mtp_draft"

[mods]
commands = [                         # run *inside* container before engine
    "pip install flash-attn",
    { command = "./patch.sh", on_failure = "warn" },
]

[env]
KEY = "val"                         # overrides launcher defaults
PORT = { from = "port" }            # or: value resolves from a recipe field at
                                     # serve time (supported target: port) —
                                     # the value is declared once, can't drift

build_profile = "r28"             # optional; named pin set from docker-images/<dir>/profiles/<name>.env

[build.versions]                    # optional; overrides pins in docker-images/<dir>/versions.env
VLLM_COMMIT = "00907fc..."          # recipe wins over profile > image pin; folded into build checksum
```

### Deployment target (`deploy`)

The `deploy` field selects how the recipe is launched on each node:

- `docker` (default): `docker run` with the recipe's `entrypoint`. Supports all features
  (mods, composed_model, image sync, extra_mounts).
- `tmux`: a detached `tmux` session running the engine directly (no Docker). The recipe's
  `entrypoint` is used when present (e.g. `["vllm", "serve"]` → `vllm serve`); otherwise
  the launcher falls back to the engine's Python module path. Docker-only features are
  rejected: `[mods]`, `[composed_model]`, image sync, and `--sync-image`/`--build-node`/
  `--force-build` CLI opts are ignored with a warning.

The `serve` command dispatches based on `recipe.deploy`:

```toml
engine = "vllm"
deploy = "tmux"                     # or "docker" (default)
entrypoint = ["vllm", "serve"]
```

Engine-specific args (`args`) are interpreted by the engine itself — the launcher doesn't
validate them, so the same `args` list works for both `vllm` and `sglang` recipes. The
`entrypoint` field is shared: both deployment paths use it, so a single recipe can be
served via either `docker` or `tmux` without changes.

### Venv creation (`venv` + `venv_setup`)

When `deploy = "tmux"`, the `venv` field specifies a Python virtualenv path that is
activated before the entrypoint (`source <venv>/bin/activate && ...`). The `venv_setup`
field is an optional set of shell commands to create and initialize the venv **if it
does not already exist** on a node. Before serve, the launcher checks
`test -f <venv>/bin/activate` on each node; if absent, it runs the setup commands
in order, then re-checks. A failure aborts the serve.

`venv_setup` accepts either an inline multi-line string (one command per line;
blank lines and `#` comments are skipped) or a `Vec<String>` list:

```toml
deploy = "tmux"
venv = "~/venvs/vllm025"
venv_setup = """
    uv venv ~/venvs/vllm025 -p 3.12
    uv pip install vllm==0.25.1 --torch-backend=cu130
    uv pip install flashinfer-python==0.6.15.dev20260712 \
      flashinfer-cubin==0.6.15.dev20260712 \
      flashinfer-jit-cache==0.6.15.dev20260712 \
      --extra-index-url https://flashinfer.ai/whl/nightly/ \
      --extra-index-url https://flashinfer.ai/whl/nightly/cu130/
    hf download poolside/Laguna-S-2.1-NVFP4
    hf download poolside/Laguna-S-2.1-DFlash-NVFP4
"""
```

Or as a list:

```toml
venv_setup = [
    "uv venv ~/venvs/vllm025 -p 3.12",
    "uv pip install vllm==0.25.1 --torch-backend=cu130",
]
```

Alternatively, `venv_setup_script` references a shell script file (relative to the
recipe directory) that is read and run on each node if the venv doesn't exist.
Script file takes precedence over `venv_setup` if both are set:

```toml
venv_setup_script = "venv_setup.sh"
```

If the venv already exists on a node, the setup commands are skipped. `venv_setup`
and `venv_setup_script` are ignored in Docker mode (the container has its own Python).

## Env vars (auto-set)

Cluster-driven: `NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX`, `GLOO_/TP_SOCKET_IFNAME`, `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include`
Defaults: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN`
Engine IP: `VLLM_HOST_IP` / `SGLANG_WORKER_IP`
Head: `VLLM_LOGGING_COLOR=1`, `TERM=xterm-256color`, `-p <port>:<port>`
Ray: `RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS`
Recipe `[env]` wins (last wins in `-e`).

## Mods

Mods run **inside** the container before the engine process via entrypoint
wrapping.  When `[mods]` is present, the recipe **must** also set `entrypoint`
(the known process to `exec` at the end).  Same `on_failure` semantics as
`before_serve` (default: `abort`).  On failure the container exits; configure
`docker run --restart` externally to restart on crash.

Implementation: `--entrypoint sh image -c 'mods && exec <original entrypoint>'`

## Built-in scripts (baked into the binary)

Host scripts under `host-scripts/` are embedded at compile time
(`include_str!` in `src/builtin_scripts.rs`) and can be referenced by
**alias** from any `before_serve` / `after_stop` list. The launcher uploads
+ runs them via SSH per node — no `scp` from the operator's machine.

Entry shape (untagged enum, three forms):

```toml
before_serve = [
  "sudo nvidia-smi -lgc",                       # bare string = raw shell command
  { command = "echo hi", on_failure = "warn" },  # raw command + on_failure
  { script = "stop_dgx_services" },              # built-in alias
  { script = "stop_dgx_services", on_failure = "warn" },
]
```

A bare string is always a raw command (today's behavior, unchanged); a
built-in is referenced only via `{ script = "<alias>" }`. Unknown aliases
error at resolution time with the list of known ones.

Built-in aliases:

| Alias | File | Purpose |
|-------|------|---------|
| `stop_dgx_services` | `host-scripts/stop-dgx-services.sh` | Stop non-essential services on a DGX node (frees unified memory) |
| `restore_dgx_services` | `host-scripts/restore-dgx-services.sh` | Restore services stopped by `stop_dgx_services` |

To add a built-in: drop the `.sh` into `host-scripts/`, add an
`include_str!` + `BuiltinScript` entry in `src/builtin_scripts.rs`, document
the alias here. Scripts stay diffable / shellcheck-able as real files.

## Flags auto-generated

- vLLM: `--tensor-parallel-size`, `--pipeline-parallel-size`, `--nnodes`, `--node-rank`, `--master-addr`, `--master-port`, `--host`, `--port`, `--headless` (workers)
- SGLang: `--tp`, `--nnodes`, `--node-rank`, `--dist-init-addr`, `--host`, `--port`
- Both: `--distributed-executor-backend` (ray only, mp is auto-detected)

## Docker image builds

**All node operations go through `sm121` commands** — `serve` (with
`--build-node`), `stop`, `logs`, `health`, `bench`. Do NOT `scp` + `bash
build.sh` or `docker run/stop` over raw SSH: the launcher owns the upload,
the tmux build session, the checksum label, the label-based container
management, and the stop-wait verification. Raw SSH/docker is allowed only
for one-off diagnostics the launcher cannot express (hung-worker `docker
rm -f` by name, log forensics) — and those get noted in the shift history.

Image distribution (`sync_image_to_nodes`) is node-to-node, never through
the Mac: the source node (newest build, or `--build-node`/`--sync-image-source`)
runs `docker save | ssh <peer> docker load` per peer, with the inner hop
host-key verified against the pinned fingerprint, and routed over the peer's
`ib_ip` (200G fabric) when it has one — with a logged fallback to `ssh_ip`
(10G mgmt) if the fabric hop fails, so a stale node-to-node `authorized_keys`
warns instead of blocking the serve.

A recipe's `docker_image` is built on demand from `docker-images/<docker_image_dir>/`
(default `<dir>` = recipe name). `sm121 serve --build-node <n>` (or the
standalone `sm121 build`) uploads the dir to a node, runs `bash build.sh` in a
tmux session, and stamps a `sm121.checksum` label; later serves skip the build
when the label matches a recursive SHA-256 of the dir.

**Pinned builds** (`docker-images/<dir>/versions.env`): every build input
(CUDA base, torch/NCCL/flashinfer/vLLM fork+commit, GPU arch, …) is a pinned
`KEY=VAL`. `build.sh` is a generic driver — it forwards every pin in
`versions.env` as a `--build-arg` to the `Dockerfile` (no hardcoded variable
list). Add a pin to `versions.env` and it is picked up automatically.
`locks/` holds `--require-hashes` lockfiles + `deps.lock` (the staged
dep-subtree fingerprints `fetch-deps.sh` verifies every build against —
dep commits, vendored-cutlass gitlink SHAs, `triton_kernels` content hash;
regenerate via `--regen-lock` / `--build-arg REGEN_LOCK=1` after a pin cut);
`checksums/` holds verified
out-of-band downloads (e.g. tiktoken encodings). See
`docker-images/glm52-nf3-hybrid/` for the reference cu130 / aarch64 / sm_121a
stack; `docker-images/solar-open2-nvfp4/` is the same stack minus the b12x
suite (stock vLLM serve).

**Recipe pin overrides** (`build_profile` + `[build.versions]`): a recipe
may override any pin without editing the image dir. A `build_profile =
"<name>"` references `docker-images/<dir>/profiles/<name>.env` (same `KEY=VAL`
format) — the way to ship a full pin set by name (an engine generation cut
across all forks: vLLM + b12x + LMCache together; see
`docker-images/jovian-judgement/profiles/r28.env`). Precedence:
image `versions.env` < profile < recipe `[build.versions]` (one-off
experiments). The merged file is written to the node and folded into the
checksum, so any pin change triggers a rebuild. Keys must match `versions.env`
names the `Dockerfile` declares as `ARG`.

When cutting a new profile, bump **every fork of the stack together** (the
source lock is cut as one set — see the profile header comment); a half-cut
(vLLM advanced, b12x/LMCache left behind) silently skips loader/perf fixes.

```toml
build_profile = "r28"             # full r28 lock: vLLM + b12x + LMCache
# or, for a one-off experiment on top of a profile:
[build.versions]
TORCH_VERSION = "2.11.0"
```

### Structured output under speculative decoding (glm52)

The glm52 image patches vLLM's structured-output reasoning-boundary logic
(upstream PRs #44297 + #44993) to fix guided decoding (`response_format`)
under MTP + reasoning parser. Without the patch, when the reasoning parser
emits the reasoning-end marker (`</think>`) under speculative decoding, the
grammar FSM fails to advance, so the content channel is unconstrained —
models emit prose or markdown-fence prefixes before JSON (tool-eval-bench
Structured Output: 50% instead of 100%). The patch is applied at image
build time by `docker-images/glm52/patch-structured-output.py`, which
modifies three files in the installed vLLM package:

1. `v1/structured_output/request.py` — adds `reasoning_end_token_index` field
2. `v1/structured_output/__init__.py` — `should_advance()` accepts
   `new_token_ids` (authoritative step delta), returns `True` for all
   constraint types at the boundary, records the index; `grammar_bitmask()`
   detects mid-window reasoning end and constrains subsequent drafts;
   adds `_find_reasoning_end_index()` and `trim_reasoning_for_advance()`
3. `v1/core/sched/scheduler.py` — `update_from_output()` passes
   `new_token_ids` to `should_advance()` and trims reasoning content before
   `accept_tokens()`

Fixes: vLLM #34650, #43388, #48228. The b12x fork predates the upstream
merge (Jul 23, 2026); this patch is a forward-port until the fork rebases.

## Composed model directories

A recipe may declare `[composed_model]` to combine multiple HF repos into a
single model directory at serve time — e.g. a base checkpoint + an expert
overlay — without mutating the HF cache. The launcher builds a `/tmp` dir
per-node with symlinks into the HF cache (container-side absolute paths) +
generated files, bind-mounts it at `mount`, and rewrites `model` to that path.

Symlinks point to `/root/.cache/huggingface/...` (the container-side HF cache
mount), not the host-side path. They are dangling on the host but resolve
correctly inside the container where the HF cache is mounted at that exact
path — avoiding the host↔container mount-root mismatch. Later sources win
(overwrite earlier symlinks), so order matters for overlays.

```toml
[composed_model]
mount = "/root/model"               # container-side target; `model` is rewritten to this

[[composed_model.sources]]          # symlinked into the composed dir (later wins)
repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ"      # base: config, weights, tokenizer
# (no glob → whole hierarchy symlinked recursively, incl. subdirs)

[[composed_model.sources]]
repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ"  # overlay: Lloyd NF4 experts
glob = "L*.safetensors"                             # only expert weight files (glob, may include / for subdirs)
exclude = ["L5.safetensors"]                       # optional: skip these paths (keep earlier source's version)

[[composed_model.files]]            # generated real files (not symlinks)
name = "nf4_layers.json"
content = '{"layers":[3,4,...,77]}'

[[composed_model.local_files]]      # hardlinked from a per-node local host path
host = "/var/tmp/engram-local/DeepSeek-V4.1-Flash/model-00047-of-00048.safetensors"
name = "model-00047-of-00048.safetensors"  # optional; defaults to the host basename
```

`local_files` hardlinks real files from node-local storage (NVMe) into the
composed dir — run AFTER the repo symlink pass, so a local file replaces the
repo's symlink at the same name (local wins). Use to serve a shard from local
disk while the rest of the model stays on the shared cache (e.g. Engram
table shards from a sparse per-rank NVMe copy instead of the NAS). Hardlinked,
NOT copied — a sparse source stays sparse and costs zero extra disk; the host
path must be on the same filesystem as `/tmp` and exist on every launch node
(missing → serve aborts).

With **no glob**, the entire snapshot hierarchy is symlinked recursively
(every file in every subdir, as individual symlinks with real parent dirs) —
so a later source can overlay individual files at any path. With a `glob`,
only matching entries are symlinked (one level; `/` allowed for a specific
subdir, e.g. `grids/*.json`) and the directory structure is preserved.
`exclude` (optional list of path patterns) skips matched files so an earlier
source's version survives — e.g. `exclude = ["L5.safetensors"]` keeps the
base checkpoint's `L5.safetensors` instead of the overlay's.
Allowed characters (glob + exclude): alphanumeric, `* ? . _ - /`.

The composed dir is cleaned up automatically on `sm121 stop`.

## Serving notes (DeepSeek-V4-Flash / DSpark on GB10)

Operational findings from the SvangenStudios agent-serving benchmark (v0.1.1,
measured on this exact stack: DS4-Flash-0731, DSpark k=5, 2× DGX Spark,
TP=2). See `specs/tech-architecture/dgx-spark-agent-serving-findings.md`
for full detail.

**`repetition_penalty` crashes the DSpark path** — an illegal-memory-access
that kills the serving container. Any gateway/proxy in front of an
`sm121`-served DSpark model **must filter `repetition_penalty`** from
requests before forwarding. `presence_penalty` and `frequency_penalty` are
verified safe. This is a DSpark-stack bug, not model behavior; the benchmark
explicitly did not repeat the destructive test.

**UMA memory gate (vLLM #48140)** — on GB10 (unified memory), vLLM's startup
KV-cache budget check effectively reads Linux `MemFree`, so ~20 GB of
reclaimable page cache (created by *reading the weights*) is booked as
unavailable. The effect: large `--max-model-len` profiles (128K–262K+) may
**deterministically refuse to boot** even when the memory should fit, while
32K–64K boots normally. Two candidate mitigations:
- run `echo 3 > /proc/sys/vm/drop_caches` (or the built-in `drop_caches`
  mod from eugr's repo) as a `before_serve` hook before the engine starts;
- a local UMA-aware patch of `gpu_worker.py` to use `MemAvailable` instead
  of `MemFree`.
The `drop_caches` workaround remains unverified on this deployment — test
before relying on it for a large-context recipe.

**`--gpu-memory-utilization` is a KV lever, not a safety lever** on GB10:
weights take ~77.7 GiB of a ~119.6 GiB node, so the GMU ceiling directly
squeezes the KV-cache budget. The benchmark found **0.80 boots but dies
under traffic**; **0.78** is the safe ceiling. Do not raise GMU to "be safe" —
that *reduces* usable KV capacity. The deepseek-v4-flash-eugr / -src / -eugr-agent
recipes all ship 0.78.

**Three engines** — an LLM server is three nearly-independent resources:
prefill (compute-bound, serializes; one 32K prefill saturates the box),
decode (bandwidth-bound, scales to ~N=4 concurrency), and admission
(scheduler-bound, **blocked during long prefill** — a short request
submitted mid-prefill waits until prefill completes, ~221× normal TTFT).
`--max-num-batched-tokens` governs how often decode gets a scheduling slot
between prefill chunks; **2048 gives 2.9× the decode share of 8192** under a
concurrent prefill, for 7.7% prefill throughput. Use the
`deepseek-v4-flash-eugr-agent` recipe (2048) for interactive agent traffic;
use `deepseek-v4-flash-eugr` (8192) for exclusive long-context batch jobs.

### Qwen3.8-27B recipes (measured)

`qwen38-*` recipes serve the Qwen3.8-27B family on a single DGX Spark (dgx1,
`qwen38` cluster). The tested winner is **`qwen38-27b-nvfp4-unsloth`** (vLLM
`v0.27.1` + MTP k=5 on unsloth NVFP4): ~21.7 tok/s decode, 100/100 agentic
tool-eval, and the MTP head is in-checkpoint. SGLang (DSpark *and* the DFlash2
fork) measured ~2.3× slower on this box despite the forum's 52-87 tok/s claims
(an `ndec.py` cached-context artifact). See
`specs/tech-architecture/qwen38-recipe-state.md` for the full measured matrix,
why each decision was made, and how to re-benchmark.

### GLM-5.3-Flash NVFP4 re-cut (measured)

`local-inference-lab/GLM-5.3-Flash-NVFP4` was **re-cut in place on 2026-09-16**
(the old ~186 GB upload is gone): 199.4 GB = 36 expert shards + 4 nonexpert +
4 MTP shards. Structure is identical to the Spark checkpoint (verified from
`model.safetensors.index.json`): `model.language_model.layers.*` with SPLIT GDN
naming, a `model.visual.*` tower (skip via `--language-model-only`), MTP head
embedded at layer 45 — so the JJ-fork engine requirements are the same as the
Spark lanes (r35 + `VLLM_DISABLED_KERNELS`; r38 pool-deadlock on 2x still
applies). Lanes: `glm53-flash-nvfp4-lab` (2x, same dir as the old pre-re-cut
lane — it had no reports/lock) and `glm53-flash-nvfp4-lab-4x` (4x).

Measured 2026-09-20 (r35, harness dev65, seed 42, hardmode):
- **2x: 164/176 (93/100) — the family best** (old 186 GB cut: 89/100; Spark r35
  2x: 92). The operator's "they improved it" hunch was right. The re-cut is
  heavier: ~93 GiB weights/rank, so auto-sizing leaves only ~2.4 GiB KV at
  GMU 0.88. With `--max-num-seqs 4` + `--max-num-batched-tokens 2048` (frees
  the profile-run peak) an explicit **10 GiB KV pin** fits: pool 1,406,229
  tokens = **2.68× a 524K request** → the 2x lane runs **512K context** (bench
  `2026-09-20_090824_512k-kv10g`: 160/176, 91/100; perf ≈ 256K config,
  pp ~1,500, tg 33/41 c1/c2). Host margin after boot: ~2-3 GiB available/node —
  12 GiB is the realistic pin ceiling.
- **4x: 159/176 (90/100)**, decode 54-68 t/s (2× the 2x), KV 49.3 GiB →
  7.35M-token pool @524K. The 4x loses 5 pts to the 2x (TC-43/45/49/68/80/88
  flakes); TC-51 fails on both (known harness recurring).
- Weights live on all 4 nodes' local NVMe (`hf-local`, snapshot
  `175ae8ce`); dgx1 is the tightest host (~55 GB free after the model).

### SparkRing MiMo lanes (MOPD — operator's fast line, 2026-09-29)

MOPD only from now on (RL lanes retired — RL keeps training artifacts).
Three lanes on the installer image
`ghcr.io/fujitsupolycom/sparkring@sha256:c977a2d2…` + their `toolchain.py
serve` entrypoint — full contract, deviations and diagnosis are in
[`specs/tech-architecture/sparkring-mimo-lanes.md`](specs/tech-architecture/sparkring-mimo-lanes.md):

- `mimo26-flash-mopd-4x-sparkring` (TP4, dgx58-switch) — benched honest
  **165/184**, c2 ≥ c1 at every depth up to 256K.
- `mimo26-flash-mopd-2x-sparkring` (TP2, **2nodes-dac**) — their qualified
  direct-pair-2 shape, `B12X_ROCE_PAIR_PATHS=2`, cluster block routes f1
  (DAC) with GID 3.
- `mimo26-pro-mopd-8x-sparkring` (TP8, **8nodes-switch**) — Pro-MOPD
  `adea8e2c5373` (573.5 GB NAS), no upstream profile → our tp4 contract
  adapted; needs fleet + `--sync-image`.

Dry-run any of them with `sm121 -d serve --recipe R --cluster C` before
touching nodes; serve timing for 2x/8x is operator-coordinated (prod
cluster must be free).

## Config fields

Config resolution: `--config <path>` > `./config.toml` (cwd) > the global
config (`~/.config/sm121/config.toml` on Linux, `~/Library/Application
Support/sm121/config.toml` on macOS — see `sm121 config path`).
`sm121 config init-global` creates a template if missing. The global config
exists so `sm121` runs from **any cwd**: it holds the two source lists below
(and may omit `ssh_user`/`ssh_key_path` — a `cluster_sources` file provides
them when the main config leaves them empty); it may carry no
`[[clusters]]` of its own — clusters may come entirely from `cluster_sources`.

```toml
ssh_user = "u"                   # may be empty — a cluster source can provide it
ssh_key_path = "/path"            # PathBuf; same

recipe_sources = ["/path/to/recipes"]       # extra recipe dirs (each `<name>/recipe.toml`); searched after ./recipes
cluster_sources = ["/path/clusters.toml"]   # extra cluster def files: `ssh_user`/`ssh_key_path`,
                                            # `[[clusters]]`, `[nccl.<name>]`, `[before_serve.<name>]`;
                                            # merged by name, this file wins

[[clusters]]
name = "c"
master_ib_ip = "10.0.0.1"            # optional; omitted → rank-0 node's ib_ip (else ssh_ip)
nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }]
extra_entities = [                  # cluster-level HA entities
    "switch.network",                # bare string = blocking (off → cluster down)
    { entity = "switch.fan", blocks = false },  # non-blocking: toggled + shown, never gates up/down
]

[nccl."c"]
ib_hca = "..."
socket_ifname = "..."
ib_gid_index = 3        # optional: omit to auto-detect (RoCE-v2 IPv4 GID, per node)
ib_disable = 0

[home_assistant]
url = "..."
token = "..."
shutdown_delay_seconds = 30

[exec_aliases]
name = "command"

[mount_aliases]                     # host paths referenced by name from recipe [[extra_mounts]]
jit_cache = "/mnt/nas/ai/jit"      # alias wins over recipe `host`; lets operators repoint mounts site-wide
hf_cache = "/mnt/nas/ai/huggingface"

[before_serve."*"]                  # global: runs on every cluster; "*" is a literal key, not a glob
commands = ["sudo nvidia-smi -lgc 0,2000"]

[before_serve."c"]                  # per-cluster: runs after the global list, can supplement/override
commands = ["cmd", { command = "cmd", on_failure = "warn" }]
```

## Image registry push (`[registry]`)

```toml
[registry]
base = "ghcr.io/kilork"   # push refs: {base}/{image-name}:{version} + :latest
```

**Versioned tags, never `:local`.** The version is the image's build
checksum (recursive SHA-256 of the image dir — the same value stamped as the
`sm121.checksum` label), so identical content always maps to the identical
tag: re-publishing "the same thing" cannot create a duplicate, changed
content is a new version tag. Each successful local build
(`sm121 build` / `serve --build-node`) then pushes
`{base}/{image-name}:<checksum:12>` + `:latest` from the build node.
`sm121 publish --push` publishes without a build: it finds the image on
`--node` (or probes all cluster nodes), takes the version from the
`sm121.checksum` label (image ID fallback for unlabeled prebuilt images),
and pushes from that node.

Duplicate guard: a publication ledger (`published.json` next to the global
config) records `{repo → {version, image digest, time}}`; a re-push with an
unchanged image digest is skipped (with a note) unless `--force`
(`build`/`serve`/`publish`). The push runs in a tmux session on the node
(`sm121-push-<label>`, log `/tmp/<label>-push.log`) with the same 30 s poll /
90 min cap as the build — a 30 GB push must not ride on one SSH session.
**The launcher stores no registry credentials**: the node must be logged in
once (`ssh <node> && docker login ghcr.io`); a failed push fails with the log
tail + a `docker login` hint. `--no-push` (build + serve) skips the push even
when `[registry]` is set; without the section, builds behave as before.

## Power

- `off`: SSH shutdown → wait → HA `turn_off` per-node entities + cluster extras
- `on`: HA `turn_on` with **ensure-powered**: a node the caller believes down (web: last
  fleet snapshot; CLI: 1.5 s TCP probe) whose plug reads on gets a **30 s plug power cycle**
  (a shorter cut leaves the PSU rail energized — the machine stays in soft-off). Each node's
  outcome becomes the job message (web) / log lines (CLI)
- jobs: the web registry persists to `web-jobs.json` next to the global config; jobs that
  were running at shutdown are marked failed with a `sm121 runlock list` hint
- `switch --from <src> --to <tgt>`: disable nodes/extras unique to `from`, enable all in `to`

### RouterOS devices (`[mikrotik]` config)

A cluster entity that names a `[mikrotik]` device is **not** an HA switch —
it is reached over the RouterOS API instead (`src/mikrotik.rs`, raw TCP
8728, variable-length word framing):

```toml
[mikrotik]
[[mikrotik.devices]]
name = "switch.dgx1"            # matched against node/entity names exactly
address = "192.168.88.1"
username = "admin"
password = "secret"
api_port = 8728                  # optional, default 8728
on_entity = "switch.lab_relay"   # optional: HA relay that powers it back on
```

- `power off` → `login` + `/system/shutdown` per matched device (after the
  HA off phase), **then the `on_entity` outlet relay is switched off via HA**
  — the API shutdown alone leaves the device powered (fans keep spinning,
  it can boot back up; an already-energized relay makes `power on` a no-op).
  A failed API call or relay warns, never blocks the rest of the flow.
- `power on` → `on_entity` relay via HA if set, otherwise a
  “power on manually” reminder (the API cannot wake an off router).
- `health` / web probe / serve auto power-on: mikrotik names are dispatched
  to the RouterOS API — ping for state, `on_entity` relay for `turn_on`
  (manual reminder when unset); never HA.
- Without a `[mikrotik]` section everything behaves exactly as before.

## NCCL_IB_GID_INDEX auto-detect

The RoCE-v2 IPv4 GID index re-numbers across reboots on DGX Spark, so a pinned
gid can silently go stale. When a cluster's `[nccl]` block **omits**
`ib_gid_index`, `serve` resolves it at launch **per node** by SSHing in and
scanning the sysfs IB GID table for the first RoCE-v2 / IPv4-mapped GID
(mirrors the Aiden serving stack: forum 372268/708). The resolved value is
injected into the docker `-e` / tmux export, so both deploy targets benefit.
Set `ib_gid_index = <n>` explicitly to opt out (pin the value). Dry-run prints
the detect command per node and renders `NCCL_IB_GID_INDEX=auto`.

## i18n

`fluent-i18n` crate, auto-detect via `set_locale(None)`. FTL in `locales/`.

## Constraints

- Files ≤500 lines (enforced by `make check-lines`); extract submodules when a file approaches the limit
- Use `rg` (ripgrep) for all code search — plain `grep -rn` across this tree aborts/hangs and is unusable; `rg` is fast and skips `.git`/gitignored paths
- `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`
- **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.
- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 69+19 = 88 scenarios in 2.6.1.dev45; the older 176-pt pool is retired) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; re-baseline the anchor lane before comparing new lanes.
- **Harness version is pinned — do not `uv tool upgrade` casually.** `tool-eval-bench` is a uv tool (`~/.local/share/uv/tools/tool-eval-bench`) installed from `SeraphimSerapis/tool-eval-bench.git`; the receipt is rev-pinned at tag `v2.7.0` (object `544afb576b3`, 2026-09-21 — the first stable release of the dev65 line; `reasoning_truncated` is now a distinct failure kind). The previous pin was `6be685f0e` (v2.6.1.dev65, campaign baseline 2026-09-11 — the 167/176 record run is on it); dev65-era anchors read slightly differently on v2.7.0 until re-baselined. The 19 dev46→dev65 commits (09-06..09-08) were **scoring-credit fixes** (correct behavior was being failed), so dev46 baselines (all GLM lanes: r28 158, r31 159, sparkring) read slightly low next to dev65 runs; cross-harness deltas include that bias. Reports stamp the harness version in the header — **A/B runs must share a harness version**. To move the pin deliberately: `uv tool install --reinstall --force "tool-eval-bench[perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<tag-or-rev>"` then re-baseline the anchor lane.
- **Bulk data moves node-to-node, never through the Mac** — the Mac→node link
  is ~32 MB/s (a 200 GB transfer via the Mac takes hours); node-to-node over the
  200G fabric does ~547 MB/s per stream. Node-to-node SSH works (the shared
  key is on every node; `ssh kilork@<200.x-ib-ip>` from a node; dgx4 has two
  RoCE IPs — 200.19 *and* 200.20 — use the `ib_ip` from config). Pattern for a
  large HF download: one `hf download` on the roomiest node (dgx4, 1.9 TB),
  then each target PULLS: `nohup bash -c "ssh <node> 'tar -hcf - -C <modeldir> snapshots trees refs' | tar xf - -C <modeldir>" &` (`-h` dereferences so targets don't need the hub blob layout). `hf download` on a node = `~/.local/bin/hf` (dgx2) or `uv tool install huggingface_hub` (others).
- **huggingface_hub ≥0.34 (the `hf` CLI) moved blobs to the hub level** —
  `hub/blobs/<xx>/<sha>` shared across models, per-model dirs are symlink trees
  (snapshot → per-model blob → hub blob). `du`/`find -type f` on a model dir
  only sees the small real files (~19 GB of the 199 GB) — judge completeness
  by `df` deltas or hub-level `du`. The 09-20 re-cut download migrated dgx4's
  cache layout this way (the Spark model there is hub-blob-backed now).
- **`pkill -f` over ssh self-matches** — the pattern also matches the remote
  shell's own `-c` command line, so `pkill -f "hf download"` kills the script
  itself mid-run (and a `nohup … hf download …` line in the same command makes
  the later pkill match again). Use the `[b]racket` trick for the pattern and
  run kill and launch in separate ssh calls.