# AGENTS.md — sm121

Rust CLI (`edition 2024`) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (`russh`). Docker default, tmux alternative.

## Dev quick-ref

| Action | Command |
|--------|---------|
| Build | `cargo build` |
| All gates | `make check` (fmt → clippy → test → ≤500 lines) |
| Dry-run | `cargo run -- -d serve --recipe deepseek-v4-flash` |
| Tests | `cargo test` (65, all passing) |
| l10n | `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent FTL files |

See [`DEVELOPMENT.md`](DEVELOPMENT.md) for architecture, module layout, and dev workflow.

## Commands

```
serve/stop/bench/logs/health/doctor/exec/power/tmux/init-host-keys/config/web/report
```

`serve` records the launch in the recipe's run ledger (`recipes/<name>/.lock`,
gitignored) and then **waits for the engine to actually come up** — polls
`GET /v1/models` on the head node up to 30 min (`--no-wait` skips); a run that
never reaches readiness is closed with its uptime and the serve fails.
Follow mode watches liveness (30 s probes; 2 misses = died): a dead engine —
or an ended log stream with a dead engine — fails loudly and closes the run.

`stop` stops ALL nodes in parallel, every node runs the verifying command
(polls up to 60 s that no labeled container remains) and failures are
aggregated: one straggler cannot leave the other nodes un-stopped, and the
error names the nodes that did not verify. The verified stop closes the run
in the ledger. `--no-wait` opts out of the per-node verification.

`report` — the table of tested recipes: markdown (default) or an interactive
HTML table via `--serve`/`--open`. It shows each recipe's run-ledger state
(● running + uptime — live in the HTML, launch count) and supports archive
curation: `--archive <name>` / `--unarchive <name>` flip the recipe's
`archived` flag (comment-preserving), archived recipes leave the output,
`--archived` shows them instead. `stop` waits until the containers are
actually stopped and removed (polls up 60 s per node) — an immediate `serve`
on the same nodes would otherwise race the dying containers' GPU release.

`bench` runs `tool-eval-bench` (from PATH) against a live serve and lands
`<UTC-date>_<label>.{md,json,log}` in `recipes/<lane>/reports/` — it does the
scratch-cwd/absolute-json/log juggling, polls the API for readiness first,
and prints the final score + perf rows. Defaults: seed 42, hardmode, perf,
concurrency 1,2; served name/port from the recipe.

**Run package (versioning per run):** every bench run also lands two more
files under the same stem — `<stem>.recipe.toml` (a byte-exact copy of the
recipe file as it stood at bench time) and `<stem>.manifest.json` (recipe +
config SHA-256, launcher git commit, tool-eval-bench version, and the served
image's identity per node — docker id + `sm121.checksum` build label,
captured over SSH at bench time). `sm121 report` reads the manifests back:
each run shows a provenance chip (`⚙ img@<buildsum12>·rcp@<sha12>`, ⚠ when
the recipe has since drifted from the copy) and the markdown table gets a
Provenance column. Old runs without a package show `—` / no chip.

`sm121 web` — local axum + SSE dashboard: **recipes panel** window (the
`sm121 report --serve` page mounted at `/recipes` — one implementation, its
own SSE stream and `/r/` viewer prefix-rewritten server-side), topology map
(physical layer:
8 GX10 boxes on a 2×4 grid, 200G fabric switch + passive 10G switch + DAC
direct links derived from config `ib_ip`/`extra_entities`, HA glyphs with
live states, cluster frames — click a frame chip to activate the cluster),
LED board (cluster-level health, non-blocking `extra_entities` as ⚙ badges),
themes, serve/stop/power jobs with type-the-name power gate, live log-less
jobs window fed by docker labels + SSE. UI checks: `cargo test ui_playwright_checks` (deno +
Playwright; one-time browser: `deno run -A src/web/pw-install.mjs`).
Design/progress: `specs/tech-architecture/web-dashboard.md`.

Cluster: exact name, prefix (`4`→`4nodes-switch`), or `auto` (smallest with NCCL + matching `required_nodes`).

`serve --solo` overrides cluster selection: finds the first free node across
all clusters and runs one single-node container there (no `--nnodes`/`--tp`/
`--master-addr` flags — the engine defaults to single-node). `--no-solo`
forces multi-node even if the recipe sets `solo = true`. A recipe can also
set `solo = true` as the default.

`serve --force` skips the occupancy check during cluster selection —
occupied nodes are kept in the available set (reachable is still required).
Use to run a lightweight test recipe alongside a live serve, or to force onto
a busy node. Dangerous for GPU recipes (OOM, port conflicts); the operator is
responsible for coexistence.

`serve --fit` (optional value) scales a recipe to a node count and lets
`tensor_parallel_size` track it (overrides the recipe's `required_nodes` and
`tensor_parallel_size`; `pipeline_parallel_size` is left untouched).
`--fit=N` scales to N nodes; bare `--fit` uses the whole selected cluster
(TP = cluster node count). With `--cluster auto`, `--fit=N` auto-picks the
smallest cluster with ≥N free nodes; bare `--fit` + `--cluster auto` picks
the smallest cluster, so pass `--cluster <name> --fit` to scale onto a
specific larger cluster. Incompatible with `--solo`.

`serve --mirror` (optional value) mounts a **per-node local mirror** of the HF

cache instead of the shared one at `/root/.cache/huggingface`. Bare `--mirror`
uses `~/.cache/hf-models`; `--mirror <path>` uses that exact path (on each
node). The mirror keeps the HF hub layout (`<mirror>/hub/models--org--model/...`),
so `refs/main` path refs, `{{repo}}` mounts, and composed-model symlinks work
unchanged. Before launch each node checks its mirror and, if a required repo
(main model + repo-based path refs + composed sources) is missing, copies it
from the shared cache (`cp -a`, one-time per-node cost); a repo absent from
BOTH locations aborts with a `huggingface-cli download` hint. Later serves read
the model locally — no streaming weights over slow shared storage at load time.

**Cluster health is cluster-level** (`health`, web dashboard): a cluster is
UP only when *all* its nodes are reachable AND its **blocking**
`extra_entities` (HA switches) report `on`. One offline node — or a blocking
switch off — takes the whole cluster down: containers on surviving nodes are
not reported (serve/stop/jobs views all ignore them; `power on` stays
available). `extra_entities` entries can opt out of gating with
`blocks = false` (e.g. a fan that may be off while the cluster serves) —
non-blocking entities are still toggled by `power` and shown as a soft badge,
but never take the cluster down. Shared rule:
`cluster_state::verdict` + `home_assistant::extra_entities_states`.

`serve --clear` deletes the **host-side `extra_mounts` folders** (alias/host
— repo-based path refs are never touched) on the launch nodes with
`sudo rm -rf` before containers start. The containers run as root, so the
cache files there are usually root-owned. Use to reset JIT/kernel caches
between serves. `-d --clear` prints the exact `sudo rm -rf '...'` command
per node instead of running it.

> ⚠️ **Cluster may be serving live traffic.** A model (often the one serving this
> very agent) may be running on the cluster at any time. Before running `stop`,
> `power off`, or `power switch`, ALWAYS first run `health` and confirm with the
> user that the cluster is not serving before proceeding. Do NOT assume the
> cluster is idle — a stray `stop`/`power off` has killed running agents mid-task
> before. `logs`, `health`, `doctor`, and dry-runs (`-d`) are always safe.

## Recipe format (`recipes/<name>/recipe.toml`)

Recipes and their image build contexts (`docker-images/`) both live here
and must move together — the served image must match the recipe revision.
<https://git.kilork.org/sm121/recipes> holds only `sm121 publish` output
(self-contained bundles) as the registry's storage.

```toml
engine = "vllm"                     # or "sglang"
docker_image = "img:tag"
model = "org/model"
parent_model = "org/base-model"     # optional; report category (medals/tree) ranks under the base model
archived = false                    # optional; true hides the recipe from `sm121 report` (set via `--archive`)
port = 30000                        # default
required_nodes = 2                  # optional; also caps the launch set (N nodes, not all)
solo = false                       # optional; single-node on first free node, no dist flags
deploy = "docker"                   # optional; "docker" (default) or "tmux"
venv = "~/venvs/vllm025"            # optional; Python venv for tmux (activates before entrypoint)
venv_setup = [...]                 # optional; commands to create venv if it doesn't exist (tmux only)
venv_setup_script = "venv.sh"      # optional; script file (relative to recipe dir) to create venv (tmux only)
tensor_parallel_size = 2
pipeline_parallel_size = 1          # vLLM only, 0 = auto
distributed_backend = "mp"          # or "ray"
shm_size = "64gb"                   # optional, only set when non-empty
entrypoint = ["vllm", "serve"]      # optional, after image; used by both docker and tmux
docker_image_dir = "glm52-nf3-hybrid"  # optional; build dir under docker-images/ (default: recipe name)
before_serve = [{ command = "echo", on_failure = "abort" }]
after_stop = [{ script = "restore_dgx_services" }]   # runs after `sm121 stop`
args = ["--kv-cache-dtype", "fp8"]

[[extra_mounts]]                    # extra bind-mounts for `docker run -v`
alias = "jit_cache"                # resolves via [mount_aliases] in config.toml; wins over `host`
host = "~/.cache/jit"              # fallback default if alias absent; `~` expanded to ssh_user home
container = "/root/.cache/jit"     # container-side target (required)
read_only = false                 # optional, default false (adds :ro)
env = "XDG_CACHE_HOME"            # optional: injects -e <name>=<container path>

# Model path reference (mutually exclusive with alias/host): resolves at
# serve time, per node, from the HF cache — refs/main → latest snapshot, so
# the path survives `huggingface-cli download` updates with no fragile
# symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the
# recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes
# the resolved **container-side** HF cache path per-node — the host cache-dir
# prefix (e.g. `/home/<user>/.cache/huggingface`) is rewritten to the
# container mount target `/root/.cache/huggingface` (no bind-mount — the HF
# cache is already mounted there, so symlinks resolve correctly).
# Absent subfolder → serve aborts with a hint to `huggingface-cli download <repo>`.
[[extra_mounts]]
repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4"
subfolder = "mtp-draft"           # optional; omit = whole snapshot dir
container = "mtp_draft"

[mods]
commands = [                         # run *inside* container before engine
    "pip install flash-attn",
    { command = "./patch.sh", on_failure = "warn" },
]

[env]
KEY = "val"                         # overrides launcher defaults
PORT = { from = "port" }            # or: value resolves from a recipe field at
                                     # serve time (supported target: port) —
                                     # the value is declared once, can't drift

build_profile = "glm53-r28"        # optional; named pin set from docker-images/<dir>/profiles/<name>.env

[build.versions]                    # optional; overrides pins in docker-images/<dir>/versions.env
VLLM_COMMIT = "00907fc..."          # recipe wins over profile > image pin; folded into build checksum
```

### Deployment target (`deploy`)

The `deploy` field selects how the recipe is launched on each node:

- `docker` (default): `docker run` with the recipe's `entrypoint`. Supports all features
  (mods, composed_model, image sync, extra_mounts).
- `tmux`: a detached `tmux` session running the engine directly (no Docker). The recipe's
  `entrypoint` is used when present (e.g. `["vllm", "serve"]` → `vllm serve`); otherwise
  the launcher falls back to the engine's Python module path. Docker-only features are
  rejected: `[mods]`, `[composed_model]`, image sync, and `--sync-image`/`--build-node`/
  `--force-build` CLI opts are ignored with a warning.

The `serve` command dispatches based on `recipe.deploy`:

```toml
engine = "vllm"
deploy = "tmux"                     # or "docker" (default)
entrypoint = ["vllm", "serve"]
```

Engine-specific args (`args`) are interpreted by the engine itself — the launcher doesn't
validate them, so the same `args` list works for both `vllm` and `sglang` recipes. The
`entrypoint` field is shared: both deployment paths use it, so a single recipe can be
served via either `docker` or `tmux` without changes.

### Venv creation (`venv` + `venv_setup`)

When `deploy = "tmux"`, the `venv` field specifies a Python virtualenv path that is
activated before the entrypoint (`source <venv>/bin/activate && ...`). The `venv_setup`
field is an optional set of shell commands to create and initialize the venv **if it
does not already exist** on a node. Before serve, the launcher checks
`test -f <venv>/bin/activate` on each node; if absent, it runs the setup commands
in order, then re-checks. A failure aborts the serve.

`venv_setup` accepts either an inline multi-line string (one command per line;
blank lines and `#` comments are skipped) or a `Vec<String>` list:

```toml
deploy = "tmux"
venv = "~/venvs/vllm025"
venv_setup = """
    uv venv ~/venvs/vllm025 -p 3.12
    uv pip install vllm==0.25.1 --torch-backend=cu130
    uv pip install flashinfer-python==0.6.15.dev20260712 \
      flashinfer-cubin==0.6.15.dev20260712 \
      flashinfer-jit-cache==0.6.15.dev20260712 \
      --extra-index-url https://flashinfer.ai/whl/nightly/ \
      --extra-index-url https://flashinfer.ai/whl/nightly/cu130/
    hf download poolside/Laguna-S-2.1-NVFP4
    hf download poolside/Laguna-S-2.1-DFlash-NVFP4
"""
```

Or as a list:

```toml
venv_setup = [
    "uv venv ~/venvs/vllm025 -p 3.12",
    "uv pip install vllm==0.25.1 --torch-backend=cu130",
]
```

Alternatively, `venv_setup_script` references a shell script file (relative to the
recipe directory) that is read and run on each node if the venv doesn't exist.
Script file takes precedence over `venv_setup` if both are set:

```toml
venv_setup_script = "venv_setup.sh"
```

If the venv already exists on a node, the setup commands are skipped. `venv_setup`
and `venv_setup_script` are ignored in Docker mode (the container has its own Python).

## Env vars (auto-set)

Cluster-driven: `NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX`, `GLOO_/TP_SOCKET_IFNAME`, `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include`
Defaults: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN`
Engine IP: `VLLM_HOST_IP` / `SGLANG_WORKER_IP`
Head: `VLLM_LOGGING_COLOR=1`, `TERM=xterm-256color`, `-p <port>:<port>`
Ray: `RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS`
Recipe `[env]` wins (last wins in `-e`).

## Mods

Mods run **inside** the container before the engine process via entrypoint
wrapping.  When `[mods]` is present, the recipe **must** also set `entrypoint`
(the known process to `exec` at the end).  Same `on_failure` semantics as
`before_serve` (default: `abort`).  On failure the container exits; configure
`docker run --restart` externally to restart on crash.

Implementation: `--entrypoint sh image -c 'mods && exec <original entrypoint>'`

## Built-in scripts (baked into the binary)

Host scripts under `host-scripts/` are embedded at compile time
(`include_str!` in `src/builtin_scripts.rs`) and can be referenced by
**alias** from any `before_serve` / `after_stop` list. The launcher uploads
+ runs them via SSH per node — no `scp` from the operator's machine.

Entry shape (untagged enum, three forms):

```toml
before_serve = [
  "sudo nvidia-smi -lgc",                       # bare string = raw shell command
  { command = "echo hi", on_failure = "warn" },  # raw command + on_failure
  { script = "stop_dgx_services" },              # built-in alias
  { script = "stop_dgx_services", on_failure = "warn" },
]
```

A bare string is always a raw command (today's behavior, unchanged); a
built-in is referenced only via `{ script = "<alias>" }`. Unknown aliases
error at resolution time with the list of known ones.

Built-in aliases:

| Alias | File | Purpose |
|-------|------|---------|
| `stop_dgx_services` | `host-scripts/stop-dgx-services.sh` | Stop non-essential services on a DGX node (frees unified memory) |
| `restore_dgx_services` | `host-scripts/restore-dgx-services.sh` | Restore services stopped by `stop_dgx_services` |

To add a built-in: drop the `.sh` into `host-scripts/`, add an
`include_str!` + `BuiltinScript` entry in `src/builtin_scripts.rs`, document
the alias here. Scripts stay diffable / shellcheck-able as real files.

## Flags auto-generated

- vLLM: `--tensor-parallel-size`, `--pipeline-parallel-size`, `--nnodes`, `--node-rank`, `--master-addr`, `--master-port`, `--host`, `--port`, `--headless` (workers)
- SGLang: `--tp`, `--nnodes`, `--node-rank`, `--dist-init-addr`, `--host`, `--port`
- Both: `--distributed-executor-backend` (ray only, mp is auto-detected)

## Docker image builds

**All node operations go through `sm121` commands** — `serve` (with
`--build-node`), `stop`, `logs`, `health`, `bench`. Do NOT `scp` + `bash
build.sh` or `docker run/stop` over raw SSH: the launcher owns the upload,
the tmux build session, the checksum label, the label-based container
management, and the stop-wait verification. Raw SSH/docker is allowed only
for one-off diagnostics the launcher cannot express (hung-worker `docker
rm -f` by name, log forensics) — and those get noted in the shift history.

A recipe's `docker_image` is built on demand from `docker-images/<docker_image_dir>/`
(default `<dir>` = recipe name). `sm121 serve --build-node <n>` (or the
standalone `sm121 build`) uploads the dir to a node, runs `bash build.sh` in a
tmux session, and stamps a `sm121.checksum` label; later serves skip the build
when the label matches a recursive SHA-256 of the dir.

**Pinned builds** (`docker-images/<dir>/versions.env`): every build input
(CUDA base, torch/NCCL/flashinfer/vLLM fork+commit, GPU arch, …) is a pinned
`KEY=VAL`. `build.sh` is a generic driver — it forwards every pin in
`versions.env` as a `--build-arg` to the `Dockerfile` (no hardcoded variable
list). Add a pin to `versions.env` and it is picked up automatically.
`locks/` holds `--require-hashes` lockfiles; `checksums/` holds verified
out-of-band downloads (e.g. tiktoken encodings). See
`docker-images/glm52-nf3-hybrid/` for the reference cu130 / aarch64 / sm_121a
stack; `docker-images/solar-open2-nvfp4/` is the same stack minus the b12x
suite (stock vLLM serve).

**Recipe pin overrides** (`build_profile` + `[build.versions]`): a recipe
may override any pin without editing the image dir. A `build_profile =
"<name>"` references `docker-images/<dir>/profiles/<name>.env` (same `KEY=VAL`
format) — the way to ship a full pin set by name (an engine generation cut
across all forks: vLLM + b12x + LMCache together; see
`docker-images/glm53-jovian-r27/profiles/glm53-r28.env`). Precedence:
image `versions.env` < profile < recipe `[build.versions]` (one-off
experiments). The merged file is written to the node and folded into the
checksum, so any pin change triggers a rebuild. Keys must match `versions.env`
names the `Dockerfile` declares as `ARG`.

When cutting a new profile, bump **every fork of the stack together** (the
source lock is cut as one set — see the profile header comment); a half-cut
(vLLM advanced, b12x/LMCache left behind) silently skips loader/perf fixes.

```toml
build_profile = "glm53-r28"        # full r28 lock: vLLM + b12x + LMCache
# or, for a one-off experiment on top of a profile:
[build.versions]
TORCH_VERSION = "2.11.0"
```

### Structured output under speculative decoding (glm52)

The glm52 image patches vLLM's structured-output reasoning-boundary logic
(upstream PRs #44297 + #44993) to fix guided decoding (`response_format`)
under MTP + reasoning parser. Without the patch, when the reasoning parser
emits the reasoning-end marker (`</think>`) under speculative decoding, the
grammar FSM fails to advance, so the content channel is unconstrained —
models emit prose or markdown-fence prefixes before JSON (tool-eval-bench
Structured Output: 50% instead of 100%). The patch is applied at image
build time by `docker-images/glm52/patch-structured-output.py`, which
modifies three files in the installed vLLM package:

1. `v1/structured_output/request.py` — adds `reasoning_end_token_index` field
2. `v1/structured_output/__init__.py` — `should_advance()` accepts
   `new_token_ids` (authoritative step delta), returns `True` for all
   constraint types at the boundary, records the index; `grammar_bitmask()`
   detects mid-window reasoning end and constrains subsequent drafts;
   adds `_find_reasoning_end_index()` and `trim_reasoning_for_advance()`
3. `v1/core/sched/scheduler.py` — `update_from_output()` passes
   `new_token_ids` to `should_advance()` and trims reasoning content before
   `accept_tokens()`

Fixes: vLLM #34650, #43388, #48228. The b12x fork predates the upstream
merge (Jul 23, 2026); this patch is a forward-port until the fork rebases.

## Composed model directories

A recipe may declare `[composed_model]` to combine multiple HF repos into a
single model directory at serve time — e.g. a base checkpoint + an expert
overlay — without mutating the HF cache. The launcher builds a `/tmp` dir
per-node with symlinks into the HF cache (container-side absolute paths) +
generated files, bind-mounts it at `mount`, and rewrites `model` to that path.

Symlinks point to `/root/.cache/huggingface/...` (the container-side HF cache
mount), not the host-side path. They are dangling on the host but resolve
correctly inside the container where the HF cache is mounted at that exact
path — avoiding the host↔container mount-root mismatch. Later sources win
(overwrite earlier symlinks), so order matters for overlays.

```toml
[composed_model]
mount = "/root/model"               # container-side target; `model` is rewritten to this

[[composed_model.sources]]          # symlinked into the composed dir (later wins)
repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ"      # base: config, weights, tokenizer
# (no glob → whole hierarchy symlinked recursively, incl. subdirs)

[[composed_model.sources]]
repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ"  # overlay: Lloyd NF4 experts
glob = "L*.safetensors"                             # only expert weight files (glob, may include / for subdirs)
exclude = ["L5.safetensors"]                       # optional: skip these paths (keep earlier source's version)

[[composed_model.files]]            # generated real files (not symlinks)
name = "nf4_layers.json"
content = '{"layers":[3,4,...,77]}'

[[composed_model.local_files]]      # hardlinked from a per-node local host path
host = "/var/tmp/engram-local/DeepSeek-V4.1-Flash/model-00047-of-00048.safetensors"
name = "model-00047-of-00048.safetensors"  # optional; defaults to the host basename
```

`local_files` hardlinks real files from node-local storage (NVMe) into the
composed dir — run AFTER the repo symlink pass, so a local file replaces the
repo's symlink at the same name (local wins). Use to serve a shard from local
disk while the rest of the model stays on the shared cache (e.g. Engram
table shards from a sparse per-rank NVMe copy instead of the NAS). Hardlinked,
NOT copied — a sparse source stays sparse and costs zero extra disk; the host
path must be on the same filesystem as `/tmp` and exist on every launch node
(missing → serve aborts).

With **no glob**, the entire snapshot hierarchy is symlinked recursively
(every file in every subdir, as individual symlinks with real parent dirs) —
so a later source can overlay individual files at any path. With a `glob`,
only matching entries are symlinked (one level; `/` allowed for a specific
subdir, e.g. `grids/*.json`) and the directory structure is preserved.
`exclude` (optional list of path patterns) skips matched files so an earlier
source's version survives — e.g. `exclude = ["L5.safetensors"]` keeps the
base checkpoint's `L5.safetensors` instead of the overlay's.
Allowed characters (glob + exclude): alphanumeric, `* ? . _ - /`.

The composed dir is cleaned up automatically on `sm121 stop`.

## Serving notes (DeepSeek-V4-Flash / DSpark on GB10)

Operational findings from the SvangenStudios agent-serving benchmark (v0.1.1,
measured on this exact stack: DS4-Flash-0731, DSpark k=5, 2× DGX Spark,
TP=2). See `specs/tech-architecture/dgx-spark-agent-serving-findings.md`
for full detail.

**`repetition_penalty` crashes the DSpark path** — an illegal-memory-access
that kills the serving container. Any gateway/proxy in front of an
`sm121`-served DSpark model **must filter `repetition_penalty`** from
requests before forwarding. `presence_penalty` and `frequency_penalty` are
verified safe. This is a DSpark-stack bug, not model behavior; the benchmark
explicitly did not repeat the destructive test.

**UMA memory gate (vLLM #48140)** — on GB10 (unified memory), vLLM's startup
KV-cache budget check effectively reads Linux `MemFree`, so ~20 GB of
reclaimable page cache (created by *reading the weights*) is booked as
unavailable. The effect: large `--max-model-len` profiles (128K–262K+) may
**deterministically refuse to boot** even when the memory should fit, while
32K–64K boots normally. Two candidate mitigations:
- run `echo 3 > /proc/sys/vm/drop_caches` (or the built-in `drop_caches`
  mod from eugr's repo) as a `before_serve` hook before the engine starts;
- a local UMA-aware patch of `gpu_worker.py` to use `MemAvailable` instead
  of `MemFree`.
The `drop_caches` workaround remains unverified on this deployment — test
before relying on it for a large-context recipe.

**`--gpu-memory-utilization` is a KV lever, not a safety lever** on GB10:
weights take ~77.7 GiB of a ~119.6 GiB node, so the GMU ceiling directly
squeezes the KV-cache budget. The benchmark found **0.80 boots but dies
under traffic**; **0.78** is the safe ceiling. Do not raise GMU to "be safe" —
that *reduces* usable KV capacity. The deepseek-v4-flash-eugr / -src / -eugr-agent
recipes all ship 0.78.

**Three engines** — an LLM server is three nearly-independent resources:
prefill (compute-bound, serializes; one 32K prefill saturates the box),
decode (bandwidth-bound, scales to ~N=4 concurrency), and admission
(scheduler-bound, **blocked during long prefill** — a short request
submitted mid-prefill waits until prefill completes, ~221× normal TTFT).
`--max-num-batched-tokens` governs how often decode gets a scheduling slot
between prefill chunks; **2048 gives 2.9× the decode share of 8192** under a
concurrent prefill, for 7.7% prefill throughput. Use the
`deepseek-v4-flash-eugr-agent` recipe (2048) for interactive agent traffic;
use `deepseek-v4-flash-eugr` (8192) for exclusive long-context batch jobs.

### Qwen3.8-27B recipes (measured)

`qwen38-*` recipes serve the Qwen3.8-27B family on a single DGX Spark (dgx1,
`qwen38` cluster). The tested winner is **`qwen38-27b-nvfp4-unsloth`** (vLLM
`v0.27.1` + MTP k=5 on unsloth NVFP4): ~21.7 tok/s decode, 100/100 agentic
tool-eval, and the MTP head is in-checkpoint. SGLang (DSpark *and* the DFlash2
fork) measured ~2.3× slower on this box despite the forum's 52-87 tok/s claims
(an `ndec.py` cached-context artifact). See
`specs/tech-architecture/qwen38-recipe-state.md` for the full measured matrix,
why each decision was made, and how to re-benchmark.

## Config fields

Config resolution: `--config <path>` > `./config.toml` (cwd) > the global
config (`~/.config/sm121/config.toml` on Linux, `~/Library/Application
Support/sm121/config.toml` on macOS — see `sm121 config path`).
`sm121 config init-global` creates a template if missing. The global config
exists so `sm121` runs from **any cwd**: it holds the two source lists below
(and may omit `ssh_user`/`ssh_key_path` — a `cluster_sources` file provides
them when the main config leaves them empty); it may carry no
`[[clusters]]` of its own — clusters may come entirely from `cluster_sources`.

```toml
ssh_user = "u"                   # may be empty — a cluster source can provide it
ssh_key_path = "/path"            # PathBuf; same

recipe_sources = ["/path/to/recipes"]       # extra recipe dirs (each `<name>/recipe.toml`); searched after ./recipes
cluster_sources = ["/path/clusters.toml"]   # extra cluster def files: `ssh_user`/`ssh_key_path`,
                                            # `[[clusters]]`, `[nccl.<name>]`, `[before_serve.<name>]`;
                                            # merged by name, this file wins

[[clusters]]
name = "c"
master_ib_ip = "10.0.0.1"            # optional; omitted → rank-0 node's ib_ip (else ssh_ip)
nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }]
extra_entities = [                  # cluster-level HA entities
    "switch.network",                # bare string = blocking (off → cluster down)
    { entity = "switch.fan", blocks = false },  # non-blocking: toggled + shown, never gates up/down
]

[nccl."c"]
ib_hca = "..."
socket_ifname = "..."
ib_gid_index = 3        # optional: omit to auto-detect (RoCE-v2 IPv4 GID, per node)
ib_disable = 0

[home_assistant]
url = "..."
token = "..."
shutdown_delay_seconds = 30

[exec_aliases]
name = "command"

[mount_aliases]                     # host paths referenced by name from recipe [[extra_mounts]]
jit_cache = "/mnt/nas/ai/jit"      # alias wins over recipe `host`; lets operators repoint mounts site-wide
hf_cache = "/mnt/nas/ai/huggingface"

[before_serve."*"]                  # global: runs on every cluster; "*" is a literal key, not a glob
commands = ["sudo nvidia-smi -lgc 0,2000"]

[before_serve."c"]                  # per-cluster: runs after the global list, can supplement/override
commands = ["cmd", { command = "cmd", on_failure = "warn" }]
```

## Power

- `off`: SSH shutdown → wait → HA `turn_off` per-node entities + cluster extras
- `on`: HA `turn_on` (no SSH, requires HA config)
- `switch --from <src> --to <tgt>`: disable nodes/extras unique to `from`, enable all in `to`

## NCCL_IB_GID_INDEX auto-detect

The RoCE-v2 IPv4 GID index re-numbers across reboots on DGX Spark, so a pinned
gid can silently go stale. When a cluster's `[nccl]` block **omits**
`ib_gid_index`, `serve` resolves it at launch **per node** by SSHing in and
scanning the sysfs IB GID table for the first RoCE-v2 / IPv4-mapped GID
(mirrors the Aiden serving stack: forum 372268/708). The resolved value is
injected into the docker `-e` / tmux export, so both deploy targets benefit.
Set `ib_gid_index = <n>` explicitly to opt out (pin the value). Dry-run prints
the detect command per node and renders `NCCL_IB_GID_INDEX=auto`.

## i18n

`fluent-i18n` crate, auto-detect via `set_locale(None)`. FTL in `locales/`.

## Constraints

- Files ≤500 lines (enforced by `make check-lines`); extract submodules when a file approaches the limit
- Use `rg` (ripgrep) for all code search — plain `grep -rn` across this tree aborts/hangs and is unusable; `rg` is fast and skips `.git`/gitignored paths
- `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`
- **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.
- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 69+19 = 88 scenarios in 2.6.1.dev45; the older 176-pt pool is retired) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; re-baseline the anchor lane before comparing new lanes.
- **Harness version is pinned — do not `uv tool upgrade` casually.** `tool-eval-bench` is a uv tool (`~/.local/share/uv/tools/tool-eval-bench`) installed from `SeraphimSerapis/tool-eval-bench.git`; the receipt is rev-pinned at `6be685f0e` (v2.6.1.dev65, the campaign baseline since 2026-09-11 — the 167/176 record run is on it). The 19 dev46→dev65 commits (09-06..09-08) were **scoring-credit fixes** (correct behavior was being failed), so dev46 baselines (all GLM lanes: r28 158, r31 159, sparkring) read slightly low next to dev65 runs; cross-harness deltas include that bias. Reports stamp the harness version in the header — **A/B runs must share a harness version**. To move the pin deliberately: `uv tool install --reinstall --force "tool-eval-bench[tui,perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<rev>"` then re-baseline the anchor lane.