docs(skills): add sm121-launcher and vllm-mod-patches skills

Project skills under .agents/skills (pi / Agent Skills standard):

- sm121-launcher: SM121 CLI operations — commands, cluster selection
  (exact/prefix/auto, --solo/--no-solo/--force), recipe anatomy, config
  fields, docker image build/checksum flow, dev gates, and the critical
  stop/power safety protocol (always health first, confirm not serving).
- vllm-mod-patches: conventions for modifying engine code in docker-images —
  minimal anchor-based patcher scripts (resolve module paths at runtime,
  refuse on source drift, idempotent --check/apply/--check), Dockerfile
  COPY+RUN wiring, env-gated default-off toggles, throwaway-container test
  workflow, and pitfalls learned from the DSpark/reasoning_effort work.

.agents/skills/sm121-launcher/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: sm121-launcher
+description: Operate the sm121 Rust CLI launcher for this repo — serve/stop/logs/health/doctor/power/tmux on the DGX Spark cluster, cluster+recipe selection, image builds, and dev gates. Use whenever working with sm121 commands, recipes, config, or the DGX Spark cluster.
+---
+
+# sm121 Launcher
+
+`sm121` is the Rust CLI (`edition 2024`) that launches vLLM/SGLang engine
+containers across the DGX Spark cluster over SSH (`russh`, Docker default,
+tmux alternative). A model may be **serving live traffic at any time** — read
+the Safety section before anything destructive.
+
+## Quick reference
+
+| Action | Command |
+|---|---|
+| Build | `cargo build` |
+| All gates (before commit) | `make check` |
+| Dry-run (safe, no SSH) | `cargo run -- -d serve --recipe <name>` |
+| Tests | `cargo test` (65) |
+| l10n | `locales/{en,ru,es,it,fr,de}/main.ftl` |
+
+## Commands
+
+`serve` / `stop` / `logs` / `health` / `doctor` / `exec` / `power` / `tmux` /
+`init-host-keys`.
+
+- `sm121 health --cluster <c>` — safe; ALWAYS run before stop/power (see Safety).
+- `sm121 logs --recipe <r> [--cluster <c>]` — safe.
+- `sm121 doctor [--repair]` — inspect/repair node+image state; safe.
+- `sm121 exec --recipe <r> -- <cmd>` — run a command in the serving container;
+  honors `exec_aliases` from `config.toml`.
+- `sm121 serve --recipe <r> [--cluster <c>] [--keep] [--follow]` — main launch;
+  `--keep` keeps the container after engine exit, `--follow` tails logs.
+- `sm121 stop --recipe <r> [--cluster <c>]` — teardown (Docker stop/rm, or tmux
+  session kill; runs `after_stop` hooks). ⚠️ see Safety.
+- `sm121 power off|on|switch --cluster <c>` — node power via HA. ⚠️ see Safety.
+- `sm121 tmux ...` — tmux ops (`serve` uses tmux when `deploy = "tmux"`).
+- `sm121 build --recipe <r>` — build/upload the recipe's docker image.
+- `sm121 init-host-keys` — seed SSH host keys on nodes.
+
+## Cluster selection
+
+- **Exact**: `--cluster 4nodes-switch`
+- **Prefix**: `--cluster 4` → `4nodes-switch` (must be unambiguous)
+- **Auto**: `--cluster auto` (or omit) → smallest cluster with NCCL config
+  matching `required_nodes`
+
+`serve --solo` overrides cluster selection: first free node across all clusters,
+single-node container (no `--nnodes/--tp/--master-addr` — engine defaults).
+`--no-solo` forces multi-node even when a recipe sets `solo = true`. `--force`
+skips the occupancy check (keeps occupied nodes in the available set; reachable
+still required) — dangerous for GPU recipes (OOM / port conflicts).
+
+## Recipe anatomy (`recipes/<name>/recipe.toml`)
+
+```toml
+engine = "vllm"                 # or "sglang"
+docker_image = "img:tag"
+docker_image_dir = "recipe-name" # build dir under docker-images/ (default: recipe name)
+model = "org/model"
+port = 30000
+required_nodes = 2
+tensor_parallel_size = 2
+pipeline_parallel_size = 1
+distributed_backend = "mp"      # or "ray"
+shm_size = "32gb"
+deploy = "docker"               # or "tmux"
+venv = "~/venvs/vllm025"        # tmux only
+venv_setup = ["..." ]           # tmux only; runs if venv missing
+entrypoint = ["vllm", "serve"]
+before_serve = [{ command = "echo", on_failure = "abort" }]
+after_stop = [{ script = "restore_dgx_services" }]
+args = ["--kv-cache-dtype", "fp8"]
+
+[[extra_mounts]]                # alias | host+container, or repo-based
+alias = "jit_cache"             # resolved via [mount_aliases] in config.toml
+container = "/root/.cache/jit"
+env = "XDG_CACHE_HOME"          # injects -e NAME=<container path>
+
+# repo-based mount (resolves HF cache path per-node at serve time):
+[[extra_mounts]]
+repo = "org/model"
+subfolder = "mtp-draft"         # optional
+container = "mtp_draft"
+
+[mods]                          # docker only; runs inside container before engine
+commands = ["pip install flash-attn"]
+
+[env]                           # overrides launcher defaults (last wins in -e)
+KEY = "val"
+
+[build.versions]                # override pins in docker-images/<dir>/versions.env
+VLLM_COMMIT = "00907fc..."
+```
+
+`before_serve` / `after_stop` entries: bare string = raw shell command;
+`{ command = "...", on_failure = "warn" }`; `{ script = "<alias>" }` references a
+built-in script baked into the binary (see **Built-in scripts** in AGENTS.md).
+
+## Config (`config.toml`, gitignored)
+
+```toml
+ssh_user, ssh_key_path
+[[clusters]] name/master_ib_ip/nodes[{name,ssh_ip,ib_ip,rank,entity}]/extra_entities
+[nccl."c"] ib_hca/socket_ifname/ib_gid_index/ib_disable
+[home_assistant] url/token/shutdown_delay_seconds
+[exec_aliases], [mount_aliases]
+[before_serve."c"] commands
+```
+
+`[mount_aliases]` lets operators repoint host paths site-wide — the alias wins
+over a recipe's `host`.
+
+## Docker image builds
+
+`sm121 serve` builds `docker_image` on demand from `docker-images/<dir>/`
+(`<dir>` = `docker_image_dir` or recipe name): uploads the dir to a node, runs
+`bash build.sh` in tmux, stamps a `sm121.checksum` label; later serves skip the
+build when the label matches a recursive SHA-256 of the dir. Pinned builds use
+`versions.env` (every build input as `KEY=VAL`, forwarded as `--build-arg`);
+recipe `[build.versions]` overrides win and fold into the checksum. `locks/` =
+`--require-hashes` lockfiles, `checksums/` = verified out-of-band downloads.
+
+## Safety (critical)
+
+> ⚠️ **Cluster may be serving live traffic.** A model (often the one serving this
+> very agent) may run on the cluster at any time. Before `stop`, `power off`,
+> or `power switch`, ALWAYS run `sm121 health --cluster <c>` first and confirm
+> with the user that the cluster is not serving. Never assume idle — a stray
+> stop/power has killed running agents mid-task before. `logs`, `health`,
+> `doctor`, and `-d` dry-runs are always safe.
+
+Known operational constraints (see AGENTS.md → *Serving notes*):
+- 0.78 `--gpu-memory-utilization` is the safe ceiling on GB10 for DS4-DSpark;
+  0.80 boots but dies under traffic.
+- `--max-num-batched-tokens 2048` for interactive agent traffic, 8192 for
+  exclusive batch jobs.
+
+## Dev workflow for new work
+
+1. Read the recipe+image pair; if touching engine code, follow the
+   `vllm-mod-patches` skill.
+2. `cargo build` then `cargo run -- -d serve --recipe <r> --cluster <c>` to
+   validate command construction (no SSH).
+3. `make check` before commit (fmt → clippy → test → ≤500 lines/file; no
+   `#[allow]` — refactor instead; no `use foo::*;`).
+
+## References
+
+- Full command reference & recipe format: `AGENTS.md`
+- Architecture / module layout / dev flow: `DEVELOPMENT.md`
+- Patch conventions for engine images: `vllm-mod-patches` skill

.agents/skills/vllm-mod-patches/SKILL.md

@@ -0,0 +1,152 @@
+---
+name: vllm-mod-patches
+description: Modify VLLM/engine code inside the docker-images build trees by authoring minimal anchor-based patcher scripts (resolve module paths at runtime, refuse on non-matching source, idempotent --check/apply/--check) and wiring them into a Dockerfile. Use whenever changing engine behavior shipped in an image, or reviewing existing patches.
+---
+
+# vLLM/Engine Mod Patches
+
+This repo ships engine modifications as **patcher scripts** run at image build
+time, not as vendored rewrites of whole files. The rules below come from the
+DeepSeek-V4 DSpark / reasoning_effort work: a full-file replacement silently
+drops independent upstream updates; a targeted patch keeps everything the image
+ships and only modifies/adds exactly what is needed.
+
+## Where things live
+
+- Patch scripts: `docker-images/<dir>/patches/*.py`
+  (e.g. `deepseek-v4-flash-eugr/patches/`, `deepseek-v4-flash-dspark38/patches/`)
+- They are `COPY`ed into the image and run in a `RUN` step (see the image's
+  `Dockerfile`). Because they live under `docker-images/<dir>/`, they fold into
+  the `sm121.checksum` — a change triggers a rebuild.
+- Host-side helper scripts (not engine code) go in `host-scripts/` and are
+  embedded via `include_str!` in `src/builtin_scripts.rs`; referenced from
+  recipes as `{ script = "<alias>" }`.
+
+## Core conventions
+
+1. **Minimal, not wholesale.** Locate the exact region to change and target it
+   with a precise string replacement. Keep the file's own structure, imports,
+   and unrelated behavior intact. Never overwrite the whole file with a
+   "reference" variant — check whether the upstream reference is a superset
+   first (diff before you copy).
+2. **Resolve paths at runtime**, never hardcode. Different images install vLLM
+   in different places (system `dist-packages`, `/opt/env/lib/python3.12/...`).
+   Resolve from Python itself:
+   ```python
+   import vllm, os
+   BASE = os.path.join(vllm.__path__[0], "tokenizers")  # or "model_executor/..."
+   ```
+3. **Anchor-based apply that refuses on drift.** Every edit is an
+   `(old, new)` pair. `apply` requires the old anchor to appear **exactly
+   once**; if the source changed, `SystemExit` and the Docker build fails
+   instead of shipping a silently-broken image.
+4. **Idempotent `--check`.** It must pass both *before* apply (pristine) and
+   *after* apply (patched), so the Dockerfile can do
+   `--check → apply → --check`. Handle overlap: when `new` embeds `old` (e.g.
+   an inserted import block that starts with the original lines), count old
+   occurrences relative to how many sit inside `new`. A file is "consistent"
+   if it is pristine-old OR already-patched — not both, not neither.
+5. **Verify after patching.** Follow the apply with `python3 -m py_compile` of
+   the touched files and remove stray `__pycache__` dirs. Keep both patcher
+   steps in the same `RUN` so a compile failure fails the build.
+6. **Optional behavior = env-gated, default OFF.** For changes that are
+   experimental or a second-test variant, gate them on an env var checked at
+   module import (e.g. `_USE_X = bool(os.environ.get("VLLM_X", "").lower() in
+   ("1","true","yes","on"))`). Unset = byte-identical behavior to stock; opt in
+   from the recipe `[env]` / `-e`. Make the Dockerfile block its own step so it
+   can also be commented out to remove entirely.
+
+## Canonical patcher skeleton
+
+```python
+#!/usr/bin/env python3
+import os, sys
+import vllm
+
+FILE = os.path.join(vllm.__path__[0], "tokenizers/deepseek_v4_encoding.py")
+OLDER = "exact text currently in the image"
+NEWER = "exact replacement text"
+
+def _consistent(src, old, new):
+    co, cn = src.count(old), src.count(new)
+    if cn >= 1:                       # patched: new present once, and any
+        return cn == 1 and co <= new.count(old)  # old is only inside new
+    return co == 1                    # pristine: old present once
+
+def main():
+    check = "--check" in sys.argv
+    src = open(FILE).read()
+    if check:
+        if not _consistent(src, OLDER, NEWER):
+            raise SystemExit(f"CHECK FAIL: {FILE} is neither pristine nor patched")
+        print(f"check OK: {FILE} state is consistent"); return
+    if src.count(OLDER) != 1:
+        raise SystemExit(f"refusing: {FILE} old anchor matched "
+                         f"{src.count(OLDER)}x; re-sync patch")
+    out = src.replace(OLDER, NEWER)
+    assert out.count(NEWER) == 1
+    open(FILE, "w").write(out)
+    print(f"patched {FILE}")
+
+if __name__ == "__main__":
+    main()
+```
+
+## Dockerfile wiring
+
+```dockerfile
+COPY patches/patch_x.py /tmp/patch_x.py
+RUN set -eux; \
+    SRC="$(python3 -c 'import vllm, os; print(vllm.__path__[0])')"; \
+    python3 /tmp/patch_x.py --check; \
+    python3 /tmp/patch_x.py; \
+    python3 /tmp/patch_x.py --check; \
+    python3 -m py_compile "$SRC/tokenizers/deepseek_v4.py" "$SRC/tokenizers/deepseek_v4_encoding.py"; \
+    find "$SRC" -name '__pycache__' -type d -prune -exec rm -rf {} + 2>/dev/null || true; \
+    rm -f /tmp/patch_x.py
+```
+
+## Testing workflow (do this before shipping)
+
+1. `cp` the patcher + a test script to a node and run it in a **throwaway
+   container** of the base image so you never mutate a live/serving image:
+   ```bash
+   docker run --rm -v /tmp/p_x.py:/tmp/p_x.py -v /tmp/t.sh:/tmp/t.sh \
+     --entrypoint bash <image> /tmp/t.sh
+   ```
+   In the test script: pristine `--check` → apply → post `--check` →
+   `py_compile` → behavior probes (import the module prefix and assert the
+   flag/values) → re-run `--check` to confirm idempotency. Then start from a
+   fresh container for the next iteration (throwaway containers reset state).
+2. Only after the anchors match cleanly on the real image, wire the `COPY`+`RUN`
+   into the Dockerfile and do `cargo run -- -d serve --recipe <r> --cluster <c>`
+   (dry-run) then, with user sign-off, `sm121 serve` / `sm121 build`.
+3. To verify a rebuild took effect, `sm121 health`/`logs`; a `versions.env`,
+   `[build.versions]` or patch change alters the checksum and forces the build.
+
+## Pitfalls (learned the hard way)
+
+- **`main()` reading the wrong object**: unpack `path` from `TARGETS[0][0]` and
+  `open(path).read()` — don't destructure a `(path, [edits])` tuple straight
+  into `src` or you patch an empty list.
+- **Overlapping anchors**: if `new` is a prefix-superset of `old`, a naive
+  `old-or-new` check reports inconsistency after apply. Use the
+  `co <= new.count(old)` form above.
+- **Stale mounts / quoting**: when testing over SSH, confirm the mounted script
+  is the latest `scp`'d copy; avoid deep nested quotes — write a `.sh` test
+  script and run it, don't inline big heredocs through `ssh`.
+- **Two images → two layouts**: production-3.8 style vs eugr style differ in
+  path and in whether `b12x` is a separate module or integrated. Re-resolve
+  paths per image; never assume.
+- **Never patch a serving container.** Patches apply at image build. If you
+  need to test on the cluster, rebuild the image (checksum-aware) and adjust the
+  recipe, per the `sm121-launcher` skill — and remember: health first, confirm
+  not serving, before any `stop`/`power`.
+
+## References
+
+- Launcher operations, safety, build flow: `sm121-launcher` skill
+- Existing examples to model after: `docker-images/deepseek-v4-flash-eugr/patches/`
+  (minimal effort + env-gated reasoning_content), `deepseek-v4-flash-dspark38/`
+  (uniform padding, b12x routing), `glm52-nf3-hybrid/` (build-time patch script)
+- Recipe image/build semantics: `AGENTS.md`