launcher/.agents/skills/vllm-mod-patches/SKILL.md

---

name: vllm-mod-patches

description: Modify VLLM/engine code inside the docker-images build trees by authoring minimal anchor-based patcher scripts (resolve module paths at runtime, refuse on non-matching source, idempotent --check/apply/--check) and wiring them into a Dockerfile. Use whenever changing engine behavior shipped in an image, or reviewing existing patches.

vLLM/Engine Mod Patches

This repo ships engine modifications as patcher scripts run at image build time, not as vendored rewrites of whole files. The rules below come from the DeepSeek-V4 DSpark / reasoning_effort work: a full-file replacement silently drops independent upstream updates; a targeted patch keeps everything the image ships and only modifies/adds exactly what is needed.

Where things live

• Patch scripts: docker-images/<dir>/patches/*.py (e.g. deepseek-v4-flash-eugr/patches/, deepseek-v4-flash-dspark38/patches/)
• They are COPYed into the image and run in a RUN step (see the image's Dockerfile). Because they live under docker-images/<dir>/, they fold into the sm121.checksum — a change triggers a rebuild.
• Host-side helper scripts (not engine code) go in host-scripts/ and are embedded via include_str! in src/builtin_scripts.rs; referenced from recipes as { script = "<alias>" }.

Core conventions

1. Minimal, not wholesale. Locate the exact region to change and target it with a precise string replacement. Keep the file's own structure, imports, and unrelated behavior intact. Never overwrite the whole file with a "reference" variant — check whether the upstream reference is a superset first (diff before you copy).

2. Resolve paths at runtime, never hardcode. Different images install vLLM in different places (system dist-packages, /opt/env/lib/python3.12/...). Resolve from Python itself:

   import vllm, os
   BASE = os.path.join(vllm.__path__[0], "tokenizers")  # or "model_executor/..."
   

3. Anchor-based apply that refuses on drift. Every edit is an (old, new) pair. apply requires the old anchor to appear exactly once; if the source changed, SystemExit and the Docker build fails instead of shipping a silently-broken image.

4. Idempotent --check. It must pass both before apply (pristine) and after apply (patched), so the Dockerfile can do --check → apply → --check. Handle overlap: when new embeds old (e.g. an inserted import block that starts with the original lines), count old occurrences relative to how many sit inside new. A file is "consistent" if it is pristine-old OR already-patched — not both, not neither.

5. Verify after patching. Follow the apply with python3 -m py_compile of the touched files and remove stray __pycache__ dirs. Keep both patcher steps in the same RUN so a compile failure fails the build.

6. Optional behavior = env-gated, default OFF. For changes that are experimental or a second-test variant, gate them on an env var checked at module import (e.g. _USE_X = bool(os.environ.get("VLLM_X", "").lower() in ("1","true","yes","on"))). Unset = byte-identical behavior to stock; opt in from the recipe [env] / -e. Make the Dockerfile block its own step so it can also be commented out to remove entirely.

Canonical patcher skeleton

#!/usr/bin/env python3
import os, sys
import vllm

FILE = os.path.join(vllm.__path__[0], "tokenizers/deepseek_v4_encoding.py")
OLDER = "exact text currently in the image"
NEWER = "exact replacement text"

def _consistent(src, old, new):
    co, cn = src.count(old), src.count(new)
    if cn >= 1:                       # patched: new present once, and any
        return cn == 1 and co <= new.count(old)  # old is only inside new
    return co == 1                    # pristine: old present once

def main():
    check = "--check" in sys.argv
    src = open(FILE).read()
    if check:
        if not _consistent(src, OLDER, NEWER):
            raise SystemExit(f"CHECK FAIL: {FILE} is neither pristine nor patched")
        print(f"check OK: {FILE} state is consistent"); return
    if src.count(OLDER) != 1:
        raise SystemExit(f"refusing: {FILE} old anchor matched "
                         f"{src.count(OLDER)}x; re-sync patch")
    out = src.replace(OLDER, NEWER)
    assert out.count(NEWER) == 1
    open(FILE, "w").write(out)
    print(f"patched {FILE}")

if __name__ == "__main__":
    main()

Dockerfile wiring

COPY patches/patch_x.py /tmp/patch_x.py
RUN set -eux; \
    SRC="$(python3 -c 'import vllm, os; print(vllm.__path__[0])')"; \
    python3 /tmp/patch_x.py --check; \
    python3 /tmp/patch_x.py; \
    python3 /tmp/patch_x.py --check; \
    python3 -m py_compile "$SRC/tokenizers/deepseek_v4.py" "$SRC/tokenizers/deepseek_v4_encoding.py"; \
    find "$SRC" -name '__pycache__' -type d -prune -exec rm -rf {} + 2>/dev/null || true; \
    rm -f /tmp/patch_x.py

Testing workflow (do this before shipping)

1. cp the patcher + a test script to a node and run it in a throwaway container of the base image so you never mutate a live/serving image:

   docker run --rm -v /tmp/p_x.py:/tmp/p_x.py -v /tmp/t.sh:/tmp/t.sh \
    --entrypoint bash <image> /tmp/t.sh
   

   In the test script: pristine --check → apply → post --check → py_compile → behavior probes (import the module prefix and assert the flag/values) → re-run --check to confirm idempotency. Then start from a fresh container for the next iteration (throwaway containers reset state).

2. Only after the anchors match cleanly on the real image, wire the COPY+RUN into the Dockerfile and do cargo run -- -d serve --recipe <r> --cluster <c> (dry-run) then, with user sign-off, sm121 serve / sm121 build.

3. To verify a rebuild took effect, sm121 health/logs; a versions.env, [build.versions] or patch change alters the checksum and forces the build.

Pitfalls (learned the hard way)

• main() reading the wrong object: unpack path from TARGETS[0][0] and open(path).read() — don't destructure a (path, [edits]) tuple straight into src or you patch an empty list.
• Overlapping anchors: if new is a prefix-superset of old, a naive old-or-new check reports inconsistency after apply. Use the co <= new.count(old) form above.
• Stale mounts / quoting: when testing over SSH, confirm the mounted script is the latest scp'd copy; avoid deep nested quotes — write a .sh test script and run it, don't inline big heredocs through ssh.
• Two images → two layouts: production-3.8 style vs eugr style differ in path and in whether b12x is a separate module or integrated. Re-resolve paths per image; never assume.
• Never patch a serving container. Patches apply at image build. If you need to test on the cluster, rebuild the image (checksum-aware) and adjust the recipe, per the sm121-launcher skill — and remember: health first, confirm not serving, before any stop/power.

References

• Launcher operations, safety, build flow: sm121-launcher skill
• Existing examples to model after: docker-images/deepseek-v4-flash-eugr/patches/ (minimal effort + env-gated reasoning_content), deepseek-v4-flash-dspark38/ (uniform padding, b12x routing), glm52-nf3-hybrid/ (build-time patch script)
• Recipe image/build semantics: AGENTS.md