---
name: vllm-mod-patches
description: Modify VLLM/engine code inside the docker-images build trees by authoring minimal anchor-based patcher scripts (resolve module paths at runtime, refuse on non-matching source, idempotent --check/apply/--check) and wiring them into a Dockerfile. Use whenever changing engine behavior shipped in an image, or reviewing existing patches.
---

# vLLM/Engine Mod Patches

This repo ships engine modifications as **patcher scripts** run at image build
time, not as vendored rewrites of whole files. The rules below come from the
DeepSeek-V4 DSpark / reasoning_effort work: a full-file replacement silently
drops independent upstream updates; a targeted patch keeps everything the image
ships and only modifies/adds exactly what is needed.

## Where things live

- Patch scripts: `docker-images/<dir>/patches/*.py`
  (e.g. `deepseek-v4-flash-eugr/patches/`, `deepseek-v4-flash-dspark38/patches/`)
- They are `COPY`ed into the image and run in a `RUN` step (see the image's
  `Dockerfile`). Because they live under `docker-images/<dir>/`, they fold into
  the `sm121.checksum` — a change triggers a rebuild.
- Host-side helper scripts (not engine code) go in `host-scripts/` and are
  embedded via `include_str!` in `src/builtin_scripts.rs`; referenced from
  recipes as `{ script = "<alias>" }`.

## Core conventions

1. **Minimal, not wholesale.** Locate the exact region to change and target it
   with a precise string replacement. Keep the file's own structure, imports,
   and unrelated behavior intact. Never overwrite the whole file with a
   "reference" variant — check whether the upstream reference is a superset
   first (diff before you copy).
2. **Resolve paths at runtime**, never hardcode. Different images install vLLM
   in different places (system `dist-packages`, `/opt/env/lib/python3.12/...`).
   Resolve from Python itself:
   ```python
   import vllm, os
   BASE = os.path.join(vllm.__path__[0], "tokenizers")  # or "model_executor/..."
   ```
3. **Anchor-based apply that refuses on drift.** Every edit is an
   `(old, new)` pair. `apply` requires the old anchor to appear **exactly
   once**; if the source changed, `SystemExit` and the Docker build fails
   instead of shipping a silently-broken image.
4. **Idempotent `--check`.** It must pass both *before* apply (pristine) and
   *after* apply (patched), so the Dockerfile can do
   `--check → apply → --check`. Handle overlap: when `new` embeds `old` (e.g.
   an inserted import block that starts with the original lines), count old
   occurrences relative to how many sit inside `new`. A file is "consistent"
   if it is pristine-old OR already-patched — not both, not neither.
5. **Verify after patching.** Follow the apply with `python3 -m py_compile` of
   the touched files and remove stray `__pycache__` dirs. Keep both patcher
   steps in the same `RUN` so a compile failure fails the build.
6. **Optional behavior = env-gated, default OFF.** For changes that are
   experimental or a second-test variant, gate them on an env var checked at
   module import (e.g. `_USE_X = bool(os.environ.get("VLLM_X", "").lower() in
   ("1","true","yes","on"))`). Unset = byte-identical behavior to stock; opt in
   from the recipe `[env]` / `-e`. Make the Dockerfile block its own step so it
   can also be commented out to remove entirely.

## Canonical patcher skeleton

```python
#!/usr/bin/env python3
import os, sys
import vllm

FILE = os.path.join(vllm.__path__[0], "tokenizers/deepseek_v4_encoding.py")
OLDER = "exact text currently in the image"
NEWER = "exact replacement text"

def _consistent(src, old, new):
    co, cn = src.count(old), src.count(new)
    if cn >= 1:                       # patched: new present once, and any
        return cn == 1 and co <= new.count(old)  # old is only inside new
    return co == 1                    # pristine: old present once

def main():
    check = "--check" in sys.argv
    src = open(FILE).read()
    if check:
        if not _consistent(src, OLDER, NEWER):
            raise SystemExit(f"CHECK FAIL: {FILE} is neither pristine nor patched")
        print(f"check OK: {FILE} state is consistent"); return
    if src.count(OLDER) != 1:
        raise SystemExit(f"refusing: {FILE} old anchor matched "
                         f"{src.count(OLDER)}x; re-sync patch")
    out = src.replace(OLDER, NEWER)
    assert out.count(NEWER) == 1
    open(FILE, "w").write(out)
    print(f"patched {FILE}")

if __name__ == "__main__":
    main()
```

## Dockerfile wiring

```dockerfile
COPY patches/patch_x.py /tmp/patch_x.py
RUN set -eux; \
    SRC="$(python3 -c 'import vllm, os; print(vllm.__path__[0])')"; \
    python3 /tmp/patch_x.py --check; \
    python3 /tmp/patch_x.py; \
    python3 /tmp/patch_x.py --check; \
    python3 -m py_compile "$SRC/tokenizers/deepseek_v4.py" "$SRC/tokenizers/deepseek_v4_encoding.py"; \
    find "$SRC" -name '__pycache__' -type d -prune -exec rm -rf {} + 2>/dev/null || true; \
    rm -f /tmp/patch_x.py
```

## Testing workflow (do this before shipping)

1. `cp` the patcher + a test script to a node and run it in a **throwaway
   container** of the base image so you never mutate a live/serving image:
   ```bash
   docker run --rm -v /tmp/p_x.py:/tmp/p_x.py -v /tmp/t.sh:/tmp/t.sh \
     --entrypoint bash <image> /tmp/t.sh
   ```
   In the test script: pristine `--check` → apply → post `--check` →
   `py_compile` → behavior probes (import the module prefix and assert the
   flag/values) → re-run `--check` to confirm idempotency. Then start from a
   fresh container for the next iteration (throwaway containers reset state).
2. Only after the anchors match cleanly on the real image, wire the `COPY`+`RUN`
   into the Dockerfile and do `cargo run -- -d serve --recipe <r> --cluster <c>`
   (dry-run) then, with user sign-off, `sm121 serve` / `sm121 build`.
3. To verify a rebuild took effect, `sm121 health`/`logs`; a `versions.env`,
   `[build.versions]` or patch change alters the checksum and forces the build.

## Pitfalls (learned the hard way)

- **`main()` reading the wrong object**: unpack `path` from `TARGETS[0][0]` and
  `open(path).read()` — don't destructure a `(path, [edits])` tuple straight
  into `src` or you patch an empty list.
- **Overlapping anchors**: if `new` is a prefix-superset of `old`, a naive
  `old-or-new` check reports inconsistency after apply. Use the
  `co <= new.count(old)` form above.
- **Stale mounts / quoting**: when testing over SSH, confirm the mounted script
  is the latest `scp`'d copy; avoid deep nested quotes — write a `.sh` test
  script and run it, don't inline big heredocs through `ssh`.
- **Two images → two layouts**: production-3.8 style vs eugr style differ in
  path and in whether `b12x` is a separate module or integrated. Re-resolve
  paths per image; never assume.
- **Never patch a serving container.** Patches apply at image build. If you
  need to test on the cluster, rebuild the image (checksum-aware) and adjust the
  recipe, per the `sm121-launcher` skill — and remember: health first, confirm
  not serving, before any `stop`/`power`.

## References

- Launcher operations, safety, build flow: `sm121-launcher` skill
- Existing examples to model after: `docker-images/deepseek-v4-flash-eugr/patches/`
  (minimal effort + env-gated reasoning_content), `deepseek-v4-flash-dspark38/`
  (uniform padding, b12x routing), `glm52-nf3-hybrid/` (build-time patch script)
- Recipe image/build semantics: `AGENTS.md`