|
|
@@ -0,0 +1,152 @@
|
|
|
+---
|
|
|
+name: vllm-mod-patches
|
|
|
+description: Modify VLLM/engine code inside the docker-images build trees by authoring minimal anchor-based patcher scripts (resolve module paths at runtime, refuse on non-matching source, idempotent --check/apply/--check) and wiring them into a Dockerfile. Use whenever changing engine behavior shipped in an image, or reviewing existing patches.
|
|
|
+---
|
|
|
+
|
|
|
+# vLLM/Engine Mod Patches
|
|
|
+
|
|
|
+This repo ships engine modifications as **patcher scripts** run at image build
|
|
|
+time, not as vendored rewrites of whole files. The rules below come from the
|
|
|
+DeepSeek-V4 DSpark / reasoning_effort work: a full-file replacement silently
|
|
|
+drops independent upstream updates; a targeted patch keeps everything the image
|
|
|
+ships and only modifies/adds exactly what is needed.
|
|
|
+
|
|
|
+## Where things live
|
|
|
+
|
|
|
+- Patch scripts: `docker-images/<dir>/patches/*.py`
|
|
|
+ (e.g. `deepseek-v4-flash-eugr/patches/`, `deepseek-v4-flash-dspark38/patches/`)
|
|
|
+- They are `COPY`ed into the image and run in a `RUN` step (see the image's
|
|
|
+ `Dockerfile`). Because they live under `docker-images/<dir>/`, they fold into
|
|
|
+ the `sm121.checksum` — a change triggers a rebuild.
|
|
|
+- Host-side helper scripts (not engine code) go in `host-scripts/` and are
|
|
|
+ embedded via `include_str!` in `src/builtin_scripts.rs`; referenced from
|
|
|
+ recipes as `{ script = "<alias>" }`.
|
|
|
+
|
|
|
+## Core conventions
|
|
|
+
|
|
|
+1. **Minimal, not wholesale.** Locate the exact region to change and target it
|
|
|
+ with a precise string replacement. Keep the file's own structure, imports,
|
|
|
+ and unrelated behavior intact. Never overwrite the whole file with a
|
|
|
+ "reference" variant — check whether the upstream reference is a superset
|
|
|
+ first (diff before you copy).
|
|
|
+2. **Resolve paths at runtime**, never hardcode. Different images install vLLM
|
|
|
+ in different places (system `dist-packages`, `/opt/env/lib/python3.12/...`).
|
|
|
+ Resolve from Python itself:
|
|
|
+ ```python
|
|
|
+ import vllm, os
|
|
|
+ BASE = os.path.join(vllm.__path__[0], "tokenizers") # or "model_executor/..."
|
|
|
+ ```
|
|
|
+3. **Anchor-based apply that refuses on drift.** Every edit is an
|
|
|
+ `(old, new)` pair. `apply` requires the old anchor to appear **exactly
|
|
|
+ once**; if the source changed, `SystemExit` and the Docker build fails
|
|
|
+ instead of shipping a silently-broken image.
|
|
|
+4. **Idempotent `--check`.** It must pass both *before* apply (pristine) and
|
|
|
+ *after* apply (patched), so the Dockerfile can do
|
|
|
+ `--check → apply → --check`. Handle overlap: when `new` embeds `old` (e.g.
|
|
|
+ an inserted import block that starts with the original lines), count old
|
|
|
+ occurrences relative to how many sit inside `new`. A file is "consistent"
|
|
|
+ if it is pristine-old OR already-patched — not both, not neither.
|
|
|
+5. **Verify after patching.** Follow the apply with `python3 -m py_compile` of
|
|
|
+ the touched files and remove stray `__pycache__` dirs. Keep both patcher
|
|
|
+ steps in the same `RUN` so a compile failure fails the build.
|
|
|
+6. **Optional behavior = env-gated, default OFF.** For changes that are
|
|
|
+ experimental or a second-test variant, gate them on an env var checked at
|
|
|
+ module import (e.g. `_USE_X = bool(os.environ.get("VLLM_X", "").lower() in
|
|
|
+ ("1","true","yes","on"))`). Unset = byte-identical behavior to stock; opt in
|
|
|
+ from the recipe `[env]` / `-e`. Make the Dockerfile block its own step so it
|
|
|
+ can also be commented out to remove entirely.
|
|
|
+
|
|
|
+## Canonical patcher skeleton
|
|
|
+
|
|
|
+```python
|
|
|
+#!/usr/bin/env python3
|
|
|
+import os, sys
|
|
|
+import vllm
|
|
|
+
|
|
|
+FILE = os.path.join(vllm.__path__[0], "tokenizers/deepseek_v4_encoding.py")
|
|
|
+OLDER = "exact text currently in the image"
|
|
|
+NEWER = "exact replacement text"
|
|
|
+
|
|
|
+def _consistent(src, old, new):
|
|
|
+ co, cn = src.count(old), src.count(new)
|
|
|
+ if cn >= 1: # patched: new present once, and any
|
|
|
+ return cn == 1 and co <= new.count(old) # old is only inside new
|
|
|
+ return co == 1 # pristine: old present once
|
|
|
+
|
|
|
+def main():
|
|
|
+ check = "--check" in sys.argv
|
|
|
+ src = open(FILE).read()
|
|
|
+ if check:
|
|
|
+ if not _consistent(src, OLDER, NEWER):
|
|
|
+ raise SystemExit(f"CHECK FAIL: {FILE} is neither pristine nor patched")
|
|
|
+ print(f"check OK: {FILE} state is consistent"); return
|
|
|
+ if src.count(OLDER) != 1:
|
|
|
+ raise SystemExit(f"refusing: {FILE} old anchor matched "
|
|
|
+ f"{src.count(OLDER)}x; re-sync patch")
|
|
|
+ out = src.replace(OLDER, NEWER)
|
|
|
+ assert out.count(NEWER) == 1
|
|
|
+ open(FILE, "w").write(out)
|
|
|
+ print(f"patched {FILE}")
|
|
|
+
|
|
|
+if __name__ == "__main__":
|
|
|
+ main()
|
|
|
+```
|
|
|
+
|
|
|
+## Dockerfile wiring
|
|
|
+
|
|
|
+```dockerfile
|
|
|
+COPY patches/patch_x.py /tmp/patch_x.py
|
|
|
+RUN set -eux; \
|
|
|
+ SRC="$(python3 -c 'import vllm, os; print(vllm.__path__[0])')"; \
|
|
|
+ python3 /tmp/patch_x.py --check; \
|
|
|
+ python3 /tmp/patch_x.py; \
|
|
|
+ python3 /tmp/patch_x.py --check; \
|
|
|
+ python3 -m py_compile "$SRC/tokenizers/deepseek_v4.py" "$SRC/tokenizers/deepseek_v4_encoding.py"; \
|
|
|
+ find "$SRC" -name '__pycache__' -type d -prune -exec rm -rf {} + 2>/dev/null || true; \
|
|
|
+ rm -f /tmp/patch_x.py
|
|
|
+```
|
|
|
+
|
|
|
+## Testing workflow (do this before shipping)
|
|
|
+
|
|
|
+1. `cp` the patcher + a test script to a node and run it in a **throwaway
|
|
|
+ container** of the base image so you never mutate a live/serving image:
|
|
|
+ ```bash
|
|
|
+ docker run --rm -v /tmp/p_x.py:/tmp/p_x.py -v /tmp/t.sh:/tmp/t.sh \
|
|
|
+ --entrypoint bash <image> /tmp/t.sh
|
|
|
+ ```
|
|
|
+ In the test script: pristine `--check` → apply → post `--check` →
|
|
|
+ `py_compile` → behavior probes (import the module prefix and assert the
|
|
|
+ flag/values) → re-run `--check` to confirm idempotency. Then start from a
|
|
|
+ fresh container for the next iteration (throwaway containers reset state).
|
|
|
+2. Only after the anchors match cleanly on the real image, wire the `COPY`+`RUN`
|
|
|
+ into the Dockerfile and do `cargo run -- -d serve --recipe <r> --cluster <c>`
|
|
|
+ (dry-run) then, with user sign-off, `sm121 serve` / `sm121 build`.
|
|
|
+3. To verify a rebuild took effect, `sm121 health`/`logs`; a `versions.env`,
|
|
|
+ `[build.versions]` or patch change alters the checksum and forces the build.
|
|
|
+
|
|
|
+## Pitfalls (learned the hard way)
|
|
|
+
|
|
|
+- **`main()` reading the wrong object**: unpack `path` from `TARGETS[0][0]` and
|
|
|
+ `open(path).read()` — don't destructure a `(path, [edits])` tuple straight
|
|
|
+ into `src` or you patch an empty list.
|
|
|
+- **Overlapping anchors**: if `new` is a prefix-superset of `old`, a naive
|
|
|
+ `old-or-new` check reports inconsistency after apply. Use the
|
|
|
+ `co <= new.count(old)` form above.
|
|
|
+- **Stale mounts / quoting**: when testing over SSH, confirm the mounted script
|
|
|
+ is the latest `scp`'d copy; avoid deep nested quotes — write a `.sh` test
|
|
|
+ script and run it, don't inline big heredocs through `ssh`.
|
|
|
+- **Two images → two layouts**: production-3.8 style vs eugr style differ in
|
|
|
+ path and in whether `b12x` is a separate module or integrated. Re-resolve
|
|
|
+ paths per image; never assume.
|
|
|
+- **Never patch a serving container.** Patches apply at image build. If you
|
|
|
+ need to test on the cluster, rebuild the image (checksum-aware) and adjust the
|
|
|
+ recipe, per the `sm121-launcher` skill — and remember: health first, confirm
|
|
|
+ not serving, before any `stop`/`power`.
|
|
|
+
|
|
|
+## References
|
|
|
+
|
|
|
+- Launcher operations, safety, build flow: `sm121-launcher` skill
|
|
|
+- Existing examples to model after: `docker-images/deepseek-v4-flash-eugr/patches/`
|
|
|
+ (minimal effort + env-gated reasoning_content), `deepseek-v4-flash-dspark38/`
|
|
|
+ (uniform padding, b12x routing), `glm52-nf3-hybrid/` (build-time patch script)
|
|
|
+- Recipe image/build semantics: `AGENTS.md`
|
