skill: jovian-images (clone-at-pin builds, rev adoption, GB10 deltas, pitfall ledger)

.agents/skills/jovian-images/SKILL.md

@@ -0,0 +1,149 @@
+---
+name: jovian-images
+description: Build and adopt the jovian-judgement (GLM-5.3-Flash) DGX Spark serving images — clone-at-pin r27+ builds from the lab's source.lock contract, rev adoption procedure, GB10 adaptation deltas, gates, and the failure-mode ledger from actual build nights. Use whenever rebuilding glm53-* images, adopting a new lab rev (r28+), adding engine deps, or debugging the clone-at-pin build in docker-images/.
+---
+
+# Jovian Image Builds (clone-at-pin, source-locked)
+
+The lab qualifies GLM-5.3-Flash serving revs and publishes a **source
+contract**: `local-inference-lab/rtx6kpro` → `models/glm-5.3-flash.md`
+(human doc) + `models/glm-5.3-flash/build/<rev>.source.lock` (exact pins)
++ `build/Dockerfile` (their RTX PRO 6000 build — reference, not ours).
+Our builds run on DGX Spark GB10 (aarch64, sm_121a, unified memory) — same
+source locks, GB10-adapted runtime.
+
+## Layout
+
+- `docker-images/glm53-jovian-r27/` — the v2 contract build (**reproducible
+  from a fresh repo clone**; no context tarballs):
+  - `versions.env` — every pin: source SHAs + tree hashes + pretend
+    package versions + runtime versions. Single source of truth.
+  - `fetch-deps.sh` — clones every vLLM build sub-dependency at the pins
+    the r27 tree itself declares; grep-gates each pin against the tree's
+    CMake; populates vendored **gitlink trees** (cutlass inside
+    flash-attention / MSA / FlashMLA / FlashKDA / qutlass).
+  - `Dockerfile` — stages: apt → torch → deepgemm (in-build clone) →
+    vllm-clone (gated) → deps-fetched → vllm-builder → b12x-clone/builder →
+    lmcache-builder → runner (installs wheels + gates).
+  - `build.sh` — generic driver; forwards every versions.env KEY=VAL as a
+    build-arg. `TAG=glm53-jovian-r27:local bash build.sh`.
+- `docker-images/glm53-jovian-r12|r18/` — LEGACY staging-era builds
+  (tarball contexts; not rebuildable — the staged trees died with the
+  machine's /tmp). Harmless history; do not extend, do not rebuild from them.
+- Production recipes reference `docker_image = "glm53-jovian-r<N>:local"`
+  and `docker_image_dir = "glm53-jovian-r<N>"`.
+
+## Pin discipline (r27 contract, verified 2026-09-06)
+
+| pkg | pin | tree |
+|---|---|---|
+| vLLM | `voipmonitor/vllm` @ `63a82f8d…` (branch `integration/glm53-r27-release-20260906`) | `f54cd9ca…` |
+| B12X | `voipmonitor/b12x` @ `e8ad299b…` (same branch name) | `f3cd8a9e…` |
+| LMCache | `local-inference-lab/LMCache` @ `822330db…` | `008ac3e0…` |
+| FlashKDA | `vllm-project/FlashKDA` @ `3b225bf2…` | (gitlink in dep) |
+| cutlass (top) | `nvidia/cutlass` @ `v4.4.2` | — |
+| FlashMLA | `a8f794d1…` (→ vendored cutlass `147f5673…`) | — |
+| MSA | `087c1618…` (→ vendored cutlass `eb61c911…`) | — |
+| qutlass | `e74319e3…` (→ vendored cutlass `b2ca083d…`) | — |
+| tml-fa4 | `b2068346…` | — |
+| flash-attention | `f3e1a4f7…` (→ vendored cutlass `62750a2b…`) | — |
+| triton-kernels | `triton-lang/triton` @ `v3.5.1` | — |
+| DeepGEMM | `deepseek-ai/DeepGEMM` `nv_dev` @ `8b1392b9…` | — |
+
+**Branch names are NOT the contract** — the LMCache lock's branch was
+renamed upstream within a day. Pins are commit SHAs + tree hashes; branch
+names live in comments only. Wheels get deterministic versions through
+`SETUPTOOLS_SCM_PRETEND_VERSION` (engine string then identifies the pin:
+`0.1.dev0+glm53.r27.63a82f8d`).
+
+## Adopting a new lab rev (r28 →)
+
+1. Read the rev's `models/glm-5.3-flash.md` + `<rev>.source.lock` from
+   rtx6kpro master (fetch raw.githubusercontent; the lock lists all four
+   package trees + trees + runtime versions).
+2. Verify each pin is fetchable (`git ls-remote` for branch-anchored
+   SHAs, API commits probe for SHA-only pins; FlashKDA's hash-grep usually
+   already covered by the tree's own cmake).
+3. Update `versions.env` (PIN/TREE pairs + LMCACHE_VERSION from the lock).
+4. **Re-sync `fetch-deps.sh` dep table by reading the NEW tree's**
+   `cmake/external_projects/*.cmake` — revs add/move deps (r27 added
+   FlashKDA, LMCache, flashinfer tvm-ffi/tilelang pins). Update the gate
+   list with every GIT_REPOSITORY/GIT_TAG pair found.
+5. Rebuild on dgx4, boot on ds4 (see gates below), then bench+report.
+
+## Building / distributing
+
+- Build on dgx4 (buildkit cache mounts: uv-cache, ccache, cargo-registry,
+  rust-target). Context is tiny now (~215 KB): `tar czf` the dir + `scp`
+  + extract + `TAG=… bash build.sh` under `setsid nohup … &`, log to
+  /tmp/r27build<N>.log.
+- Alternatively `sm121 build --recipe <lane> --build-node` — fine for THIS
+  size (chunks at 60 KB/ssh-trip); the r18-era 417 MB dir was what made it
+  crawl — with clone-at-pin, sm121 build also stamps `sm121.checksum`
+  (manually built images stay unlabeled → informational STALE warning).
+- Distribute: `docker save <img> | ssh dst docker load` (the
+  `docker save -o -` variant is broken on this stack). Run transfers with
+  `setsid nohup bash -c '...' < /dev/null > log 2>&1 &` (128 MB/s node-to-
+  node; 16.6 GB ≈ 2 min).
+
+## Boot gates (already in the Dockerfile runner; re-check after serve)
+
+Imports: `vllm` (string carries the pin), `b12x`, `lmcache` (+
+`lmcache.cuda_ops`/`lmcache_native`/`lmcache_fs`), `vllm._flashkda_C`,
+B12X enum resolves to the b12x backend, `nvfp4_ds_mla` present in
+`vllm/config/cache.py`. Serving: expect the KV-size log line
+(`GPU KV cache size: N tokens`) ~10 min in, application startup ~5 min
+after — **when JIT mounts are warm** (~9 min total; ~45 min cold/first).
+
+## GB10 adaptation deltas (RTX lock → Spark)
+
+- Runtime: CUDA 13.2 aarch64 / torch 2.13.0+cu130 / flashinfer 0.6.17 /
+  torch-bundled NCCL — all matches the TREE's own requirements
+  (`requirements/cuda.txt`); their RTX runtime (13.3/0.6.18/NCCL 2.31.2)
+  is a documented deviation, re-evaluate only when the tree pins change.
+- `TORCH_CUDA_ARCH_LIST=12.1a` (incl. the LMCache wheel build — theirs=12.0).
+- `patch_cuda_sm12x.py` + eugr patch set, all idempotent and run in full:
+  `preserve_sm12x_target` (12.1a collapses to 12.0 without it),
+  `sm120_cooperative_topk` (kept SM90-only), `spark_kv_cache_cleanup`
+  (UMA/MemFree — see note below), b12x c128a alignment, flashinfer
+  swigluoai, speculator cudagraph pool, routed-expert weight shape.
+- InstantTensor loader is fork-builtin (no separate pip package needed).
+- env for serving comes from recipes; their launcher envs (SPECULATOR/…)
+  are RTX launcher abstractions — translate the equivalent into recipe
+  args/env when needed.
+
+## Pitfall ledger (scar tissue — do not re-learn)
+
+- **Gitlink = empty dir**: a `git clone` materializes vendored gitlink
+  paths as empty dirs; a `[ -d ]` check is a TRAP (saw this twice: r18's
+  `deps-src.tar.gz` stubs → cmake INSTALL death; r27's FlashMLA
+  `csrc/cutlass` → compile death). Gate on `.git` presence inside the
+  path (fetch-deps.sh `populate_gitlinks` does this generically; AMD-only
+  trees composable_kernel/aiter are skipped intentionally).
+- **Top-level cutlass is pinned too** (`VLLM_CUTLASS_SRC_DIR=/deps2/cutlass`
+  + `CUTLASS_INCLUDE_DIR_PRE`) — keep it at the tree's declared tag (v4.4.2),
+  NOT the dep-gitlink ones.
+- macOS staging poisons contexts: 23k AppleDouble `._*` files + provenance
+  xattrs in the r18 deps tarball. With clone-at-pin this cannot recur; if
+  you ever build a context on macOS, use `COPYFILE_DISABLE=1 tar
+  --no-xattrs` and `find . -name '._*' -delete`.
+- Preferences/danger: `rg -rn` is a trap (`-r` = replace; use `rg -n`);
+  `rg --include` does not exist (`-g '*.py'`); node zsh aborts on unmatched
+  globs; the sm121 launcher's chunked uploader streams 60 KB per ssh round
+  trip (fine for 215 KB contexts, hopeless for 400 MB dirs).
+- Engine quirks learned by bench nights (see
+  `specs/tech-architecture/glm53-recipe-state.md` + `history/night-shifts/`):
+  Spark-checkpoint MXFP8 MTP experts are not TP4-tileable (`mm_mxfp8`),
+  B12X rejects drafter-config KV dtypes on SM121 (FA fp8 needs FA3/FA4),
+  2x-lane memory ceiling ≈ 10 GiB KV pin (4x NVFP4 has room for 16 GiB),
+  JIT caches persist per node under `~/sm121-cache/jit-121f/*`.
+
+## Bench conventions (tool-eval-bench)
+
+- Tool: `2.6.1.dev45+gcf54b4bfe` (uv tool), `--hardmode` = 88 scenarios
+  (69 standard + 19 Hard Mode), `--perf` = llama-benchy pp2048/tg128 c1+c2.
+- Gate: quality score is the primary gate; seeds fixed (42); the canonical
+  anchor + per-lane baseline numbers live in
+  `specs/tech-architecture/glm53-recipe-state.md`.
+- Reports land in `recipes/<lane>/reports/<UTC-stamp>_<label>.md` and the
+  combined score+perf tables go to `history/night-shifts/<date>.md`.