--- name: jovian-images description: Build and adopt the jovian-judgement (GLM-5.3-Flash) DGX Spark serving images — clone-at-pin r27+ builds from the lab's source.lock contract, rev adoption procedure, GB10 adaptation deltas, gates, and the failure-mode ledger from actual build nights. Use whenever rebuilding glm53-* images, adopting a new lab rev (r28+), adding engine deps, or debugging the clone-at-pin build in docker-images/. --- # Jovian Image Builds (clone-at-pin, source-locked) The lab qualifies GLM-5.3-Flash serving revs and publishes a **source contract**: `local-inference-lab/rtx6kpro` → `models/glm-5.3-flash.md` (human doc) + `models/glm-5.3-flash/build/.source.lock` (exact pins) + `build/Dockerfile` (their RTX PRO 6000 build — reference, not ours). Our builds run on DGX Spark GB10 (aarch64, sm_121a, unified memory) — same source locks, GB10-adapted runtime. ## Layout - `docker-images/glm53-jovian-r27/` — the v2 contract build (**reproducible from a fresh repo clone**; no context tarballs): - `versions.env` — every pin: source SHAs + tree hashes + pretend package versions + runtime versions. Single source of truth. - `fetch-deps.sh` — clones every vLLM build sub-dependency at the pins the r27 tree itself declares; grep-gates each pin against the tree's CMake; populates vendored **gitlink trees** (cutlass inside flash-attention / MSA / FlashMLA / FlashKDA / qutlass). - `Dockerfile` — stages: apt → torch → deepgemm (in-build clone) → vllm-clone (gated) → deps-fetched → vllm-builder → b12x-clone/builder → lmcache-builder → runner (installs wheels + gates). - `build.sh` — generic driver; forwards every versions.env KEY=VAL as a build-arg. `TAG=glm53-jovian-r27:local bash build.sh`. - Legacy r12/r18 staging-era build dirs were REMOVED from the repo (2026-09-07): their tarball contexts were not rebuildable anyway. The r18 image (`glm53-jovian-r18:local`) lives on the nodes and the recipes keep referencing it; `docker_image_dir` for those recipes is now informational only (no build dir). The r27 dir is the current build lineage. - Production recipes reference `docker_image = "glm53-jovian-r:local"` and `docker_image_dir = "glm53-jovian-r"` (r27 = the live build dir). ## Pin discipline (r27 contract, verified 2026-09-06) | pkg | pin | tree | |---|---|---| | vLLM | `voipmonitor/vllm` @ `63a82f8d…` (branch `integration/glm53-r27-release-20260906`) | `f54cd9ca…` | | B12X | `voipmonitor/b12x` @ `e8ad299b…` (same branch name) | `f3cd8a9e…` | | LMCache | `local-inference-lab/LMCache` @ `822330db…` | `008ac3e0…` | | FlashKDA | `vllm-project/FlashKDA` @ `3b225bf2…` | (gitlink in dep) | | cutlass (top) | `nvidia/cutlass` @ `v4.4.2` | — | | FlashMLA | `a8f794d1…` (→ vendored cutlass `147f5673…`) | — | | MSA | `087c1618…` (→ vendored cutlass `eb61c911…`) | — | | qutlass | `e74319e3…` (→ vendored cutlass `b2ca083d…`) | — | | tml-fa4 | `b2068346…` | — | | flash-attention | `f3e1a4f7…` (→ vendored cutlass `62750a2b…`) | — | | triton-kernels | `triton-lang/triton` @ `v3.5.1` | — | | DeepGEMM | `deepseek-ai/DeepGEMM` `nv_dev` @ `8b1392b9…` | — | **Branch names are NOT the contract** — the LMCache lock's branch was renamed upstream within a day. Pins are commit SHAs + tree hashes; branch names live in comments only. Wheels get deterministic versions through `SETUPTOOLS_SCM_PRETEND_VERSION` (engine string then identifies the pin: `0.1.dev0+glm53.r27.63a82f8d`). ## Adopting a new lab rev (r28 →) 1. Read the rev's `models/glm-5.3-flash.md` + `.source.lock` from rtx6kpro master (fetch raw.githubusercontent; the lock lists all four package trees + trees + runtime versions). 2. Verify each pin is fetchable (`git ls-remote` for branch-anchored SHAs, API commits probe for SHA-only pins; FlashKDA's hash-grep usually already covered by the tree's own cmake). 3. Update `versions.env` (PIN/TREE pairs + LMCACHE_VERSION from the lock). 4. **Re-sync `fetch-deps.sh` dep table by reading the NEW tree's** `cmake/external_projects/*.cmake` — revs add/move deps (r27 added FlashKDA, LMCache, flashinfer tvm-ffi/tilelang pins). Update the gate list with every GIT_REPOSITORY/GIT_TAG pair found. 5. Rebuild on dgx4, boot on ds4 (see gates below), then bench+report. ## Building / distributing - Build on dgx4 (buildkit cache mounts: uv-cache, ccache, cargo-registry, rust-target). Context is tiny now (~215 KB): `tar czf` the dir + `scp` + extract + `TAG=… bash build.sh` under `setsid nohup … &`, log to /tmp/r27build.log. - Alternatively `sm121 build --recipe --build-node` — fine for THIS size (chunks at 60 KB/ssh-trip); the r18-era 417 MB dir was what made it crawl — with clone-at-pin, sm121 build also stamps `sm121.checksum` (manually built images stay unlabeled → informational STALE warning). - Distribute: `docker save | ssh dst docker load` (the `docker save -o -` variant is broken on this stack). Run transfers with `setsid nohup bash -c '...' < /dev/null > log 2>&1 &` (128 MB/s node-to- node; 16.6 GB ≈ 2 min). ## Boot gates (already in the Dockerfile runner; re-check after serve) Imports: `vllm` (string carries the pin), `b12x`, `lmcache` (+ `lmcache.cuda_ops`/`lmcache_native`/`lmcache_fs`), `vllm._flashkda_C`, B12X enum resolves to the b12x backend, `nvfp4_ds_mla` present in `vllm/config/cache.py`. Serving: expect the KV-size log line (`GPU KV cache size: N tokens`) ~10 min in, application startup ~5 min after — **when JIT mounts are warm** (~9 min total; ~45 min cold/first). ## GB10 adaptation deltas (RTX lock → Spark) - Runtime: CUDA 13.2 aarch64 / torch 2.13.0+cu130 / flashinfer 0.6.17 / torch-bundled NCCL — all matches the TREE's own requirements (`requirements/cuda.txt`); their RTX runtime (13.3/0.6.18/NCCL 2.31.2) is a documented deviation, re-evaluate only when the tree pins change. - `TORCH_CUDA_ARCH_LIST=12.1a` (incl. the LMCache wheel build — theirs=12.0). - `patch_cuda_sm12x.py` + eugr patch set, all idempotent and run in full: `preserve_sm12x_target` (12.1a collapses to 12.0 without it), `sm120_cooperative_topk` (kept SM90-only), `spark_kv_cache_cleanup` (UMA/MemFree — see note below), b12x c128a alignment, flashinfer swigluoai, speculator cudagraph pool, routed-expert weight shape. - InstantTensor loader is fork-builtin (no separate pip package needed). - env for serving comes from recipes; their launcher envs (SPECULATOR/…) are RTX launcher abstractions — translate the equivalent into recipe args/env when needed. ## Pitfall ledger (scar tissue — do not re-learn) - **Gitlink = empty dir**: a `git clone` materializes vendored gitlink paths as empty dirs; a `[ -d ]` check is a TRAP (saw this twice: r18's `deps-src.tar.gz` stubs → cmake INSTALL death; r27's FlashMLA `csrc/cutlass` → compile death). Gate on `.git` presence inside the path (fetch-deps.sh `populate_gitlinks` does this generically; AMD-only trees composable_kernel/aiter are skipped intentionally). - **Top-level cutlass is pinned too** (`VLLM_CUTLASS_SRC_DIR=/deps2/cutlass` + `CUTLASS_INCLUDE_DIR_PRE`) — keep it at the tree's declared tag (v4.4.2), NOT the dep-gitlink ones. - macOS staging poisons contexts: 23k AppleDouble `._*` files + provenance xattrs in the r18 deps tarball. With clone-at-pin this cannot recur; if you ever build a context on macOS, use `COPYFILE_DISABLE=1 tar --no-xattrs` and `find . -name '._*' -delete`. - Preferences/danger: `rg -rn` is a trap (`-r` = replace; use `rg -n`); `rg --include` does not exist (`-g '*.py'`); node zsh aborts on unmatched globs; the sm121 launcher's chunked uploader streams 60 KB per ssh round trip (fine for 215 KB contexts, hopeless for 400 MB dirs). - Engine quirks learned by bench nights (see `specs/tech-architecture/glm53-recipe-state.md` + `history/night-shifts/`): Spark-checkpoint MXFP8 MTP experts are not TP4-tileable (`mm_mxfp8`), B12X rejects drafter-config KV dtypes on SM121 (FA fp8 needs FA3/FA4), 2x-lane memory ceiling ≈ 10 GiB KV pin (4x NVFP4 has room for 16 GiB), JIT caches persist per node under `~/sm121-cache/jit-121f/*`. ## Bench conventions (tool-eval-bench) - Tool: `2.6.1.dev45+gcf54b4bfe` (uv tool), `--hardmode` = 88 scenarios (69 standard + 19 Hard Mode), `--perf` = llama-benchy pp2048/tg128 c1+c2. - Gate: quality score is the primary gate; seeds fixed (42); the canonical anchor + per-lane baseline numbers live in `specs/tech-architecture/glm53-recipe-state.md`. - Reports land in `recipes//reports/_