---
name: jovian-images
description: Build and adopt the jovian-judgement (GLM-5.3-Flash) DGX Spark serving images — clone-at-pin r27+ builds from the lab's source.lock contract, rev adoption procedure, GB10 adaptation deltas, gates, and the failure-mode ledger from actual build nights. Use whenever rebuilding glm53-* images, adopting a new lab rev (r28+), adding engine deps, or debugging the clone-at-pin build in docker-images/.
---

# Jovian Image Builds (clone-at-pin, source-locked)

The lab qualifies GLM-5.3-Flash serving revs and publishes a **source
contract**: `local-inference-lab/rtx6kpro` → `models/glm-5.3-flash.md`
(human doc) + `models/glm-5.3-flash/build/<rev>.source.lock` (exact pins)
+ `build/Dockerfile` (their RTX PRO 6000 build — reference, not ours).
Our builds run on DGX Spark GB10 (aarch64, sm_121a, unified memory) — same
source locks, GB10-adapted runtime.

## Layout

- `docker-images/jovian-judgement/` — the v2 contract build (**reproducible
  from a fresh repo clone**; no context tarballs):
  - `versions.env` — every pin: source SHAs + tree hashes + pretend
    package versions + runtime versions. Single source of truth.
  - `profiles/<name>.env` — named pin sets for recipe adoption (same
    `KEY=VAL` format; only the keys that differ from `versions.env`). A
    recipe references one via `build_profile = "<name>"`; merge order:
    versions.env < profile < recipe `[build.versions]`. `r28.env`
    is the r28 lock (vLLM 62f6385c + b12x 7216f6b9 + LMCache 934c287a).
    The profile file lives in the build dir → it ships with the context
    and moves the dir checksum.
  - `fetch-deps.sh` — clones every vLLM build sub-dependency at the pins
    the r27 tree itself declares; grep-gates each pin against the tree's
    CMake; populates vendored **gitlink trees** (cutlass inside
    flash-attention / MSA / FlashMLA / FlashKDA / qutlass).
  - `Dockerfile` — stages: apt → torch → deepgemm (in-build clone) →
    vllm-clone (gated) → deps-fetched → vllm-builder → b12x-clone/builder →
    lmcache-builder → runner (installs wheels + gates).
  - `build.sh` — generic driver; forwards every versions.env KEY=VAL as a
    build-arg. `TAG=jovian-judgement:r35-glm53 bash build.sh`.
- Legacy r12/r18 staging-era build dirs were REMOVED from the repo
  (2026-09-07): their tarball contexts were not rebuildable anyway. The
  r18 image (`glm53-jovian-r18:local`) lives on the nodes and the recipes
  keep referencing it; `docker_image_dir` for those recipes is now
  informational only (no build dir). The jovian-judgement dir is the current build
  lineage.
- Production recipes reference `docker_image = "jovian-judgement:<tag>"`
  where the tag names the pin cut + target model (e.g. `r35-glm53`,
  `r37-ds41`; built from the matching `profiles/<profile>.env`) and
  `docker_image_dir = "jovian-judgement"` (the live build dir).

## Pin discipline (r27 contract, verified 2026-09-06)

| pkg | pin | tree |
|---|---|---|
| vLLM | `voipmonitor/vllm` @ `63a82f8d…` (branch `integration/glm53-r27-release-20260906`) | `f54cd9ca…` |
| B12X | `voipmonitor/b12x` @ `e8ad299b…` (same branch name) | `f3cd8a9e…` |
| LMCache | `local-inference-lab/LMCache` @ `822330db…` | `008ac3e0…` |
| r28 profile | `profiles/r28.env`: vLLM `62f6385c…` / b12x `7216f6b9…` / LMCache `934c287a…` (one 09-06 cut) | `6361a741…` / `be0572c5…` / `a860684a…` |
| r39 (lineage split, 2026-09-21) | `jovian-judgement-r39/profiles/r39.env`: vLLM `9e5d1793fa3` / b12x `c2dc1cf022` (same-moment cut; RoCE-TC commit) / LMCache held `29bc5a2e`. The lab moved every external cmake pin (cutlass v4.4.2→v4.7.1, DeepGEMM→vllm-project fork, +DeepSelect) → **own dir** `jovian-judgement-r39/` with its own fetch-deps.sh + deps.lock; the r27/r35 dir keeps the r38-era contract | `01f937ec…` / `1f77ebff…` / `5a88a1ea…` |
| FlashKDA | `vllm-project/FlashKDA` @ `3b225bf2…` | (gitlink in dep) |
| cutlass (top) | `nvidia/cutlass` @ `v4.4.2` | — |
| FlashMLA | `a8f794d1…` (→ vendored cutlass `147f5673…`) | — |
| MSA | `087c1618…` (→ vendored cutlass `eb61c911…`) | — |
| qutlass | `e74319e3…` (→ vendored cutlass `b2ca083d…`) | — |
| tml-fa4 | `b2068346…` | — |
| flash-attention | `f3e1a4f7…` (→ vendored cutlass `62750a2b…`) | — |
| triton-kernels | `triton-lang/triton` @ `v3.5.1` | — |
| DeepGEMM | `deepseek-ai/DeepGEMM` `nv_dev` @ `8b1392b9…` | — |

**Branch names are NOT the contract** — the LMCache lock's branch was
renamed upstream within a day. Pins are commit SHAs + tree hashes; branch
names live in comments only. Wheels get deterministic versions through
`SETUPTOOLS_SCM_PRETEND_VERSION` (engine string then identifies the pin:
`0.1.dev0+glm53.r27.63a82f8d`).

## Adopting a new lab rev (r28 →)

1. Read the rev's `models/glm-5.3-flash.md` + `<rev>.source.lock` from
   rtx6kpro master (fetch raw.githubusercontent; the lock lists all four
   package trees + trees + runtime versions).
2. **Verify the FULL pin set, fork by fork** (the 2026-09-08 r28 lesson):
   the source lock is cut across ALL forks within minutes of each other
   (vLLM + b12x + LMCache ±1 min). For each fork list the integration
   branches (`/repos/<owner>/<repo>/branches`, page through — the r2x lines
   live past page 1), compare branch-tip vs current pin with the compare
   API (`/compare/<old>...<new>` → `ahead_by`/`behind_by`), and take the
   tree hash from the commit. Advancing only vLLM while leaving b12x/LMCache
   behind is a half-cut: it may bench fine but silently skips loader/perf
   fixes. A rev is adopted as ONE set — write it as a `profiles/<rev>.env`
   (or bump `versions.env` for the base image), never as scattered
   per-recipe `[build.versions]` copy-pastes.
3. Verify each pin is fetchable (`git ls-remote` for branch-anchored
   SHAs, API commits probe for SHA-only pins; FlashKDA's hash-grep usually
   already covered by the tree's own cmake).
4. **Re-sync `fetch-deps.sh` dep table by reading the NEW tree's**
   `cmake/external_projects/*.cmake` — revs add/move deps (r27 added
   FlashKDA, LMCache, flashinfer tvm-ffi/tilelang pins). Update the gate
   list with every GIT_REPOSITORY/GIT_TAG pair found. (Note: b12x/LMCache
   are NOT in the fetch-deps gate list — they are separate clone stages;
   their vLLM-tree compatibility is established by the lock, not the gates.)
5. Rebuild on the build node, boot on the lane, then bench+report.

## Building / distributing

- Build on dgx4 (buildkit cache mounts: uv-cache, ccache, cargo-registry,
  rust-target). Context is tiny now (~215 KB): `tar czf` the dir + `scp`
  + extract + `TAG=… bash build.sh` under `setsid nohup … &`, log to
  /tmp/r27build<N>.log.
- Alternatively `sm121 build --recipe <lane> --build-node` — fine for THIS
  size (chunks at 60 KB/ssh-trip); the r18-era 417 MB dir was what made it
  crawl — with clone-at-pin, sm121 build also stamps `sm121.checksum`
  (manually built images stay unlabeled → informational STALE warning).
- Distribute: `docker save <img> | ssh dst docker load` (the
  `docker save -o -` variant is broken on this stack). Run transfers with
  `setsid nohup bash -c '...' < /dev/null > log 2>&1 &` (128 MB/s node-to-
  node; 16.6 GB ≈ 2 min).

## Boot gates (already in the Dockerfile runner; re-check after serve)

Imports: `vllm` (string carries the pin), `b12x`, `lmcache` (+
`lmcache.cuda_ops`/`lmcache_native`/`lmcache_fs`), `vllm._flashkda_C`,
B12X enum resolves to the b12x backend, `nvfp4_ds_mla` present in
`vllm/config/cache.py`. Serving: expect the KV-size log line
(`GPU KV cache size: N tokens`) ~10 min in, application startup ~5 min
after — **when JIT mounts are warm** (~9 min total; ~45 min cold/first).

## GB10 adaptation deltas (RTX lock → Spark)

- Runtime: CUDA 13.2 aarch64 / torch 2.13.0+cu130 / flashinfer 0.6.17 /
  torch-bundled NCCL — all matches the TREE's own requirements
  (`requirements/cuda.txt`); their RTX runtime (13.3/0.6.18/NCCL 2.31.2)
  is a documented deviation, re-evaluate only when the tree pins change.
- `TORCH_CUDA_ARCH_LIST=12.1a` (incl. the LMCache wheel build — theirs=12.0).
- `patch_cuda_sm12x.py` + eugr patch set, all idempotent and run in full:
  `preserve_sm12x_target` (12.1a collapses to 12.0 without it),
  `sm120_cooperative_topk` (kept SM90-only), `spark_kv_cache_cleanup`
  (UMA/MemFree — see note below), b12x c128a alignment, flashinfer
  swigluoai, speculator cudagraph pool, routed-expert weight shape.
- InstantTensor loader is fork-builtin (no separate pip package needed).
- env for serving comes from recipes; their launcher envs (SPECULATOR/…)
  are RTX launcher abstractions — translate the equivalent into recipe
  args/env when needed.

## Pitfall ledger (scar tissue — do not re-learn)

- **Gitlink = empty dir**: a `git clone` materializes vendored gitlink
  paths as empty dirs; a `[ -d ]` check is a TRAP (saw this twice: r18's
  `deps-src.tar.gz` stubs → cmake INSTALL death; r27's FlashMLA
  `csrc/cutlass` → compile death). Gate on `.git` presence inside the
  path (fetch-deps.sh `populate_gitlinks` does this generically; AMD-only
  trees composable_kernel/aiter are skipped intentionally).
- **Top-level cutlass is pinned too** (`VLLM_CUTLASS_SRC_DIR=/deps2/cutlass`
  + `CUTLASS_INCLUDE_DIR_PRE`) — keep it at the tree's declared tag (v4.4.2),
  NOT the dep-gitlink ones.
- macOS staging poisons contexts: 23k AppleDouble `._*` files + provenance
  xattrs in the r18 deps tarball. With clone-at-pin this cannot recur; if
  you ever build a context on macOS, use `COPYFILE_DISABLE=1 tar
  --no-xattrs` and `find . -name '._*' -delete`.
- Preferences/danger: `rg -rn` is a trap (`-r` = replace; use `rg -n`);
  `rg --include` does not exist (`-g '*.py'`); node zsh aborts on unmatched
  globs; the sm121 launcher's chunked uploader streams 60 KB per ssh round
  trip (fine for 215 KB contexts, hopeless for 400 MB dirs).
- **TOML context trap in recipes (2026-09-08)**: a bare top-level key placed
  AFTER a `[[extra_mounts]]` block parses into the LAST mount table and is
  silently ignored (serde has no deny_unknown_fields) — the build ran with
  the base pins while the recipe "had" the override. Table headers
  (`[build.versions]`) reset the context, so tables are safe anywhere; bare
  keys (e.g. `build_profile`) must sit before the first `[section]`.
  Verify with `python3 -c "import tomllib; ..."` that the key is top-level.
- **Half-cut revs (2026-09-08, r28)**: advancing only the vLLM pin of a
  rev while leaving b12x/LMCache at the previous rev builds a mixed stack
  that still works and benches — but skipped 21 b12x commits (GLM/DeepSeek
  loader fixes, MoE perf) and 3 LMCache MP-transfer fixes. The r28 bench
  ran on that mix before the audit. Always cut/adopt the FULL fork set
  (see "Adopting a new lab rev" step 2) and record the set in one profile.
- **Bench pipe pump (2026-09-08)**: `sm121 bench`'s output pump read into
  an empty `Vec` — `read()` on a zero-length slice returns 0 = "EOF", the
  pipe closed, and tool-eval-bench died on its first stderr write with
  EPIPE → silent exit 120, empty log. Fixed in `src/bench.rs` (8 KiB
  buffer + EINTR retry + regression test). If a child of a launcher
  command dies with 120 and zero output, suspect a dead pipe pump, not the
  child.
- Engine quirks learned by bench nights (see
  `specs/tech-architecture/glm53-recipe-state.md` + `history/night-shifts/`):
  Spark-checkpoint MXFP8 MTP experts are not TP4-tileable (`mm_mxfp8`),
  B12X rejects drafter-config KV dtypes on SM121 (FA fp8 needs FA3/FA4),
  2x-lane memory ceiling ≈ 10 GiB KV pin (4x NVFP4 has room for 16 GiB),
  JIT caches persist per node under `~/sm121-cache/jit-121f/*`.

## Bench conventions (tool-eval-bench)

- Tool: `2.6.1.dev46+g9d412464a` (uv tool), `--hardmode` = 88 scenarios
  (69 standard + 19 Hard Mode), `--perf` = llama-benchy pp2048/tg128 c1+c2.
- Gate: quality score is the primary gate; seeds fixed (42); the canonical
  anchor + per-lane baseline numbers live in
  `specs/tech-architecture/glm53-recipe-state.md`.
- Reports land in `recipes/<lane>/reports/<UTC-stamp>_<label>.md` and the
  combined score+perf tables go to `history/night-shifts/<date>.md`.