name: jovian-images

description: Build and adopt the jovian-judgement (GLM-5.3-Flash) DGX Spark serving images — clone-at-pin r27+ builds from the lab's source.lock contract, rev adoption procedure, GB10 adaptation deltas, gates, and the failure-mode ledger from actual build nights. Use whenever rebuilding glm53-* images, adopting a new lab rev (r28+), adding engine deps, or debugging the clone-at-pin build in docker-images/.

Jovian Image Builds (clone-at-pin, source-locked)

The lab qualifies GLM-5.3-Flash serving revs and publishes a source contract: local-inference-lab/rtx6kpro → models/glm-5.3-flash.md (human doc) + models/glm-5.3-flash/build/<rev>.source.lock (exact pins)

build/Dockerfile (their RTX PRO 6000 build — reference, not ours). Our builds run on DGX Spark GB10 (aarch64, sm_121a, unified memory) — same source locks, GB10-adapted runtime.

Layout

docker-images/jovian-judgement/ — the v2 contract build (reproducible from a fresh repo clone; no context tarballs):
- versions.env — every pin: source SHAs + tree hashes + pretend package versions + runtime versions. Single source of truth.
- profiles/<name>.env — named pin sets for recipe adoption (same KEY=VAL format; only the keys that differ from versions.env). A recipe references one via build_profile = "<name>"; merge order: versions.env < profile < recipe [build.versions]. r28.env is the r28 lock (vLLM 62f6385c + b12x 7216f6b9 + LMCache 934c287a). The profile file lives in the build dir → it ships with the context and moves the dir checksum.
- fetch-deps.sh — clones every vLLM build sub-dependency at the pins the r27 tree itself declares; grep-gates each pin against the tree's CMake; populates vendored gitlink trees (cutlass inside flash-attention / MSA / FlashMLA / FlashKDA / qutlass).
- Dockerfile — stages: apt → torch → deepgemm (in-build clone) → vllm-clone (gated) → deps-fetched → vllm-builder → b12x-clone/builder → lmcache-builder → runner (installs wheels + gates).
- build.sh — generic driver; forwards every versions.env KEY=VAL as a build-arg. TAG=jovian-judgement:r35-glm53 bash build.sh.
Legacy r12/r18 staging-era build dirs were REMOVED from the repo (2026-09-07): their tarball contexts were not rebuildable anyway. The r18 image (glm53-jovian-r18:local) lives on the nodes and the recipes keep referencing it; docker_image_dir for those recipes is now informational only (no build dir). The jovian-judgement dir is the current build lineage.
Production recipes reference docker_image = "jovian-judgement:<tag>" where the tag names the pin cut + target model (e.g. r35-glm53, r37-ds41; built from the matching profiles/<profile>.env) and docker_image_dir = "jovian-judgement" (the live build dir).

Pin discipline (r27 contract, verified 2026-09-06)

pkg	pin	tree
vLLM	`voipmonitor/vllm` @ `63a82f8d…` (branch `integration/glm53-r27-release-20260906`)	`f54cd9ca…`
B12X	`voipmonitor/b12x` @ `e8ad299b…` (same branch name)	`f3cd8a9e…`
LMCache	`local-inference-lab/LMCache` @ `822330db…`	`008ac3e0…`
r28 profile	`profiles/r28.env`: vLLM `62f6385c…` / b12x `7216f6b9…` / LMCache `934c287a…` (one 09-06 cut)	`6361a741…` / `be0572c5…` / `a860684a…`
r39 (lineage split, 2026-09-21)	`jovian-judgement-r39/profiles/r39.env`: vLLM `9e5d1793fa3` / b12x `c2dc1cf022` (same-moment cut; RoCE-TC commit) / LMCache held `29bc5a2e`. The lab moved every external cmake pin (cutlass v4.4.2→v4.7.1, DeepGEMM→vllm-project fork, +DeepSelect) → own dir `jovian-judgement-r39/` with its own fetch-deps.sh + deps.lock; the r27/r35 dir keeps the r38-era contract	`01f937ec…` / `1f77ebff…` / `5a88a1ea…`
FlashKDA	`vllm-project/FlashKDA` @ `3b225bf2…`	(gitlink in dep)
cutlass (top)	`nvidia/cutlass` @ `v4.4.2`	—
FlashMLA	`a8f794d1…` (→ vendored cutlass `147f5673…`)	—
MSA	`087c1618…` (→ vendored cutlass `eb61c911…`)	—
qutlass	`e74319e3…` (→ vendored cutlass `b2ca083d…`)	—
tml-fa4	`b2068346…`	—
flash-attention	`f3e1a4f7…` (→ vendored cutlass `62750a2b…`)	—
triton-kernels	`triton-lang/triton` @ `v3.5.1`	—
DeepGEMM	`deepseek-ai/DeepGEMM` `nv_dev` @ `8b1392b9…`	—

Branch names are NOT the contract — the LMCache lock's branch was renamed upstream within a day. Pins are commit SHAs + tree hashes; branch names live in comments only. Wheels get deterministic versions through SETUPTOOLS_SCM_PRETEND_VERSION (engine string then identifies the pin: 0.1.dev0+glm53.r27.63a82f8d).

Adopting a new lab rev (r28 →)

Read the rev's models/glm-5.3-flash.md + <rev>.source.lock from rtx6kpro master (fetch raw.githubusercontent; the lock lists all four package trees + trees + runtime versions).
Verify the FULL pin set, fork by fork (the 2026-09-08 r28 lesson): the source lock is cut across ALL forks within minutes of each other (vLLM + b12x + LMCache ±1 min). For each fork list the integration branches (/repos/<owner>/<repo>/branches, page through — the r2x lines live past page 1), compare branch-tip vs current pin with the compare API (/compare/<old>...<new> → ahead_by/behind_by), and take the tree hash from the commit. Advancing only vLLM while leaving b12x/LMCache behind is a half-cut: it may bench fine but silently skips loader/perf fixes. A rev is adopted as ONE set — write it as a profiles/<rev>.env (or bump versions.env for the base image), never as scattered per-recipe [build.versions] copy-pastes.
Verify each pin is fetchable (git ls-remote for branch-anchored SHAs, API commits probe for SHA-only pins; FlashKDA's hash-grep usually already covered by the tree's own cmake).
Re-sync fetch-deps.sh dep table by reading the NEW tree's cmake/external_projects/*.cmake — revs add/move deps (r27 added FlashKDA, LMCache, flashinfer tvm-ffi/tilelang pins). Update the gate list with every GIT_REPOSITORY/GIT_TAG pair found. (Note: b12x/LMCache are NOT in the fetch-deps gate list — they are separate clone stages; their vLLM-tree compatibility is established by the lock, not the gates.)
Rebuild on the build node, boot on the lane, then bench+report.

Building / distributing

Build on dgx4 (buildkit cache mounts: uv-cache, ccache, cargo-registry, rust-target). Context is tiny now (~215 KB): tar czf the dir + scp
- extract + TAG=… bash build.sh under setsid nohup … &, log to /tmp/r27build.log.
Alternatively sm121 build --recipe <lane> --build-node — fine for THIS size (chunks at 60 KB/ssh-trip); the r18-era 417 MB dir was what made it crawl — with clone-at-pin, sm121 build also stamps sm121.checksum (manually built images stay unlabeled → informational STALE warning).
Distribute: docker save <img> | ssh dst docker load (the docker save -o - variant is broken on this stack). Run transfers with setsid nohup bash -c '...' < /dev/null > log 2>&1 & (128 MB/s node-to- node; 16.6 GB ≈ 2 min).

Boot gates (already in the Dockerfile runner; re-check after serve)

Imports: vllm (string carries the pin), b12x, lmcache (+ lmcache.cuda_ops/lmcache_native/lmcache_fs), vllm._flashkda_C, B12X enum resolves to the b12x backend, nvfp4_ds_mla present in vllm/config/cache.py. Serving: expect the KV-size log line (GPU KV cache size: N tokens) ~10 min in, application startup ~5 min after — when JIT mounts are warm (~9 min total; ~45 min cold/first).

GB10 adaptation deltas (RTX lock → Spark)

Runtime: CUDA 13.2 aarch64 / torch 2.13.0+cu130 / flashinfer 0.6.17 / torch-bundled NCCL — all matches the TREE's own requirements (requirements/cuda.txt); their RTX runtime (13.3/0.6.18/NCCL 2.31.2) is a documented deviation, re-evaluate only when the tree pins change.
TORCH_CUDA_ARCH_LIST=12.1a (incl. the LMCache wheel build — theirs=12.0).
patch_cuda_sm12x.py + eugr patch set, all idempotent and run in full: preserve_sm12x_target (12.1a collapses to 12.0 without it), sm120_cooperative_topk (kept SM90-only), spark_kv_cache_cleanup (UMA/MemFree — see note below), b12x c128a alignment, flashinfer swigluoai, speculator cudagraph pool, routed-expert weight shape.
InstantTensor loader is fork-builtin (no separate pip package needed).
env for serving comes from recipes; their launcher envs (SPECULATOR/…) are RTX launcher abstractions — translate the equivalent into recipe args/env when needed.

Pitfall ledger (scar tissue — do not re-learn)

Gitlink = empty dir: a git clone materializes vendored gitlink paths as empty dirs; a [ -d ] check is a TRAP (saw this twice: r18's deps-src.tar.gz stubs → cmake INSTALL death; r27's FlashMLA csrc/cutlass → compile death). Gate on .git presence inside the path (fetch-deps.sh populate_gitlinks does this generically; AMD-only trees composable_kernel/aiter are skipped intentionally).
Top-level cutlass is pinned too (VLLM_CUTLASS_SRC_DIR=/deps2/cutlass
- CUTLASS_INCLUDE_DIR_PRE) — keep it at the tree's declared tag (v4.4.2), NOT the dep-gitlink ones.
macOS staging poisons contexts: 23k AppleDouble ._* files + provenance xattrs in the r18 deps tarball. With clone-at-pin this cannot recur; if you ever build a context on macOS, use COPYFILE_DISABLE=1 tar --no-xattrs and find . -name '._*' -delete.
Preferences/danger: rg -rn is a trap (-r = replace; use rg -n); rg --include does not exist (-g '*.py'); node zsh aborts on unmatched globs; the sm121 launcher's chunked uploader streams 60 KB per ssh round trip (fine for 215 KB contexts, hopeless for 400 MB dirs).
TOML context trap in recipes (2026-09-08): a bare top-level key placed AFTER a [[extra_mounts]] block parses into the LAST mount table and is silently ignored (serde has no deny_unknown_fields) — the build ran with the base pins while the recipe "had" the override. Table headers ([build.versions]) reset the context, so tables are safe anywhere; bare keys (e.g. build_profile) must sit before the first [section]. Verify with python3 -c "import tomllib; ..." that the key is top-level.
Half-cut revs (2026-09-08, r28): advancing only the vLLM pin of a rev while leaving b12x/LMCache at the previous rev builds a mixed stack that still works and benches — but skipped 21 b12x commits (GLM/DeepSeek loader fixes, MoE perf) and 3 LMCache MP-transfer fixes. The r28 bench ran on that mix before the audit. Always cut/adopt the FULL fork set (see "Adopting a new lab rev" step 2) and record the set in one profile.
Bench pipe pump (2026-09-08): sm121 bench's output pump read into an empty Vec — read() on a zero-length slice returns 0 = "EOF", the pipe closed, and tool-eval-bench died on its first stderr write with EPIPE → silent exit 120, empty log. Fixed in src/bench.rs (8 KiB buffer + EINTR retry + regression test). If a child of a launcher command dies with 120 and zero output, suspect a dead pipe pump, not the child.
Engine quirks learned by bench nights (see specs/tech-architecture/glm53-recipe-state.md + history/night-shifts/): Spark-checkpoint MXFP8 MTP experts are not TP4-tileable (mm_mxfp8), B12X rejects drafter-config KV dtypes on SM121 (FA fp8 needs FA3/FA4), 2x-lane memory ceiling ≈ 10 GiB KV pin (4x NVFP4 has room for 16 GiB), JIT caches persist per node under ~/sm121-cache/jit-121f/*.

Bench conventions (tool-eval-bench)

Tool: 2.6.1.dev46+g9d412464a (uv tool), --hardmode = 88 scenarios (69 standard + 19 Hard Mode), --perf = llama-benchy pp2048/tg128 c1+c2.
Gate: quality score is the primary gate; seeds fixed (42); the canonical anchor + per-lane baseline numbers live in specs/tech-architecture/glm53-recipe-state.md.
Reports land in recipes/<lane>/reports/<UTC-stamp>_<label>.md and the combined score+perf tables go to history/night-shifts/<date>.md.

SKILL.md 12 KB Historia Czysty