launcher/docker-images/mia-tp4/.env.tp4.example

# 4x DGX Spark profile (TP4/EP4). Copy to .env.tp4 and edit. Never commit .env.tp4.
# Used by ./start-tp4.sh; ./start.sh keeps using .env (3 Sparks).

# ─── cluster ──────────────────────────────────────────────────────────────────
HEAD_IP=10.0.0.1
WORKER_IPS="10.0.0.2 10.0.0.3 10.0.0.4"
WORKER_HOSTS="spark2 spark3 spark4"
WORKER_USER=zurih
SSH_IDENTITY=$HOME/.ssh/id_ed25519_shared

# ─── fabric (ConnectX-7 RoCE) ─────────────────────────────────────────────────
# Gloo/NCCL TCP bootstrap on the LAN NIC. Data plane prefers RoCE (NCCL_NET=IB).
FABRIC_IFACE=enp1s0f1np1
GLOO_SOCKET_IFNAME=enP7s7
NCCL_SOCKET_IFNAME=enP7s7
IB_HCA=rocep1s0f0,rocep1s0f1
NCCL_NET=IB
NCCL_IB_DISABLE=0
NCCL_P2P_DISABLE=1
NCCL_SHM_DISABLE=1
NCCL_DEBUG=WARN
# NCCL connection buffers live in pinned host RAM, which on GB10 is GPU memory:
# the defaults (4 MiB Simple + 4.69 MiB LL128 + 0.5 MiB LL per connection, 512
# connections) cost 4.7 GiB per node. Decode all-reduces are 61 KB (LL protocol),
# prefill uses Simple; LL128 buys nothing here. Measured: 5.0 GB -> 0.14 GB pinned.
NCCL_BUFFSIZE=1048576
NCCL_LL128_BUFFSIZE=262144
NCCL_PROTO=^LL128
NCCL_MAX_NCHANNELS=8
# Set NCCL_DEBUG=INFO for one boot to see communicator/channel/protocol choices.
NCCL_DEBUG_SUBSYS=INIT,ENV
NCCL_HOST_DIR=$HOME/nccl-2.30.7

# ─── switchless ring (optional, 4 nodes without a RoCE switch) ────────────────
# A 4-node ring has no direct fabric path between opposite nodes (rank0 <->
# rank2), and RoCE queue pairs do not follow IP routing, so NCCL's Tree/PAT
# transports fail to connect. Setting this to 1:
#   1. injects ring-only NCCL settings into every rank, and
#   2. overlays the patched host NCCL (NCCL_HOST_DIR) onto the container's pip
#      NCCL path (NCCL_PIP_SO) instead of using LD_LIBRARY_PATH, because DeepEP
#      refuses to start when two NCCL runtimes are visible.
# Step 2 needs a NCCL carrying the sparkring switchless-cycle patch; stock NCCL
# silently ignores NCCL_SWITCHLESS_RING_ONLY and still fails on the Tree
# transport. See docs/switchless-ring.md for the patch set and a build recipe.
#NCCL_SWITCHLESS_RING_ONLY=1
#NCCL_HOST_DIR=$HOME/nccl-2.30.7
#NCCL_WORKER_DIR=/opt/patched-nccl  # optional absolute path on every worker
#NCCL_ALGO=Ring
#NCCL_IB_SUBNET_PREFIX_LEN=24
#NCCL_MIN_NCHANNELS=4
#NCCL_MAX_NCHANNELS=4
#NCCL_P2P_LEVEL=SYS
# Four channels keep pinned host RAM low on the tested ring.
# GB10 UMA safeguard used by the tested ring: consume each checkpoint tensor before
# queueing the next host-to-device copy. Not part of the switched production line
# below (measured with the default loader and DSV41_FAST_LOAD's own pacing).
#DSV41_SERIAL_WEIGHT_LOAD=1
# Local weights on every node (a ring has no fabric-wide NFS path): docs/switchless-ring.md.
#NFS_SHARE=0
#
# Both ConnectX-7 cards (four RDMA devices) on the ring need a NCCL build with sparkring's
# dual-PCI-domain patch plus four NCCL_IB_* flags on every rank, passed through
# EXTRA_CONTAINER_ENV: docs/switchless-ring.md ("Devices past the second are never advertised").
#
# RoCEnante on the ring. Its one-shot all-reduce/all-gather writes into every peer, and a ring has no
# link between opposite nodes, so without more a ring runs the production line with
# SGLANG_ROCE_ALLREDUCE=0 and no DSV41_ROCE_GATHER. docs/switchless-ring.md ("RoCEnante on the ring")
# builds the opposite-node path in the neighbours' ConnectX-7 hardware (FujitsuPolycom/sparkring) and
# scripts/ring_mesh/plan.py prints the line for your fleet: the peer maps depend on the cabling and
# on the TP rank order. Append it to the production EXTRA_CONTAINER_ENV, replacing that line's
# B12X_ROCE_HCA, SGLANG_ROCE_MAX_SIZE and DSV41_ROCE_GATHER values:
#   DSV41_ROCE_RING=1 SGLANG_ROCE_ALLREDUCE=1 SGLANG_ROCE_MAX_SIZE=262144 DSV41_ROCE_GATHER=262144 B12X_ROCE_HCA=rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1 B12X_ROCE_PEER_HCA_MAPS=<plan.py> B12X_ROCE_TWO_WAVE_THRESHOLD_BYTES=0
# 262144 needs hairpin_queue_size 8192 on every fabric function (dsv41-mesh.service sets it at boot);
# at the driver default of 1024 use 81920.

# ─── model / image ────────────────────────────────────────────────────────────
MODEL_DIR=$HOME/NewModels/DeepSeek-V4.1-Flash
COMMON_MODEL=/var/tmp/DeepSeek-V4.1-Flash
# Same digest as the Dockerfile FROM: the dev-dsv41 tag moves, the digest does not.
BASE_IMAGE=lmsysorg/sglang:dev-dsv41@sha256:3dbc313030a6ef2c5d7de8ecf48e9aece722694a82182cb618cc82b588816349
IMAGE=dsv41-4x-spark:local
PORT=8888

# ─── weights on spark2/spark3/spark4 ─────────────────────────────────────────────────
# Default: from the head over NFSv4. On a switched fabric give one server
# address for all workers, or one per worker in WORKER_IPS order. For a ring,
# use local worker bind volumes with NFS_SHARE=0 (docs/switchless-ring.md), or
# supply a separately reachable NFS TCP path and matching ACLs.
NFS_SHARE=1
NFS_VOLUME=dsv41-weights
NFS_EXPORT_NAME=dsv41-native
NFS_SERVER_IPS="10.0.22.1"
NFS_CLIENTS=10.0.0.2,10.0.0.3,10.0.0.4,10.0.22.0/24

# ─── parallelism (4 Sparks) ──────────────────────────────────────────────────
# TP=4 EP=4 matches world size. heads=64 / o_groups=8 / draft experts=128 /
# vocab=129280 all divide by 4, so no TP padding is needed (DSV41_TP_PAD=0 below).
NNODES=4
TP_SIZE=4
# 2 for the base and canary images (FlashInfer's MXFP4 MoE needs the per-rank expert width to be
# a multiple of 128). The production line below runs EP_SIZE=1 (every rank a 576-wide slice of all
# 384 experts, routed MoE on b12x main via DSV41_MOE_B12X_NEXT): no EP-group straggler at the MoE
# all-reduce. Set EP_SIZE=1 together with that line.
EP_SIZE=2
DIST_PORT=20000

# ─── serving ──────────────────────────────────────────────────────────────────
# NVMe Engram offload is required on Spark (host RAM == GPU unified memory).
OFFLOAD_MODE=nvme
# Per-host cache budget, split across the 2 Engram layers and the ranks sharing
# that host. 0 disables it: with the packed local shards a miss is ONE local
# read, and the measured miss rate costs ~145 reads/s against a drive that does
# ~112k IOPS. Host RAM is GPU memory here and rank 0 has ~0.3 GB available, so
# the GiB this used to hold is worth far more as headroom than as hit rate.
DSV41_CACHE_GIB=4
# GB10 NVMe: ~3.5k IOPS at QD=1, ~112k at QD=64. The Engram callback blocks the
# compute stream, so misses are serviced by a pool rather than one at a time.
DSV41_IO_THREADS=96
# Pinning the owned scale rows makes a miss cost one pread instead of two
# (~1.8x on the callback), but it needs ~1 GiB/layer/rank of EAGER host RAM, and
# host RAM here is GPU memory: with weights at ~104 GB the pin raised SGLang's
# "minimum viable" mem-fraction from ~0.940 to 0.9569 and left no KV cache.
# Off until the packed local shard lands, which gets the same single read from
# disk layout instead of RAM.
DSV41_RESIDENT_SCALES=0
DSV41_CACHE_WAYS=16
DSV41_STATS_SECONDS=60
CONTEXT_LENGTH=1048576
# 0.80 of each rank's 121.7 GiB is the static pool (weights + KV + CUDA graphs);
# the other ~24 GiB stays outside it for the prefill transient (the indexer's
# per-chunk buffers, see CHUNKED_PREFILL_SIZE) and runtime slack. The KV budget
# SGLang reports on the boot line (full_token=) shrinks with this fraction, so
# MAX_TOTAL_TOKENS below is sized for the 0.80 budget, not the old 0.90 one.
MEM_FRACTION_STATIC=0.80
# Rank 0 pays ~1.3 GiB the workers do not (HTTP server + tokenizer +
# detokenizer) and also serves the checkpoint over NFS. The knob exists to tune
# it separately, but note LOWERING it does not give the head headroom: weights
# cost the same either way, so a smaller fraction only starves the KV pool.
# Measured on this box: 0.9436 is where KV goes negative, and 0.945 still leaves
# only 0.12 GB -- less than the 0.34 GB DSV4 SWA cap. 0.95 is the working value.
# Those are TP3 measurements; at TP4 the head simply tracks MEM_FRACTION_STATIC.
HEAD_MEM_FRACTION_STATIC=0.80
# 8 concurrent requests. This also sets --cuda-graph-max-bs-decode (start.sh
# passes the same value through), so CUDA-graph capture covers batch tiers 1-8.
MAX_RUNNING_REQUESTS=16
# 1024-token prefill chunks. The indexer's per-chunk transient peaks at about
# 14 B x chunk x prefix (measured, docs/chunked-prefill-memory.md): ~15 GB for a
# 1024-token chunk at 1M context, ~60 GB for 4096, which no rank has. Verified:
# a 1M-context needle test passed on TP4 at 1024 (the 3-Spark fleet reached
# 208k single / 186k concurrent at the same chunk). Raise it for faster
# short-prompt prefill only after a 1M prompt has passed
# scripts/verify/memguard.py at the larger size.
CHUNKED_PREFILL_SIZE=4096
# Pin the KV pool. Unpinned, SGLang sizes it from whatever is free at load time,
# which on a unified-memory box is the OS MemAvailable: it swings with the page
# cache and makes the pool (and decode speed) a boot lottery. Pinned, SGLang
# demands exactly this, and a pin that does not fit either dies ("no room ...
# within the available X GB") or is silently clamped down -- both have happened
# on this stack -- so the pin has to sit under the smallest budget a boot will
# ever compute, not at the ceiling.
# Read the budget off the boot log:
#   DSV4 memory calculation: bytes_per_full_token=1670.75 ... full_token=<N>
# At TP4 with MEM_FRACTION_STATIC=0.90 that line reported ~16M tokens (~26.7 GB
# per rank). This profile runs 0.80, which takes ~13 GB off that budget, and
# pins 8,000,000 x 1,670.75 B = 13.4 GB of KV per rank under it. Tested: the
# profile boots and serves at these values and a 1M-context needle test passed.
# If the pool ever has to shrink, lower this pin rather than raising the memory
# fraction: the fraction is what keeps the prefill transient safe. With
# MAX_RUNNING_REQUESTS=8 the pool holds 8 requests averaging 1,000,000 tokens
# each (8 x 1Mi = 8,388,608 would be the pin for all 8 at the full context).
# Pages are touched only when used, so the pool costs nothing until it fills.
MAX_TOTAL_TOKENS=8000000
SPEC_ALGO=DSPARK
DSPARK_BLOCK_SIZE=5
# Profile with sglang.benchmark.dspark_sps_profiler into state/dspark_sps.json;
# until that file exists DSpark runs the verify-all schedule and gains nothing.
DSPARK_SPS_TABLE=/state/dspark_sps.json
DSPARK_STS_TABLE=/state/dspark_sts.json

# ─── engram local shards ──────────────────────────────────────────────────────
# ./start.sh pack repacks each rank's owned Engram rows (weight+scale adjacent)
# onto node-local NVMe: ~68 GiB/node, one read per miss instead of two, and no
# NFS hop on the workers. Serving picks them up automatically when present.
ENGRAM_DIR=$HOME/dsv41-engram
DSV41_PACKED_DIR=/engram
SKIP_PREPARE=1
SKIP_VERIFY=1
SKIP_SMOKE=0
SMOKE_QUICK=1

# ─── kernels (GB10 / SM121) ───────────────────────────────────────────────────
# The 32x32 ue8m0 block-FP8 dense projections only have a Triton kernel unless a
# FlashInfer backend is named, which routes them to the MXFP8 dense kernels.
# --watchdog-timeout covers the CuTe-DSL JIT compile of the b12x kernels on the
# first forward after a boot.
# --min-free-slots-delay 1 disables the admission delayer, which is what makes
# MAX_RUNNING_REQUESTS=8 actually reachable. SGLang treats DSPARK as
# DFlash-family (spec_info.py: is_dflash_family = is_dflash or is_dspark), and
# for that family with max-running-requests >= 8 it auto-enables a threshold of
#   min(4, max(2, (max_running_requests + 5) // 6))
# (srt/managers/min_free_slots_delayer.py), i.e. 2 at 8 requests and 3 at 16.
# New prefills are then held until that many slots are free, so with steady
# arrivals the batch never sustains the configured max -- ~7 of 8, ~13-14 of 16.
# An explicit value wins and <= 1 disables it. Externally reported: +43% c8
# aggregate on a 3-Spark fleet. Not yet measured here. Drop this flag to get the
# batched admissions back if per-admission cost (DSpark's draft prefill pass)
# ever dominates at high concurrency.
# --sleep-on-idle: the scheduler stops busy-polling when there is nothing to do. Measured
# 2026-09-18: head scheduler idle CPU 47 % -> 14 %, workers 5 %; first response after 20 s
# of idle 0.18-0.21 s either way; decode unchanged. Suggested by RustyAiLab (Mia #16).
EXTRA_SGLANG_ARGS="--fp8-gemm-backend flashinfer_cutlass --watchdog-timeout 1800 --min-free-slots-delay 1 --enable-deepseek-v4-fp4-indexer --enable-cache-report --sleep-on-idle"
# With DSV41_FAST_LOAD=1 (below) the loader's shard window is what bounds the host
# memory the eager reads hold: num_threads + 1 shards, ~3.4 GB each on TP4/EP2.
# Append to the line above:
#   --model-loader-extra-config {\"num_threads\":1}
# Which FlashInfer MXFP8 kernel serves those projections (adapter/mxfp8_b12x.py):
# b12x is FlashInfer's own SM120/121 choice (16|32-row tiles, ~3x the CUTLASS
# SM120 kernel at decode M); cudnn is the alternative; cutlass restores SGLang's
# default, which measured no faster than Triton (both ~50 ms of a decode step).
DSV41_MXFP8_BACKEND=b12x
# adapter/shared_pad_k.py: at TP4 the shared expert's down_proj is N=5120 K=576, and
# 576 is not a multiple of 128, so b12x rejects it and this ONE shape in the model
# falls back to a CUTLASS kernel that pads M to 128 and runs at 25% occupancy
# (52 GB/s against the 196 GB/s b12x reaches on the same data). Padding K to 640 with
# zero weights puts it back on b12x: measured bit-identical (0 of 30720 elements
# differ), costs 14 MB per rank, and removes 0.90 ms from a 61.3 ms decode step.
# Padding alone buys nothing -- CUTLASS at K=640 measured 0.7% SLOWER than at 576 --
# so this is worth nothing on a stack where b12x is not the MXFP8 backend.
DSV41_SHARED_PAD_K=1
# FlashInfer's fused MoE finalize sums the top-k expert outputs with atomic bf16
# adds. The autotuner picks it for the 32-token bucket (17-32 token prefills,
# decode at concurrency 3-4), which made greedy outputs nondeterministic by up to
# ~1 nat on first-token logprobs. Off = separate finalize kernel, ~0.3 ms/step.
SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=0

# Reasoning budget for thinking-mode requests that carry no reasoning_effort.
# SGLang's V4.1 encoder maps low/high/xhigh/max to 25/50/75/100 and defaults to
# "high" = 50; the publisher's encoding/encoding.py says low=50, high=75, max=100
# with default 75. adapter/sitecustomize.py remaps the named tiers to the
# publisher's table; this pins the default as a number so it holds even without
# the remap. Integer 1-100 or low/high/xhigh/max. Per request: OpenAI
# reasoning_effort (a tier, or a 0-0.99 float = budget/100) or
# chat_template_kwargs {"reasoning_effort": N}. Only rendered in thinking mode.
SGLANG_DSV41_REASONING_EFFORT=75

# When a chat request omits max_tokens / max_completion_tokens, fill this in and
# also clamp larger client values. 32768 would have stopped the 714k-token
# racing-animation loop while covering the visual/HTML trials (~15k). 0 restores
# SGLang's fill-the-remaining-context behaviour (1M on TP4). Raise it for long
# thinking-on traces.
DSV41_MAX_NEW_TOKENS=32768

# Decode-side loop abort. --watchdog-timeout only fires when a step hangs; a live
# token stream (VIS-04: 714k tokens of racing-animation HTML) is healthy as far
# as the GPU watchdog is concerned. Finish with stop/repetition after four
# copies of a ≥32-token n-gram, 64 identical tokens, or eight identical decoded
# lines. The 2× EXL3 recipe has no equivalent. 0 disables.
DSV41_LOOP_ABORT=1
DSV41_LOOP_NGRAM=32
DSV41_LOOP_REPEATS=4
DSV41_LOOP_LINE_REPEATS=8

# PyTorch caching allocator mode. The DSV4 prefill allocates a larger indexer-logits
# buffer for every 2048-token chunk (each chunk scores the whole prefix so far); the
# native allocator cannot reuse smaller freed blocks for the next larger request and
# reserved memory grows with the SQUARE of the prompt length (measured: 8 GiB reserved,
# 0 live, after 64 growing steps), which is the 64k-128k prefill OOM on the head.
# expandable_segments coalesces freed space: 0.6 GiB after 128 steps in the same test.
# NOT expandable_segments:True: on the 3-node GB10 fleet it made every prefill of more
# than 64 query tokens return NaN logits (REPORT.md section 17). Untested at TP4.
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False

# adapter/prefill_empty_cache.py (in the shared image): after every prefill chunk of a
# sequence at least this long, return PyTorch's unused cached blocks so a long prompt
# costs one chunk's transient instead of the sum over chunks (REPORT.md section 17b).
# The per-chunk indexer buffer is CHUNKED_PREFILL_SIZE x prefix x 4 B per ratio-1 source
# layer, ~14 B x chunk x prefix at the peak across layers: at 1M context a 1024-token
# chunk peaks around 15 GB, against the ~24 GiB per rank that MEM_FRACTION_STATIC=0.80
# leaves outside the static pool (verified: 1M-context needle test passed on TP4). Use
# scripts/verify/memguard.py when ramping new prompt shapes, and drop the chunk to 512
# if the guard fires. 0 disables the hook.
DSV41_PREFILL_EMPTY_CACHE_TOKENS=8192

# ─── TP4 runtime ──────────────────────────────────────────────────────────────
# Memory per rank at TP4: ~77 GiB of MXFP4 experts + FP8 dense weights and a
# ~2.5 GiB draft, so ~40 GB of the 121.7 GiB stays free (vs ~6 GB at TP3). That
# pays for the settings above: the full 1M context, an 8M-token KV pool
# (1,670.75 B/token/rank = 13.4 GB), 8 concurrent requests, 1024-token prefill
# chunks (~15 GB peak indexer transient at 1M), and a 0.80 memory fraction that
# keeps ~24 GiB per rank outside the static pool for that transient. Packed Engram
# shards are named engram-l<layer>-r<rank>of4.bin and can share ENGRAM_DIR with a
# 3-Spark profile. At TP4 every shard holds real weights, so no head padding.
DSV41_TP_PAD=0
WORKER_DIR=/home/$WORKER_USER/dsv41-4x-spark

# ─── Extra container env (mirrored to every rank by start.sh) ───────────────
# DSV41_INDEXER_CHUNKED=1 arms the sglang#39187 backport (adapter/indexer_chunked*.py):
# the dense prefill indexer scores each request in row chunks of at most
# DSV41_INDEXER_LOGITS_BUDGET_BYTES (default 2 GiB) of fp32 logits and publishes only
# the tail rows of the candidate masks. That bounds the per-chunk transient that made
# CHUNKED_PREFILL_SIZE=1024 necessary (docs/chunked-prefill-memory.md), so the 4096
# chunk above is safe at any context: measured head MemAvailable low-water 7.0-7.7 GiB
# during a 262k cold prefill on both images (2026-09-16 window, knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4).
# adapter/fast_load.py (docs/fast-load.md): this rank's checkpoint tensors are read
# eagerly by a 16-thread pread pool into anonymous host memory instead of being
# page-faulted through the loader's mmap at 0.5 GB/s, copies are paced to
# DSV41_FAST_LOAD_INFLIGHT_GB so the reads stay just ahead of them, and the DSpark
# draft load opens only the 3 shards that hold mtp.* tensors. load_weight 285 s -> 77 s,
# engine start 356 s -> 125 s, values bitwise identical (tests/test_fast_load_pacing.py,
# and the in-image checkpoint test described in the docs). EP_SIZE / N_EXPERTS are
# fallbacks for when the parallel context is not readable from the loader.
# The eager tensors are views into 256 MiB pinned slabs (DSV41_FAST_LOAD_SLAB_MB, 0 = one
# pinned buffer per tensor); DSV41_FAST_LOAD_MEMLOG=1 logs host counters and pinned
# allocation counts at the end of each load (diagnostics only).
#   DSV41_FAST_LOAD=1 DSV41_FAST_LOAD_EP_SIZE=2 DSV41_FAST_LOAD_N_EXPERTS=384 DSV41_FAST_LOAD_INFLIGHT_GB=6
# adapter/engram_prefetch.py: the Engram row lookups (NVMe misses inside a host callback,
# 2.2-2.4 ms of GPU idle per step on real text) run on a side stream right after the
# hash ids exist and join only at the layer's gather. Same-image A/B 2026-09-18:
# step 51.7 -> 49.5 ms, English essay c1 42.6 -> 45.1 tok/s, sparkDash prose c1 57-59 ->
# 60-61, prose c4 120 -> 124, structured 116 -> 121; bitwise identical rows
# (DSV41_ENGRAM_PREFETCH_CHECK=1 re-runs the old path and counts differences: 0).
#   DSV41_ENGRAM_PREFETCH=1
# adapter/wo_a_w8.py: the verify/draft wo_a projection (2-8 rows) reads the checkpoint's own
# e4m3 bytes plus one exponent per 32x32 block instead of the bf16 copy the engine makes at load,
# in a copy of the stock Triton kernel that rebuilds the same bf16 tile in registers. A layer is
# converted only when its fp8 twin reproduces the bf16 weight exactly (43 of 43 here). 43 layers
# at M=6: 3.43 -> 2.20 ms; decode step at c1 52.9 -> 51.7 ms; the MXFP8-epilogue path is bitwise
# identical, the plain bf16 path differs in fp32 accumulation order only.
#   DSV41_WO_A_W8=1
# adapter/draft_tau.py: draft proposal temperature for sampled requests. Proposal and acceptance
# use the same tempered q, so the output distribution is unchanged (speculative sampling is exact
# for any q); at the model card's T=1 / top_p=0.95, 0.8 raises accepted tokens per step by ~1.2 %
# (offline, held-out traffic). Needs SGLANG_DSPARK_FOLDED_SAMPLING=2.
#   DSV41_DRAFT_TAU=0.7
# adapter/draft_head_fp8.py: the draft reads an fp8 copy of the (shared) LM head; the target's
# logits are untouched, so outputs cannot change. 1405 -> 721 us per step, acceptance unchanged.
#   DSV41_DRAFT_HEAD_FP8=1
# DSV41_DRAFT_HEAD_FP8_IMPL=tp4 selects adapter/draft_head_fp8_tp4.py: row tiles sized to the batch
# (16/64/128), so the fp8 head is streamed once per call; same logits per row, faster at 40-64 rows.
#   DSV41_DRAFT_HEAD_FP8_IMPL=tp4
# adapter/block_verify.py: block verification (Sun et al., ICLR 2025) instead of token-by-token
# rejection for sampled rows; exact output distribution (chi-square test in tests/), more tokens
# accepted per step: +1.8 % on prose, +2.5 % on coding-with-thinking (engine-exact p and q).
#   DSV41_BLOCK_VERIFY=1
# adapter/folded_result_fence.py: folded all-greedy verify results are cloned off the verify
# graph's persistent buffers before the overlapped copy to host (sglang#40919 race).
#   DSV41_FOLDED_FENCE=1
# adapter/verify_cap.py: per-request verify length. conf:T verifies only the leading drafts whose
# running product of the draft confidence head's survival stays >= T; the other verify rows reuse
# the anchor row's experts (no extra expert reads, ~2 ms per row at c1) and acceptance is capped at
# the verified drafts. Exact (greedy identical, sampled via block_verify). conf:0.1 measured best:
# prose c1 +5 %, sampled thinking +5 %, code flat. 1..5 = fixed verified drafts (measurement only).
#   DSV41_VERIFY_CAP=conf:0.1
# adapter/autotune_keep.py: keep the FlashInfer autotune cache across boots under EP (sglang#40320).
#   DSV41_AUTOTUNE_KEEP=1
# adapter/wo_a_w8.py, two more switches (need DSV41_WO_A_W8=1): MID runs verify/draft wo_a at 9-192
# rows (c2..c32) from the fp8 twin too; DROP then releases the bf16 copy (722 MB per rank -> KV pool).
#   DSV41_WO_A_W8_MID=1 DSV41_WO_A_W8_DROP=1
# adapter/row_store.cpp: keep one Engram layer's row cache in the GB10 display reservation (outside
# MemAvailable; host needs nvidia_drm modeset=1 fbdev=0 and no desktop, see docs/display-reserve.md).
#   DSV41_ENGRAM_DRM_NODE=/dev/dri/card0
# adapter/replicated_split.py: ReplicatedLinear layers whose prefix ends with one of the listed
# suffixes are column-split over TP (each rank runs the same quantized linear on its 128-row weight
# tiles, then an all-gather). Per layer and only if the slice is bit-identical to the stock layer on
# every rank (checked at boot). engram.wkv (183 MB read by every rank at both Engram layers) and
# wqkv_a: c1 step -1.4 ms, prose c1 +3.7 %, greedy identical.
#   DSV41_REPLICATED_SPLIT=wqkv_a,engram.wkv
# adapter/router_live.py (needs DSV41_VERIFY_CAP): the router kernel loads the anchor row's scores for
# dead verify rows, so they get the anchor's experts without the separate remap launch (40 per step);
# built from the engine's own router source, bit-identical outputs.
#   DSV41_ROUTER_LIVE=1
# adapter/draft_main_proj.py: the draft's replicated main_proj (105 MB fp8 on every rank) as a
# 1/TP column shard (fp8, exact weight bytes) + all-gather; draft only, target untouched.
#   DSV41_DRAFT_MAIN_PROJ_SPLIT=1
# adapter/roce_gather.py (canary-roce image): TP all-gathers up to N bytes per rank (the draft's
# vocab-parallel logits, the splits above) take the RoCEnante one-shot kernel instead of NCCL.
#   DSV41_ROCE_GATHER=2097152
EXTRA_CONTAINER_ENV="DSV41_INDEXER_CHUNKED=1"

# The dsv4.1 branch (canary images) probes a Rust toolchain to build its image
# preprocessor and can hang in `cargo --version` before the HTTP server starts
# (seen 2026-09-17: launch_server stuck 20+ min, workers up, /health 000).
# `never` skips Cargo and keeps the PIL path. Harmless on the base image.
SGLANG_RUST_BUILD_MODE=never

# ─── Optional: upstream dsv4.1 branch image (Dockerfile.canary) ──────────────
# ./start-tp4.sh build builds it on every node with BUILD_DOCKERFILE below (the
# pinned branch tree is fetched and hash-checked during the build; #BUILD_ARGS passes
# extra docker build flags, e.g. --build-arg PIP_INDEX=<mirror>/simple without
# pypi.org). Then point IMAGE at it and force the graph-folded draft sampler
# (greedy-folded by default):
#IMAGE=dsv41-4x-spark:canary
#BUILD_DOCKERFILE=Dockerfile.canary
#EXTRA_CONTAINER_ENV="DSV41_INDEXER_CHUNKED=1 SGLANG_DSPARK_FOLDED_SAMPLING=2 SGLANG_RUST_BUILD_MODE=never"

# ─── Optional: canary + RoCEnante (Dockerfile.canary-roce) ─────────────────────
# Routes TP SUM all-reduces up to SGLANG_ROCE_MAX_SIZE bytes (2 MiB covers the
# 16-request decode step) to the b12x one-shot RDMA all-reduce over both rails.
#IMAGE=dsv41-4x-spark:canary-roce
#BUILD_DOCKERFILE=Dockerfile.canary-roce
#EXTRA_CONTAINER_ENV="DSV41_INDEXER_CHUNKED=1 SGLANG_DSPARK_FOLDED_SAMPLING=2 SGLANG_ROCE_ALLREDUCE=1 SGLANG_ROCE_MAX_SIZE=2097152 B12X_ROCE_HCA=rocep1s0f0,roceP2p1s0f0 B12X_ROCE_CACHE_DIR=/state/b12x-roce B12X_COMPILE_CACHE_DIR=/state/b12x-compile SGLANG_RUST_BUILD_MODE=never"
# Prefill TP split (SG18) on top of either canary image, from 32k context:
#EXTRA_CONTAINER_ENV="DSV41_INDEXER_CHUNKED=1 SGLANG_DSPARK_FOLDED_SAMPLING=2 SGLANG_ROCE_ALLREDUCE=1 SGLANG_ROCE_MAX_SIZE=2097152 B12X_ROCE_HCA=rocep1s0f0,roceP2p1s0f0 B12X_ROCE_CACHE_DIR=/state/b12x-roce B12X_COMPILE_CACHE_DIR=/state/b12x-compile SPARK_PREFILL_TP_SPLIT=1 SPARK_PREFILL_TP_MIN_CONTEXT=32768 SPARK_PREFILL_TP_MIN_ROWS=1024 SGLANG_RUST_BUILD_MODE=never"
# The full production profile of docs/tp4.md (canary-roce image, EP_SIZE=1 above):
#EXTRA_CONTAINER_ENV="DSV41_INDEXER_CHUNKED=1 SGLANG_DSPARK_FOLDED_SAMPLING=2 SGLANG_ROCE_ALLREDUCE=1 SGLANG_ROCE_MAX_SIZE=2097152 B12X_ROCE_HCA=rocep1s0f0,roceP2p1s0f0 B12X_ROCE_CACHE_DIR=/state/b12x-roce B12X_COMPILE_CACHE_DIR=/state/b12x-compile SPARK_PREFILL_TP_SPLIT=1 SPARK_PREFILL_TP_MIN_CONTEXT=32768 SPARK_PREFILL_TP_MIN_ROWS=1024 SGLANG_RUST_BUILD_MODE=never DSV41_FAST_LOAD=1 DSV41_FAST_LOAD_EP_SIZE=1 DSV41_FAST_LOAD_TP_SLICE=auto DSV41_FAST_LOAD_N_EXPERTS=384 DSV41_FAST_LOAD_INFLIGHT_GB=6 DSV41_ENGRAM_PREFETCH=1 DSV41_SHARED_PAD_K=1 DSV41_WO_A_W8=1 DSV41_DRAFT_TAU=0.7 DSV41_DRAFT_HEAD_FP8=1 DSV41_DRAFT_HEAD_FP8_IMPL=tp4 DSV41_BLOCK_VERIFY=1 DSV41_FOLDED_FENCE=1 DSV41_VERIFY_CAP=conf:0.1 DSV41_AUTOTUNE_KEEP=1 DSV41_WO_A_W8_MID=1 DSV41_WO_A_W8_DROP=1 DSV41_REPLICATED_SPLIT=wqkv_a,engram.wkv DSV41_DRAFT_MAIN_PROJ_SPLIT=1 DSV41_ROCE_GATHER=2097152 DSV41_ROUTER_LIVE=1 DSV41_MOE_B12X_NEXT=1 DSV41_MOE_B12X_NEXT_DETERMINISTIC=1 DSV41_HC_FUSED=1 DSV41_PREFILL_SP=1 DSV41_PREFILL_SP_FP8=1 DSV41_L2_PREFETCH=1"