# Greedy decoding is not deterministic at long context

GLM-5.3-Flash-NVFP4, vLLM `0.1.dev20051+g487ecf187`, 4x GB10 (sm121) at TP=4.

## What happens

At `temperature=0` with a fixed seed, identical requests sometimes return
different completions. Most differences are invisible. The one that reaches a
user is a corrupted **tool name**: `glm47_moe` sets `validate_tool_names=True`,
so a call whose name is not in the request's tool list emits **zero deltas** and
finishes `stop` with no content and no tool calls. Clients report it as "model
returned a completed response with no content" and retry it as a dead endpoint.

## Cause

**The marlin NVFP4 MoE is not bit-deterministic at M >> 1**, and the model
amplifies that into whole-token differences.

Measured, in order:

1. Among runs that answer *correctly* at the failing position, P(correct token)
   ranges 1.00 down to 0.48 -- 35 distinct values in 38 runs. A wrong token is
   the tail of that spread, not a separate failure mode.
2. Per-layer tap across identical runs: layers 0-2 and layer 3's attention input
   are bit-identical; layer 3's attention output takes 12 distinct values.
3. Fixing the FA2 input order makes layer 3 bit-identical, and the first
   divergence moves to layer 4 with identical input -- so it arrives in the KDA
   recurrent state, which prefill produced.
4. Prefill tap: layer 3's `mlp_out`, the first marlin MoE, differs in 8 of 8
   runs at M=800.
5. In-engine, evaluating that MoE block twice on the same input differs at
   M=800 and M=2304, and is bit-stable at M=1.
6. Padding the prompt so the uncached recompute is a single row makes decode
   bit-identical across 11 runs, spread 0.000000.

Amplification is what turns ulp noise into a different token: 288-expert
routing plus eleven sparse selections per token. With layer 3 pinned, the
indexer's 512th logit still spreads by 1-30 units by layer 39.

## Do not use the prefill re-score as an oracle

An earlier version of this document treated a token's logprob under
`prompt_logprobs` as ground truth and read a 13-21 logprob gap as proof that
decode was wrong. It is not: the re-score is its own multi-row prefill, so it is
another draw of the same nondeterministic computation, taken at n=1. It shows
that two draws disagreed, not which one is right.

## Ruled out, by measurement

Top-k selection, both kernels: values match `torch.topk` exactly at every
coarse-bin population up to a whole row in one bin, and real indexer logits hold
exact boundary ties in 3 of 34,320 layer-rows, none at the layer where
divergence enters. The FA2 sparse-MLA kernel (exact, bit-repeatable). DeepGEMM
MQA logits (bit-deterministic; paged equals non-paged). CUDA graphs (eager
reproduces). Speculative decoding (off reproduces). KV dtype, memory pressure,
prefix caching, temperature, conversation history, the request payload, the
uninitialised `topk_indices_buffer`, the MoE router's Python fallback (the fused
kernel runs here), and TP/NCCL (ranks agree bitwise).

Consequently neither `hpc.topk_filtered` (Tencent/hpc-ops#93) nor the upstream
sm121 determinism set -- vllm#55122, the MoE finalize knob, vllm#53899 -- touches
this. `use_fused_finalize` belongs to the FlashInfer CUTLASS path; the backend
here is marlin.

## Reading the rates

Per-condition rates ranged 2.5%-22.5%, including the same configuration twice at
13% and 22.5%. At n=40 nothing in that spread is distinguishable, and comparing
conditions on those numbers cost a night. Size the sample from the expected rate
first, or measure determinism directly with a per-layer tap, which decides on a
single pair of runs.

## Writing a tap

Three ways it silently produces nothing, all of them paid for:

- `GLM53_TAP` in the container does not reach the worker: mentat spawns actors
  without forwarding it. Gate on a file under `/logs` instead.
- Gating on `t.shape[0] == 1` to exclude prefill excludes all of decode, because
  speculative decode carries 1 + draft tokens per step.
- `.item()` inside the forward invalidates CUDA graph capture and crash-loops the
  engine. Accumulate into a preallocated device tensor, and keep the step counter
  on device too: host code does not run on graph replay, so a Python counter
  freezes at its capture value.

## The MoE kernel is not the whole story

2026-09-13, NVIDIA's ModelOpt NVFP4 checkpoint with `MOE_BACKEND=flashinfer_cutlass`
(which does run on sm_121 -- the entrypoint's `cudaErrorNoKernelImageForDevice`
warning and the upstream issues are about auto-selection skipping CUTLASS, not
about the kernel being absent):

| condition | distinct completions from 16 |
|---|---|
| marlin, RedHat weights, content only, 48 tokens | 3 |
| CUTLASS, NVIDIA weights, content only, 48 tokens | 9, sizes [6,2,2,1,1,1,1,1,1] |
| CUTLASS, NVIDIA weights, reasoning included, 400 tokens | 16 |

Replacing the kernel this document blames did not restore reproducibility, and
under matched conditions it is worse. Two variables differ between the first two
rows, kernel and checkpoint, so the attribution needs marlin on the NVIDIA
weights -- one flag, the same boot -- before the kernel is cleared or convicted.
One sample of 16 each, and the sizing warning above applies.

The third row is not comparable to the others: it hashes the reasoning trace as
well as the content, which finds divergence a 48-token content hash cannot. That
is the better test of the forward pass and the worse test for comparing against
the baseline.

## Open

A deterministic NVFP4 MoE at M >> 1 on sm121, in any backend. The
single-row-recompute trick removes the variance but only for prompts whose
uncached tail is one token.

## Reproducing

`repro/` holds three scripts and what each one measures. `repro/greedy_nondet.py`
is the small one: standard library, one repeated paragraph, 3 distinct
completions from 16 identical greedy requests at 42k and 1 from 16 at 2k.