# Working on this repo

This is the recipe I run GLM-5.3-Flash with on four GB10 boxes. People copy it,
so a change that makes it slower or subtly wrong costs more than the change
was worth. Most of what follows is about proving a change is neither before it
lands.

If you're an agent working here, read this whole file first. If you're a
person, the same rules apply to you.

## How changes land

Every change goes up as a pull request against `main`. One change per PR,
with the gate results in the description. I merge.

Before anything leaves your machine, scrub it. No IP addresses, hostnames, box
names, usernames or paths from your own setup. The compose files use relative
paths and `.env` values for a reason.

Don't push to a PR branch until the change passes the gate below. Pushing
something "to see what CI thinks" doesn't work here: there's no CI that can run
this model, and the PR is the record of what was measured.

A docs-only change doesn't need the gate. A setup fix that helps someone
starting fresh (an entrypoint guard, a `.env.example` comment, a
troubleshooting entry) needs its own logic tested, not the speed suite.

## The gate

A change passes when:

1. every quality check below is inside its bar, and
2. prefill and decode are at par or faster than the last full gate, on
   repeated samples, not one run.

Run it on the head node, from the repo root, with the stack up and nothing
else using it:

```
gate/run.sh full my-change
```

`gate/run.sh quick my-change` is the reduced gate. It skips HumanEval, the
count probe, RigMark and half the needle and prefill samples. It's fine for
checking your own work as you go. It's only enough for a PR when I've said
so for that change.

The full gate takes a couple of hours. GSM8K and HumanEval need their data in
`QUALITY_DIR` (see `experimental/quality/quality.py`). RigMark needs a checkout
of [RigMark](https://github.com/alexellis/rigmark) at c5a0db0 in
`RIGMARK_DIR`. The gate skips it without one, and then the PR isn't done.

### Quality

| Check | Script | Bar |
|---|---|---|
| Smoketest | `smoketest/run.sh` | 8/8 |
| NLL on three fixed texts | `gate/nll.py compare` | notes +0.008 to +0.025, the others within ±0.007 |
| GSM8K, first 250 | `experimental/quality/quality.py` | 95.5 to 98.5% |
| HumanEval | `quality.py`, then `run_he.py` with no network | 154 to 156 of 164 |
| Count to 200, 5 runs | `gate/count.py` | 0 corrupt |
| Tool call at 42k context, 40 runs | `dev/repro/toolcall_corruption.py` | 0 diverging on the last gate (stock: 10) |
| Agent tools | `experimental/quality/agent_tools.py` | 10/10, 0 bad args |
| Needle, 32k to 480k | `dev/repro/needle.py` | 12/12 |
| RigMark output gates | RigMark `bench.py` | 6/6 |

The bars are wide because greedy decoding here isn't reproducible across
boots. FlashInfer's autotuner picks different kernels on each boot, the
rounding changes, and borderline answers flip. HumanEval has about a dozen
problems that pass or fail from boot to boot, and
`experimental/quality/he_detail.py` shows which ones flipped between runs. GSM8K runs 8 requests at once, so
batching changes run to run too: the same config on the same boot has scored
96.4 and then 98.4.

So one number near an edge proves nothing. A real regression either falls well
outside a bar or moves several checks together. The last one I caught (NVFP4
for the attention projections) took GSM8K to 95.6 and HumanEval to 154 at
once. When you're unsure, compare the two paths inside one boot, where
generated text is byte for byte deterministic.

NLL isn't. `nll-ref.json` was recorded with the dense layers left in bf16
(`fp8.yaml` left out), and FP8 dense moved the notes text up by about 0.015.
Since then it has read anywhere from +0.008 to +0.025, sometimes 0.006 apart
on the same boot. Watch the other two texts, and watch for all three moving
the same way. To add a text, drop it in `gate/texts/`, boot without
`fp8.yaml`, and record a new reference with `gate/nll.py record`.

### Speed

| Measure | Script | Samples |
|---|---|---|
| Prefill, 32k and 128k, cold | `gate/prefill.py` | 4 (mean) |
| Decode, structured / code / prose | `gate/decode.py` | median of 3 |
| Decode, older prompt set | `gate/decode_stream.py` | median of 3 |
| Concurrent streams 1 to 16, mixed prompts | `gate/conc.py` | 1 |
| Preemptions | `/metrics` | must stay 0 |

Par means within the noise of the last full gate: about 0.5% for prefill and
1 to 2% for decode. Anything lower is a loss, even if it's small. I would
rather not have a change than carry a 1% regression forever.

The last full gate (2026-09-28):

| | |
|---|---|
| Prefill 32k / 128k | 4,946 / 4,750 tok/s |
| Decode structured / code / prose | 170.3 / 120.8 / 65.8 tok/s |
| Decode, older prompts | 162.2 / 127.6 / 89.2 tok/s |
| Concurrent 1 / 2 / 4 / 8 / 16 | 125 / 103 / 146 / 192 / 251 tok/s |
| RigMark code / prose / structured | 107.9 / 61.7 / 157.1 tok/s |

When your PR lands with better numbers, update this table in the same PR.

## kindling.json

`kindling.json` at the repo root is the README's results table in a form a
program can read. The Kindling AI site reads it from `main`, so whatever is
merged there is what people see.

Keep it in step with the README:

- When a PR changes the README results table, change `kindling.json` in the
  same PR. Same numbers, same rounding. A PR that changes one and not the
  other isn't done.
- It mirrors the README table, not the last-full-gate table above. The gate
  table moves with every PR; the README is what we publish.
- One entry in `configs` per TP column. Leave out a number that wasn't
  measured. Don't write 0, and don't fill a gap with an estimate.
- Check it parses before you push: `python3 -m json.tool kindling.json`.

The fields:

| field | what |
|---|---|
| `title`, `variant`, `summary` | the card heading, the quant tag, one or two sentences |
| `model`, `hardware`, `stack` | the checkpoint, the boxes, a few short tags |
| `default_config` | the `id` of the config shown first |
| `configs[].id`, `label`, `nodes` | e.g. `tp4`, `TP=4`, `4` |
| `configs[].decode` | single-stream decode tok/s by workload: `code`, `prose`, `structured` |
| `configs[].prefill` | cold prefill, `[{ "context": tokens, "tps": tok/s }]` |
| `configs[].concurrency` | aggregate tok/s, `[{ "streams": n, "aggregate_tps": tok/s }]` |
| `configs[].max_context`, `kv_pool_tokens`, `max_concurrent` | integers: tokens, tokens, requests |
| `configs[].quality` | `[{ "name": ..., "value": ... }]`, shown as short text |
| `configs[].notes` | one line of caveats |

Context lengths are in tokens: 32k is `32768`, 128k is `131072`.

## Extra checks for some changes

The gate covers the common cases. Some changes need more.

Kernels and anything else that changes numerics get a bit-exact or
tolerance test in `dev/patch-tests/` or `dev/kernel-tests/` first, against the
path they replace. Say in the PR which one it is and why a tolerance is safe,
if you needed one.

Anything that touches scheduling or runs differently with more than one
request gets the concurrency sweep too. `gate/conc_workload.py` runs one
workload (code, prose, structured or mixed) at the concurrency levels you give
it, each stream with its own prompt:

```
for w in code prose mixed; do python3 gate/conc_workload.py $w 1 2 4 8 16 32 64; done
```

Report it against the same sweep on `main`. The targets I care about are 155
tok/s at 4 streams and 350 at 16, on the mixed set.

Memory and KV changes need the boot log's `GPU KV cache size` line before and
after, and a run at `MAX_NUM_SEQS` streams with vLLM's `Running:` count, to
show they all fit.

Anything that changes how weights are processed needs a bump of
`SNAPSHOT_VERSION` in `experimental/snapshot/weight_snapshot.py` (code) or a
new `VLLM_WEIGHT_SNAPSHOT_TAG` (environment knobs), or a boot without
restoring. A shape change fails the restore loudly. A same-shape change
restores stale weights and says nothing.

## Things that have cost a day

Measure prefill on cold prompts. A repeated prompt measures the prefix cache:
a 200k probe that took 198 s cold came back in 6 s warm. `gate/prefill.py`
draws fresh text for each run.

Count tokens, not stream events. With speculative decoding one event carries
several tokens, and events per second undercount decode badly.

Keep the FlashInfer autotune cache ephemeral. If the four ranks read caches
that differ, TP=4 deadlocks.

Don't benchmark on a box with a desktop session running. It shares the
unified memory and the power budget with the model, and the entrypoint warns
about it.

A worker at `ROLE=head` crash-loops on mentat's "group already has an active
driver session" while the real head waits for GPUs. The entrypoint refuses to
start that way now, but it's the first thing to check when a boot hangs.