kilork

kilork pushed to main at sm121/recipes

d74b8f37af glm53-mxfp4-fp8-4x-r39: KV pin 7.5 GiB for the 131072 window (6.68 GiB needed, 6 GiB pinned -> refused)

2 hours ago

kilork pushed to main at sm121/recipes

559e8d4db9 glm53-mxfp4-fp8-4x-r39: DSpark speculator attempt recorded — dies in the draft KV builder on r39 (needs vLLM 0.32) RedHatAI/GLM-5.3-speculator.dspark (2.5 GB, 3-layer Qwen3 draft, hidden states from target layers 2/20/39/58/75, up to 8 drafts, 1M ctx, 41% acceptance / 4.29 accepted length at 524K-1M) is the right drafter for this model, but on our r39 fork it passes config validation and draft-backend selection and then dies inside qwen3_dflash._build_context_kv_buffers while loading draft weights. The speculator targets vLLM 0.32 / speculators 0.7.x — retry on a vLLM-main image. Lane restored to the working MTP k=3 at 131072 context.

10 hours ago

kilork pushed to main at sm121/recipes

35de736d83 glm53-mxfp4-fp8-4x-r39: 196K window hung the host on a 150K prefill — back to 131072 / 6 GiB KV / 2048 chunks A ~150K prompt blew the RPC budget (EngineDead, HTTP 500) and then thrashed dgx6+dgx7 (ping alive, sshd starved) — an HA plug cycle was needed. The KV pool was never the limit: the prefill scratch does not fit the ~11 GiB left after an 11 GiB pin behind 98.86 GiB of weights. 131072 with a 6 GiB pin and 2048-token chunks leaves ~16 GiB for scratch and is the configuration this lane keeps.

11 hours ago

kilork pushed to main at sm121/recipes

baabab7674 glm53-mxfp4-fp8-4x-r39: DCP4 rejected by every MLA backend ('DCP not supported'); window now 196608 at a 10 GiB KV pin Measured on both forks (jovian r39 and spark-glm53-karmic, block 64 and 256): explicit FLASHINFER_MLA_SPARSE_SM120 + DCP4 -> 'DCP not supported'; with auto-selection no backend qualifies at all. The DSA sparse MLA path has no DCP implementation, so the 512K/1M windows of the Flash sibling are out of reach for this checkpoint on 4 ranks. Window is therefore KV-pin-bound only: 52.7 KB/token/rank, ~20 GiB left after weights -> 196608 at 10 GiB (this lane), ~304K would need TP8.

13 hours ago

kilork pushed to main at sm121/recipes

866d05a06c glm53-mxfp4-fp8-4x-r39: MTP k=5 measured — same throughput as k=3 (33-37 c1, 47.9 c2), acceptance 4.4/5 Verification, not acceptance, is the limit: 87-93% of drafts are accepted at depth 5 and the ladder does not add t/s over depth 3. Both double the un-speculated decode (19.0 -> ~35 t/s).

14 hours ago

kilork pushed to main at sm121/recipes

8efe2f3d82 glm53-mxfp4-fp8-4x-r39: MTP k=3 breaks the decode ceiling — 19.0 -> 33-35 t/s c1, 47.6 t/s c2 The full model is bandwidth-bound at ~99 GiB of weights per rank, so kernel swaps did nothing (karmic b12x MoE: 19.2). Speculation does: MTP k=3 (head is in the checkpoint, no drafter loaded) gives acceptance length 3.2-3.6 of 3, decode 33.4-34.9 t/s single-request and 47.6 t/s aggregate at c2. Weights 96.34 -> 98.86 GiB/rank, window drops to 32768 to pay for the draft cache. Note also records the measurement caveat: with speculation SSE chunks bundle several tokens, so count usage.completion_tokens, not chunks.

14 hours ago

kilork pushed to main at sm121/recipes

75276e8670 glm53-mxfp4-fp8-4x-karmic: A/B vs r39 — native b12x MoE experts buy +1% (19.2 vs 19.0 t/s) Same RedHat checkpoint, same cluster, same args. karmic's B12X_MXFP4_MXFP8 expert kernel vs r39's MarlinExperts: decode is bandwidth-bound at 96 GiB per rank, so the Marlin A16 dequant was never the limiter. MXFP4 linears stay on Marlin in both (--linear-backend b12x does not change the selection for these shapes). Karmic-only boot requirements recorded: --block-size 64 (256 gives a padded BLNHC layout the engine refuses) and enable_jit_warmup=false (the b12x tuning cache binds a temporary padded KV pool and trips the MLA bind assert).

21 hours ago

kilork pushed to main at sm121/recipes

1c4d95db9d glm53-mxfp4-fp8-4x-r39: measured on v2.8.0 — 88/100 (170/194), Hard Mode 46/56, 1 safety-critical, 8.8 s median turn The full model buys no quality over the GLM-5.3-Flash 2x lane (95/100, 3.6 s) while costing 28 GiB more per rank and 2.4x the turn time at 64K context with FP4 linears on Marlin. NOTE.md records the boot recipe, the four failed configurations and what is left to try (MTP k=3, native b12x FP4 GEMM, DCP4).

1 day ago

kilork pushed to main at sm121/recipes

76abd98c9c glm53-mxfp4-fp8-4x-r39: GLM-5.3 (full) RedHat MXFP4xFP8 boots at TP4 on four Sparks Working boot recipe after four attempts, each failing differently: - kindling spark-glm53:v10: multi-node worker init dies (gloo is_in_the_same_node) - jovian r39 + --attention-backend B12X: TypeError DeepseekV32B12xAttention got unexpected 'index_group_builder' (re-cut indexer API newer than r39's class) - r39 + --linear-backend marlin: prepare_fp8_layer_for_marlin on ReplicatedLinear -> AttributeError output_size_per_partition - r39 default dispatch: FlashInfer mm_fp4 rejects capability 121 (cute-dsl) FIX: VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel — MXFP4 linears fall to Marlin, fp8_block layers keep their own backend. Loads 96.34 GiB/rank, KV 79,616 tokens at 65536 ctx, ~19 t/s decode single request. Plus the eugr/vllm-main fallback recipe (image verified: vLLM 8244d1c44, compressed-tensors 0.17.0 with mxfp4/mxfp8 schemes, GlmMoeDsa registered).

1 day ago

kilork pushed to main at sm121/recipes

fc210f50bd glm53-kindling-2x: reproducibility run on v2.8.0 — 95/100 (185/194), Hard Mode 52/56 (93%), zero safety-critical, 3.6 s median turn Second independent run of the same lane (94/100 the day before) and the exact reproduction of the 2026-10-15 185/194. Best quality on the fleet, on two nodes.

1 day ago

kilork pushed to main at sm121/recipes

817cb85d66 glm53-mxfp4-fp8-4x: GLM-5.3 (full, 78 layers) RedHatAI MXFP4xFP8 on 4 Sparks at TP4 415.4 GB / ~4.5 bits per param -> ~97 GiB of weights per rank in a ~119 GiB unified pool: a first boot with a 32K window and 4 sequences, raise after the pool is measured. Engine spark-glm53:v10 (kindling fork) already registers GlmMoeDsaForCausalLM and the glm47 parsers the checkpoint wants; MTP k=5 per the RedHat card. Staged on the free stand (dgx5-8).

1 day ago

kilork pushed to main at sm121/recipes

337f442e74 dsv41-flash-1017-2x-tensorfold: session cache A/B — same 89/100, median turn 6.7s -> 4.2s (responsiveness 23 -> 38) Same harness, same recipe, only TF_DSV41_SESSIONS flipped; the isolated 8K-prefix check shows the effect clearly (74.5 s cold vs 4.4 s on a repeated prefix). Fresh-context prefill stays ~4-5 s per 2K, so the lane keeps its batch/long-gen shape. Staging fixes from the dgx7-8 bring-up recorded in NOTE.md.

1 day ago

kilork pushed to main at sm121/recipes

994a1ffa05 dsv41-flash-1017-2x-tensorfold: staging script moves into the recipe (scripts/prepare.sh) + TF_DSV41_SESSIONS=1 experiment Host scripts that know about a recipe belong in the recipe repo: the launcher now resolves { script = "scripts/prepare.sh" } relative to the recipe dir and runs it with NODE_RANK (serve-local rank), so the staging code versions with the recipe. prepare.sh also gained three fixes from the dgx7-8 bring-up: the model rev resolves from refs/main (a re-cut snapshot no longer breaks the tar), the Engram packer runs in the image python (host python is PEP-668 managed), and the packer gets the whole HF hub mounted (snapshot -> blob -> hub-blob symlinks). Session cache is ON for the A/B (TF_DSV41_SESSIONS=1): a second question over the same 8K prefix took 4.4 s vs 74.5 s cold.

1 day ago

kilork pushed to main at sm121/recipes

7e9da804f3 glm53-kindling-2x: re-benchmarked on harness v2.8.0 — 94/100 (183/194), Hard Mode 50/56 (89%), zero safety-critical failures Run on dgx5-6 (dgx58-switch, 2 nodes, TP2, DFlash2 + adaptive-k) after the 2026-10-21 image drift (spark-glm53:v10 label 62737b… / 29c473… vs the recipe dir's 8e60… — serve warns but still starts the existing build). Same harness as the fresh baselines, so directly comparable: glm53-kindling-2x 94/100, Hard Mode 89%, 0 safety-critical, turn 3.5 s beam-1.0-4x 92/100 (90-92), Hard Mode 86-95%, 3 safety-critical, 2.6 s 1017 sglang 4x 86-87/100, Hard Mode 75%, turn 1.4 s The 2-node GLM lane is the strongest quality result on the fleet and leaves two nodes idle.

1 day ago

kilork pushed to main at sm121/recipes

b1f054231b beam lane: 384K context (13.6x concurrency) + needle-in-haystack at 131K and 327K tokens (both correct, 60s/175s TTFT)

1 day ago

kilork pushed to main at sm121/recipes

502080b104 beam lane: depth sweep (31-35 tok/s to 128K) + the second/third quality runs (92/100, Hard Mode 95%)

1 day ago

kilork pushed to main at sm121/recipes

3248479cdf beam lane: real-workload TTFT measured (0.41/0.51 s with prefix cache)

2 days ago

kilork pushed to main at sm121/recipes

a3c43d9bb1 beam lane: spec-k7 quality/perf reports + real-work operational contract Quality with DFlash2 k=7 is unchanged (90/100, 175 pts, Hard Mode 48/56) while responsiveness scores 56 (2.6 s median turn) vs 34 (4.7 s) without speculation. Operational findings recorded for real use: thinking counts against max_tokens (a 4.7K-token doc-QA request burns 600-900 reasoning tokens, so max_tokens 600 returns empty content with finish_reason=length — use >= 1500); reasoning_effort is a prompt-side hint that does not reliably shorten thinking on hard tasks; three verifiable long-context questions answered correctly in 15-17 s.

2 days ago

kilork pushed to main at sm121/recipes

d135e829d3 beam lane: DFlash2 speculation works with FLASHINFER — +40% decode, responsiveness 34->56 The drafter is constrained from both sides on GB10: FLASH_ATTN rejects fp8 KV (FA3/FA4 only), B12X rejects non-causal attention, and the drafter code itself asserts attn_backend in (FLASH_ATTN, FLASHINFER) — but FLASHINFER + fp8 KV boots. Adaptive verification is unavailable on GB10 (both FA and FLASHINFER refuse the varying query lengths it needs), so it is forced off. Measured (llama-benchy pp2048 tg128, same harness): tg 26.2 -> 36.8 at d0 c1 (+40%), 41.1 -> 55.1 at c2, and it holds at d4096/d8192. Quality unchanged at 90/100 (175 points, Hard Mode 48/56 = 86%, Safety 92%), while the median turn drops 4.7 s -> 2.6 s and responsiveness scores 34 -> 56. Cost: KV pool 6.55M -> 5.27M tokens. Also documents the per-request reasoning budget (chat_template_kwargs. reasoning_effort, default 512=max) — the remaining latency lever.

2 days ago

kilork pushed to main at sm121/recipes

fe606b76bb dsv41-flash-1017-4x-sglang: land the 2026-10-18 A/B reports and the 24/24 concurrency bump

2 days ago