 Alexander Korolev
|
bc5aa791bf
8nodes-switch cluster in config (dual-rail 200G, ranks 0-7), dgx7/8 nas mount fixed, night plan 09-18: 8x ready, DS4.1-8x in-memory engram
|
1 mês atrás |
|
Alexander Korolev
|
73dd1db126
night plan 2026-09-17: full GLM-5.3 + DSpark K8 (speculator.dspark, downloaded), r37 typo verdict (boot D + eugr 0913), 8x trials on the new cables, lab lil-catalog policies
|
1 mês atrás |
|
Alexander Korolev
|
523924b032
remove stray file 1 (shell redirect accident)
|
1 mês atrás |
|
Alexander Korolev
|
f011388d0a
web: SSE streams can no longer die silently or stay dead — 15s keepalive comments on both endpoints, live-chip watchdog rebuilds a stale stream / a gave-up EventSource, bounce on visibility/online; supervisor isolates ticks and survives poisoned locks
|
1 mês atrás |
|
Alexander Korolev
|
706b7a5e03
topo: an HA-OFF switch can never have green ports - fabric port dots and 200G cable flow now require the switch to report on (fabPortOn policy + 6 playwright probes)
|
1 mês atrás |
|
Alexander Korolev
|
2f717631ff
report: stale cross-check of run ledger vs live probe (orange + runlock hint); new 'sm121 runlock list|close' offline ledger command
|
1 mês atrás |
|
Alexander Korolev
|
d5bbe7e54e
bench: dev66 glm53 eugr 2x re-run 161/176 1326pp/28.1tg/1458ms (same image as 09-11, quality lane verdict unchanged) + history (stale ledger fix)
|
1 mês atrás |
|
Alexander Korolev
|
c7d5cae0ee
bug report draft: r37 tree DS4.1 text garbling (bisect A-C, D pending)
|
1 mês atrás |
|
Alexander Korolev
|
65529d1c6c
history: HA shows dgx1-4 powered ON - nodes either hung or mgmt segment down; physical check needed
|
1 mês atrás |
|
Alexander Korolev
|
e0e73cf562
history: cluster-wide outage during Boot D - nodes dark mid-load, investigating recovery
|
1 mês atrás |
|
Alexander Korolev
|
9bbfae13d1
r37 typo bisect: A (no linear-b12x), B (prod spec K5), C (full prod contract) all still garbled; D (engram off) in progress - cause isolated to the r37 tree, not the flags
|
1 mês atrás |
|
Alexander Korolev
|
208a173b0c
history: morning R37 typo investigation (LM-head quant found + disproven as sole cause; linear-backend b12x bisect started)
|
1 mês atrás |
|
Alexander Korolev
|
902e989c68
r37 recipe: VLLM_MXFP8_LM_HEAD=0 - the lab tree's runtime LM-head MXFP8 quantization (default ON, absent from the eugr prod image) is the suspected cause of the typo regression
|
1 mês atrás |
|
Alexander Korolev
|
4fdabdb105
night shift final: full MXFP4 works on 4x at 131K (158/176, DCP4 blocked by engine bug), R37 = 2.3x prefill champion (3,830 pp) but 150/176 quality; teardown notes
|
1 mês atrás |
|
Alexander Korolev
|
5adc5325ff
P2 result: R37 DS4.1 4x - 150/176 (85, safety-capped) but 3,830 pp = 2.3x prod prefill, TTFT 832 ms (2.1x better); reports landed (dev65-r37-ds41-4x)
|
1 mês atrás |
|
Alexander Korolev
|
4cb61b3aef
P0 result: full GLM-5.3 MXFP4 on 4x GB10 - 158/176 (90), 934 pp d0; reports landed (dev65-full-mxfp4-131k)
|
1 mês atrás |
|
Alexander Korolev
|
bbeb6bd553
mxfp4 full recipe: switch to the r37 image - r35's DCP4 sparse decode path crashes on this model (reshape x4 DCP factor in attention.py during graph capture)
|
1 mês atrás |
|
Alexander Korolev
|
e2af78e409
night plan interim: full MXFP4 probe booted and passed (51.6 KB/token/rank KV, smoke green); DCP4 interleave fixed to 1
|
1 mês atrás |
|
Alexander Korolev
|
36bb9825d6
mxfp4 full recipe: cp-kv-cache-interleave-size 1 - the DCP sparse indexer (DSA top-k path) only supports interleave 1 on r35 (4 is the Flash dense-MLA rule, not this path)
|
1 mês atrás |
|
Alexander Korolev
|
675ccdb307
mxfp4 recipes: --linear-backend marlin - the b12x mxfp4 warmup (ungated) crashes on this checkpoint's shapes (rank 3 vs 4 in b12x dense_gemm); marlin A16 is the guaranteed path
|
1 mês atrás |
|
Alexander Korolev
|
9cf76da709
mxfp4 probe: KV pin 4->8 GiB - engine needs 6.58 GiB/rank for one 131072-token request (51.6 KB/token/rank measured)
|
1 mês atrás |
|
Alexander Korolev
|
57e6336047
mxfp4 recipes: --linear-backend b12x (native SM120 GEMM) - the flashinfer mxfp4 linear kernel wrongly claims sm_12x support and crashes in cute-dsl mm_fp4; --moe-backend is inert for the compressed-tensors MoE scheme (it picks MarlinExperts itself)
|
1 mês atrás |
|
Alexander Korolev
|
17672b1322
mxfp4 recipes: --moe-backend flashinfer_cutlass (the Sept-tree CLI family; _afp8 was not a valid choice on r35 - first boot argparse)
|
1 mês atrás |
|
Alexander Korolev
|
87e866512c
night shift: P0 probe recipe (no DCP, 131K, 4 GiB KV) - step 1 of the full-model boot ladder
|
1 mês atrás |
|
Alexander Korolev
|
501e627b2b
night plan: R37 image built+distributed (label 9350429b) and the MXFP4 checkpoint complete on the NAS - both lanes ready, switch needs the go-ahead
|
1 mês atrás |
|
Alexander Korolev
|
beda9d96ae
web: persist window layout (geometry/minimized/closed/maximized + open optional windows) in localStorage across reloads; recipes panel drops its own title and live chip when hosted (body.embed) and reopens from the menu when minimized; closed named windows keep a dock chip
|
1 mês atrás |
|
Alexander Korolev
|
b5e08df460
glm53 full MXFP4: explicit --moe-backend flashinfer_cutlass_afp8 + document the MXFP4 ingestion path
|
1 mês atrás |
|
Alexander Korolev
|
e3e9cbd3b1
web header: guard #maxinfo[hidden] (display:flex defeated the hidden attribute, so the maximize chrome sat in the toolbar permanently); assert header visibility, not the .hidden property, in playwright
|
1 mês atrás |
|
Alexander Korolev
|
8433106ff4
night-shift 2026-09-13 plan + R37 (jovian) build profile
|
1 mês atrás |
|
Alexander Korolev
|
5c54bf0af5
web windows: maximized window drops its titlebar too — header carries the title plus restore/minimize/close for that window
|
1 mês atrás |
