소스 검색

dflash A/B: attention_backend TRITON_ATTN→B12X — on the eugr build only B12X accepts fp8 kv-cache (the main engine's backend); kindling's TRITON_ATTN works on our overlay image, not here

Alexander Korolev 5 일 전
부모
커밋
5939b55b07
1개의 변경된 파일과 1개의 추가작업 그리고 1개의 파일을 삭제
  1. 1 1
      recipes/glm53-flash-fp4local-2x-dflash/recipe.toml

+ 1 - 1
recipes/glm53-flash-fp4local-2x-dflash/recipe.toml

@@ -63,7 +63,7 @@ args = [
     "--enable-auto-tool-choice",
     "--gpu-memory-utilization", "0.87",
     "--speculative-config", """
-'{"method":"dflash","model":"{{dflash2}}","num_speculative_tokens":7,"rejection_sample_method":"standard","attention_backend":"TRITON_ATTN","draft_load_config":{"load_format":"auto"}}'""",
+'{"method":"dflash","model":"{{dflash2}}","num_speculative_tokens":7,"rejection_sample_method":"standard","attention_backend":"B12X","draft_load_config":{"load_format":"auto"}}'""",
 ]
 
 # DFlash2 drafter (A/B vs the card's embedded MTP head): distilled against the