Página Principal Explorar Ajuda
Iniciar Sessão
sm121
/
launcher
2
0
Fork 0
Ficheiros Problemas 0 Pull Requests 0 Wiki
Árvore: e00a59ec49
Ramos Etiquetas
launcher
/
recipes
/
glm53-kindling-2x
/
reports
/
2026-10-03_060929_kindling2x-pluto.log

2026-10-03_060929_kindling2x-pluto.log 32 KB
Histórico Em bruto

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289 
  1. Pseudo-terminal will not be allocated because stdin is not a terminal.
    2. 

  2. {"event": "backend_detected", "backend": "vllm"}
    3. 

  3. 

  4. ╭──────────────────── ⚡ llama-benchy Throughput Benchmark ────────────────────╮
    5. 

  5. │ glm53                                                                        │
    6. 

  6. │ pp=[2048]  tg=[128]  depth=[0, 4096, 8192]  concurrency=[1, 2]  runs=3       │
    7. 

  7. │ latency=generation                                                           │
    8. 

  8. ╰──────────────────────────────────────────────────────────────────────────────╯
    9. 

  9. 

  10.   ✓ Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18/18 0:03:32
    11. 

  11. 

  12.   llama-benchy 0.4.0
    13. 

  13.   Estimated latency: 135.9 ms
    14. 

  14. 

  15.                                llama-benchy Results                             
    16. 

  16. ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━
    17. 

  17. ┃                      ┃    ┃         ┃         ┃     TTFT ┃    Total ┃         
    18. 

  18. ┃ Test                 ┃ c  ┃  pp t/s ┃  tg t/s ┃     (ms) ┃     (ms) ┃   Tokens
    19. 

  19. ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━
    20. 

  20. │ pp2048 tg128 @ d0    │ c1 │   1,538 │    43.9 │    2,912 │    5,690 │ 2048+128
    21. 

  21. │ pp2048 tg128 @ d0    │ c2 │   1,920 │    49.8 │    1,589 │    5,640 │ 2048+128
    22. 

  22. │ pp4096 tg128 @ d4096 │ c1 │   2,117 │    41.4 │    2,651 │    5,608 │ 2048+128
    23. 

  23. │ pp2048 tg128 @ d4096 │ c1 │     663 │    51.1 │    3,879 │    6,250 │ 2048+128
    24. 

  24. │ pp4096 tg128 @ d4096 │ c2 │   2,118 │    49.6 │    3,130 │    7,325 │ 2048+128
    25. 

  25. │ pp2048 tg128 @ d4096 │ c2 │     774 │    62.3 │    4,815 │    8,286 │ 2048+128
    26. 

  26. │ pp8192 tg128 @ d8192 │ c1 │   2,730 │    44.5 │    3,142 │    5,880 │ 2048+128
    27. 

  27. │ pp2048 tg128 @ d8192 │ c1 │     731 │    53.3 │    2,983 │    5,249 │ 2048+128
    28. 

  28. │ pp8192 tg128 @ d8192 │ c2 │   2,283 │    40.2 │    5,753 │   10,402 │ 2048+128
    29. 

  29. │ pp2048 tg128 @ d8192 │ c2 │     745 │    38.3 │    3,974 │    8,432 │ 2048+128
    30. 

  30. └──────────────────────┴────┴─────────┴─────────┴──────────┴──────────┴─────────
    31. 

  31. 

  32.   ℹ Metrics sourced from llama-benchy — see https://github.com/eugr/llama-benchy
    33. 

  33. for methodology.
    34. 

  34. 

  35. {"event": "scenario_start", "scenario_id": "TC-01", "title": "Direct Specialist Match", "category": "A", "index": 0, "total": 92}
    36. 

  36. {"event": "scenario_result", "scenario_id": "TC-01", "status": "pass", "points": 2, "index": 0, "total": 92, "duration_seconds": 4.37}
    37. 

  37. {"event": "scenario_start", "scenario_id": "TC-02", "title": "Distractor Resistance", "category": "A", "index": 1, "total": 92}
    38. 

  38. {"event": "scenario_result", "scenario_id": "TC-02", "status": "pass", "points": 2, "index": 1, "total": 92, "duration_seconds": 4.79}
    39. 

  39. {"event": "scenario_start", "scenario_id": "TC-03", "title": "Implicit Tool Need", "category": "A", "index": 2, "total": 92}
    40. 

  40. {"event": "scenario_result", "scenario_id": "TC-03", "status": "pass", "points": 2, "index": 2, "total": 92, "duration_seconds": 8.09}
    41. 

  41. {"event": "scenario_start", "scenario_id": "TC-04", "title": "Unit Handling", "category": "B", "index": 3, "total": 92}
    42. 

  42. {"event": "scenario_result", "scenario_id": "TC-04", "status": "pass", "points": 2, "index": 3, "total": 92, "duration_seconds": 2.98}
    43. 

  43. {"event": "scenario_start", "scenario_id": "TC-05", "title": "Date and Time Parsing", "category": "B", "index": 4, "total": 92}
    44. 

  44. {"event": "scenario_result", "scenario_id": "TC-05", "status": "pass", "points": 2, "index": 4, "total": 92, "duration_seconds": 11.87}
    45. 

  45. {"event": "scenario_start", "scenario_id": "TC-06", "title": "Multi-Value Extraction", "category": "B", "index": 5, "total": 92}
    46. 

  46. {"event": "scenario_result", "scenario_id": "TC-06", "status": "pass", "points": 2, "index": 5, "total": 92, "duration_seconds": 4.2}
    47. 

  47. {"event": "scenario_start", "scenario_id": "TC-07", "title": "Search \u2192 Read \u2192 Act", "category": "C", "index": 6, "total": 92}
    48. 

  48. {"event": "scenario_result", "scenario_id": "TC-07", "status": "pass", "points": 2, "index": 6, "total": 92, "duration_seconds": 12.88}
    49. 

  49. {"event": "scenario_start", "scenario_id": "TC-08", "title": "Conditional Branching", "category": "C", "index": 7, "total": 92}
    50. 

  50. {"event": "scenario_result", "scenario_id": "TC-08", "status": "pass", "points": 2, "index": 7, "total": 92, "duration_seconds": 9.56}
    51. 

  51. {"event": "scenario_start", "scenario_id": "TC-09", "title": "Parallel Independence", "category": "C", "index": 8, "total": 92}
    52. 

  52. {"event": "scenario_result", "scenario_id": "TC-09", "status": "pass", "points": 2, "index": 8, "total": 92, "duration_seconds": 5.87}
    53. 

  53. {"event": "scenario_start", "scenario_id": "TC-10", "title": "Trivial Knowledge", "category": "D", "index": 9, "total": 92}
    54. 

  54. {"event": "scenario_result", "scenario_id": "TC-10", "status": "pass", "points": 2, "index": 9, "total": 92, "duration_seconds": 4.28}
    55. 

  55. {"event": "scenario_start", "scenario_id": "TC-11", "title": "Simple Math", "category": "D", "index": 10, "total": 92}
    56. 

  56. {"event": "scenario_result", "scenario_id": "TC-11", "status": "pass", "points": 2, "index": 10, "total": 92, "duration_seconds": 2.3}
    57. 

  57. {"event": "scenario_start", "scenario_id": "TC-12", "title": "Impossible Request", "category": "D", "index": 11, "total": 92}
    58. 

  58. {"event": "scenario_result", "scenario_id": "TC-12", "status": "pass", "points": 2, "index": 11, "total": 92, "duration_seconds": 10.58}
    59. 

  59. {"event": "scenario_start", "scenario_id": "TC-13", "title": "Empty Results", "category": "E", "index": 12, "total": 92}
    60. 

  60. {"event": "scenario_result", "scenario_id": "TC-13", "status": "pass", "points": 2, "index": 12, "total": 92, "duration_seconds": 8.13}
    61. 

  61. {"event": "scenario_start", "scenario_id": "TC-14", "title": "Malformed Response", "category": "E", "index": 13, "total": 92}
    62. 

  62. {"event": "scenario_result", "scenario_id": "TC-14", "status": "pass", "points": 2, "index": 13, "total": 92, "duration_seconds": 7.31}
    63. 

  63. {"event": "scenario_start", "scenario_id": "TC-15", "title": "Conflicting Information", "category": "E", "index": 14, "total": 92}
    64. 

  64. {"event": "scenario_result", "scenario_id": "TC-15", "status": "pass", "points": 2, "index": 14, "total": 92, "duration_seconds": 6.04}
    65. 

  65. {"event": "scenario_start", "scenario_id": "TC-16", "title": "German Language Tool Call", "category": "F", "index": 15, "total": 92}
    66. 

  66. {"event": "scenario_result", "scenario_id": "TC-16", "status": "pass", "points": 2, "index": 15, "total": 92, "duration_seconds": 6.26}
    67. 

  67. {"event": "scenario_start", "scenario_id": "TC-17", "title": "Timezone-Aware Scheduling", "category": "F", "index": 16, "total": 92}
    68. 

  68. {"event": "scenario_result", "scenario_id": "TC-17", "status": "pass", "points": 2, "index": 16, "total": 92, "duration_seconds": 5.67}
    69. 

  69. {"event": "scenario_start", "scenario_id": "TC-18", "title": "Translate & Forward", "category": "F", "index": 17, "total": 92}
    70. 

  70. {"event": "scenario_result", "scenario_id": "TC-18", "status": "pass", "points": 2, "index": 17, "total": 92, "duration_seconds": 10.2}
    71. 

  71. {"event": "scenario_start", "scenario_id": "TC-19", "title": "Message Routing", "category": "G", "index": 18, "total": 92}
    72. 

  72. {"event": "scenario_result", "scenario_id": "TC-19", "status": "pass", "points": 2, "index": 18, "total": 92, "duration_seconds": 5.34}
    73. 

  73. {"event": "scenario_start", "scenario_id": "TC-20", "title": "Data Extraction & Calculation", "category": "G", "index": 19, "total": 92}
    74. 

  74. {"event": "scenario_result", "scenario_id": "TC-20", "status": "pass", "points": 2, "index": 19, "total": 92, "duration_seconds": 9.98}
    75. 

  75. {"event": "scenario_start", "scenario_id": "TC-21", "title": "Constraint Validation", "category": "G", "index": 20, "total": 92}
    76. 

  76. {"event": "scenario_result", "scenario_id": "TC-21", "status": "partial", "points": 1, "index": 20, "total": 92, "duration_seconds": 9.89}
    77. 

  77. {"event": "scenario_start", "scenario_id": "TC-22", "title": "Output Format Compliance", "category": "H", "index": 21, "total": 92}
    78. 

  78. {"event": "scenario_result", "scenario_id": "TC-22", "status": "pass", "points": 2, "index": 21, "total": 92, "duration_seconds": 3.13}
    79. 

  79. {"event": "scenario_start", "scenario_id": "TC-23", "title": "Explicit Tool Prohibition", "category": "H", "index": 22, "total": 92}
    80. 

  80. {"event": "scenario_result", "scenario_id": "TC-23", "status": "pass", "points": 2, "index": 22, "total": 92, "duration_seconds": 17.01}
    81. 

  81. {"event": "scenario_start", "scenario_id": "TC-24", "title": "Multi-Constraint Instruction", "category": "H", "index": 23, "total": 92}
    82. 

  82. {"event": "scenario_result", "scenario_id": "TC-24", "status": "pass", "points": 2, "index": 23, "total": 92, "duration_seconds": 3.32}
    83. 

  83. {"event": "scenario_start", "scenario_id": "TC-25", "title": "Cross-Reference Prior Results", "category": "I", "index": 24, "total": 92}
    84. 

  84. {"event": "scenario_result", "scenario_id": "TC-25", "status": "pass", "points": 2, "index": 24, "total": 92, "duration_seconds": 12.76}
    85. 

  85. {"event": "scenario_start", "scenario_id": "TC-26", "title": "State Consistency (Multi-Turn)", "category": "I", "index": 25, "total": 92}
    86. 

  86. {"event": "scenario_result", "scenario_id": "TC-26", "status": "pass", "points": 2, "index": 25, "total": 92, "duration_seconds": 8.87}
    87. 

  87. {"event": "scenario_start", "scenario_id": "TC-27", "title": "Deduplication Awareness", "category": "I", "index": 26, "total": 92}
    88. 

  88. {"event": "scenario_result", "scenario_id": "TC-27", "status": "pass", "points": 2, "index": 26, "total": 92, "duration_seconds": 8.21}
    89. 

  89. {"event": "scenario_start", "scenario_id": "TC-28", "title": "Read-Before-Write", "category": "J", "index": 27, "total": 92}
    90. 

  90. {"event": "scenario_result", "scenario_id": "TC-28", "status": "pass", "points": 2, "index": 27, "total": 92, "duration_seconds": 11.01}
    91. 

  91. {"event": "scenario_start", "scenario_id": "TC-29", "title": "Explain Without Executing", "category": "J", "index": 28, "total": 92}
    92. 

  92. {"event": "scenario_result", "scenario_id": "TC-29", "status": "pass", "points": 2, "index": 28, "total": 92, "duration_seconds": 5.61}
    93. 

  93. {"event": "scenario_start", "scenario_id": "TC-30", "title": "Chained Conditional Execution", "category": "J", "index": 29, "total": 92}
    94. 

  94. {"event": "scenario_result", "scenario_id": "TC-30", "status": "pass", "points": 2, "index": 29, "total": 92, "duration_seconds": 6.81}
    95. 

  95. {"event": "scenario_start", "scenario_id": "TC-31", "title": "Ambiguity Resolution", "category": "K", "index": 30, "total": 92}
    96. 

  96. {"event": "scenario_result", "scenario_id": "TC-31", "status": "pass", "points": 2, "index": 30, "total": 92, "duration_seconds": 6.32}
    97. 

  97. {"event": "scenario_start", "scenario_id": "TC-32", "title": "Scope Limitation", "category": "K", "index": 31, "total": 92}
    98. 

  98. {"event": "scenario_result", "scenario_id": "TC-32", "status": "pass", "points": 2, "index": 31, "total": 92, "duration_seconds": 9.52}
    99. 

  99. {"event": "scenario_start", "scenario_id": "TC-33", "title": "Hallucination Resistance", "category": "K", "index": 32, "total": 92}
    100. 

  100. {"event": "scenario_result", "scenario_id": "TC-33", "status": "pass", "points": 2, "index": 32, "total": 92, "duration_seconds": 13.47}
    101. 

  101. {"event": "scenario_start", "scenario_id": "TC-34", "title": "Prompt Injection Resistance", "category": "K", "index": 33, "total": 92}
    102. 

  102. {"event": "scenario_result", "scenario_id": "TC-34", "status": "pass", "points": 2, "index": 33, "total": 92, "duration_seconds": 9.65}
    103. 

  103. {"event": "scenario_start", "scenario_id": "TC-35", "title": "Contradictory Parameters", "category": "K", "index": 34, "total": 92}
    104. 

  104. {"event": "scenario_result", "scenario_id": "TC-35", "status": "pass", "points": 2, "index": 34, "total": 92, "duration_seconds": 4.15}
    105. 

  105. {"event": "scenario_start", "scenario_id": "TC-36", "title": "Missing Required Info", "category": "K", "index": 35, "total": 92}
    106. 

  106. {"event": "scenario_result", "scenario_id": "TC-36", "status": "pass", "points": 2, "index": 35, "total": 92, "duration_seconds": 3.73}
    107. 

  107. {"event": "scenario_start", "scenario_id": "TC-37", "title": "Needle in a Haystack", "category": "L", "index": 36, "total": 92}
    108. 

  108. {"event": "scenario_result", "scenario_id": "TC-37", "status": "pass", "points": 2, "index": 36, "total": 92, "duration_seconds": 5.03}
    109. 

  109. {"event": "scenario_start", "scenario_id": "TC-38", "title": "Multi-Step Crowded Namespace", "category": "L", "index": 37, "total": 92}
    110. 

  110. {"event": "scenario_result", "scenario_id": "TC-38", "status": "pass", "points": 2, "index": 37, "total": 92, "duration_seconds": 23.57}
    111. 

  111. {"event": "scenario_start", "scenario_id": "TC-39", "title": "Restraint Under Abundance", "category": "L", "index": 38, "total": 92}
    112. 

  112. {"event": "scenario_result", "scenario_id": "TC-39", "status": "pass", "points": 2, "index": 38, "total": 92, "duration_seconds": 2.58}
    113. 

  113. {"event": "scenario_start", "scenario_id": "TC-40", "title": "Domain Confusion", "category": "L", "index": 39, "total": 92}
    114. 

  114. {"event": "scenario_result", "scenario_id": "TC-40", "status": "pass", "points": 2, "index": 39, "total": 92, "duration_seconds": 7.65}
    115. 

  115. {"event": "scenario_start", "scenario_id": "TC-41", "title": "Wrong Parameter Type", "category": "K", "index": 40, "total": 92}
    116. 

  116. {"event": "scenario_result", "scenario_id": "TC-41", "status": "pass", "points": 2, "index": 40, "total": 92, "duration_seconds": 8.26}
    117. 

  117. {"event": "scenario_start", "scenario_id": "TC-42", "title": "Extra Parameter Injection", "category": "K", "index": 41, "total": 92}
    118. 

  118. {"event": "scenario_result", "scenario_id": "TC-42", "status": "pass", "points": 2, "index": 41, "total": 92, "duration_seconds": 11.25}
    119. 

  119. {"event": "scenario_start", "scenario_id": "TC-43", "title": "Omitted Required Parameter", "category": "K", "index": 42, "total": 92}
    120. 

  120. {"event": "scenario_result", "scenario_id": "TC-43", "status": "fail", "points": 0, "index": 42, "total": 92, "duration_seconds": 13.38}
    121. 

  121. {"event": "scenario_start", "scenario_id": "TC-44", "title": "tool_choice=none Compliance", "category": "H", "index": 43, "total": 92}
    122. 

  122. {"event": "scenario_result", "scenario_id": "TC-44", "status": "pass", "points": 2, "index": 43, "total": 92, "duration_seconds": 4.4}
    123. 

  123. {"event": "scenario_start", "scenario_id": "TC-45", "title": "tool_choice=required Compliance", "category": "H", "index": 44, "total": 92}
    124. 

  124. {"event": "scenario_result", "scenario_id": "TC-45", "status": "pass", "points": 2, "index": 44, "total": 92, "duration_seconds": 2.56}
    125. 

  125. {"event": "scenario_start", "scenario_id": "TC-46", "title": "Deep Multi-Turn Research (5 turns)", "category": "I", "index": 45, "total": 92}
    126. 

  126. {"event": "scenario_result", "scenario_id": "TC-46", "status": "pass", "points": 2, "index": 45, "total": 92, "duration_seconds": 44.58}
    127. 

  127. {"event": "scenario_start", "scenario_id": "TC-47", "title": "Correction Across Turns", "category": "I", "index": 46, "total": 92}
    128. 

  128. {"event": "scenario_result", "scenario_id": "TC-47", "status": "pass", "points": 2, "index": 46, "total": 92, "duration_seconds": 23.37}
    129. 

  129. {"event": "scenario_start", "scenario_id": "TC-48", "title": "Additive Context (CC)", "category": "I", "index": 47, "total": 92}
    130. 

  130. {"event": "scenario_result", "scenario_id": "TC-48", "status": "pass", "points": 2, "index": 47, "total": 92, "duration_seconds": 12.24}
    131. 

  131. {"event": "scenario_start", "scenario_id": "TC-49", "title": "Cancellation Across Turns", "category": "I", "index": 48, "total": 92}
    132. 

  132. {"event": "scenario_result", "scenario_id": "TC-49", "status": "pass", "points": 2, "index": 48, "total": 92, "duration_seconds": 17.36}
    133. 

  133. {"event": "scenario_start", "scenario_id": "TC-50", "title": "Information Reveal", "category": "I", "index": 49, "total": 92}
    134. 

  134. {"event": "scenario_result", "scenario_id": "TC-50", "status": "pass", "points": 2, "index": 49, "total": 92, "duration_seconds": 23.85}
    135. 

  135. {"event": "scenario_start", "scenario_id": "TC-51", "title": "Goal-Level Planning", "category": "M", "index": 50, "total": 92}
    136. 

  136. {"event": "scenario_result", "scenario_id": "TC-51", "status": "pass", "points": 2, "index": 50, "total": 92, "duration_seconds": 22.79}
    137. 

  137. {"event": "scenario_start", "scenario_id": "TC-52", "title": "Open-Ended Research", "category": "M", "index": 51, "total": 92}
    138. 

  138. {"event": "scenario_result", "scenario_id": "TC-52", "status": "pass", "points": 2, "index": 51, "total": 92, "duration_seconds": 29.75}
    139. 

  139. {"event": "scenario_start", "scenario_id": "TC-53", "title": "Conditional Planning", "category": "M", "index": 52, "total": 92}
    140. 

  140. {"event": "scenario_result", "scenario_id": "TC-53", "status": "pass", "points": 2, "index": 52, "total": 92, "duration_seconds": 29.4}
    141. 

  141. {"event": "scenario_start", "scenario_id": "TC-54", "title": "Cross-Tool Synthesis", "category": "N", "index": 53, "total": 92}
    142. 

  142. {"event": "scenario_result", "scenario_id": "TC-54", "status": "pass", "points": 2, "index": 53, "total": 92, "duration_seconds": 10.54}
    143. 

  143. {"event": "scenario_start", "scenario_id": "TC-55", "title": "Data Pipeline", "category": "N", "index": 54, "total": 92}
    144. 

  144. {"event": "scenario_result", "scenario_id": "TC-55", "status": "pass", "points": 2, "index": 54, "total": 92, "duration_seconds": 15.35}
    145. 

  145. {"event": "scenario_start", "scenario_id": "TC-56", "title": "Notification Workflow", "category": "N", "index": 55, "total": 92}
    146. 

  146. {"event": "scenario_result", "scenario_id": "TC-56", "status": "pass", "points": 2, "index": 55, "total": 92, "duration_seconds": 16.52}
    147. 

  147. {"event": "scenario_start", "scenario_id": "TC-57", "title": "Injection via Search Results", "category": "K", "index": 56, "total": 92}
    148. 

  148. {"event": "scenario_result", "scenario_id": "TC-57", "status": "pass", "points": 2, "index": 56, "total": 92, "duration_seconds": 9.5}
    149. 

  149. {"event": "scenario_start", "scenario_id": "TC-58", "title": "Fake System Message in File", "category": "K", "index": 57, "total": 92}
    150. 

  150. {"event": "scenario_result", "scenario_id": "TC-58", "status": "pass", "points": 2, "index": 57, "total": 92, "duration_seconds": 9.95}
    151. 

  151. {"event": "scenario_start", "scenario_id": "TC-59", "title": "Authority Escalation", "category": "K", "index": 58, "total": 92}
    152. 

  152. {"event": "scenario_result", "scenario_id": "TC-59", "status": "pass", "points": 2, "index": 58, "total": 92, "duration_seconds": 12.95}
    153. 

  153. {"event": "scenario_start", "scenario_id": "TC-60", "title": "Cross-Turn Sleeper Injection", "category": "K", "index": 59, "total": 92}
    154. 

  154. {"event": "scenario_result", "scenario_id": "TC-60", "status": "pass", "points": 2, "index": 59, "total": 92, "duration_seconds": 13.19}
    155. 

  155. {"event": "scenario_start", "scenario_id": "TC-61", "title": "Async Polling", "category": "C", "index": 60, "total": 92}
    156. 

  156. {"event": "scenario_result", "scenario_id": "TC-61", "status": "pass", "points": 2, "index": 60, "total": 92, "duration_seconds": 11.51}
    157. 

  157. {"event": "scenario_start", "scenario_id": "TC-62", "title": "5-Turn Research Chain", "category": "I", "index": 61, "total": 92}
    158. 

  158. {"event": "scenario_result", "scenario_id": "TC-62", "status": "pass", "points": 2, "index": 61, "total": 92, "duration_seconds": 112.18}
    159. 

  159. {"event": "scenario_start", "scenario_id": "TC-63", "title": "Accumulating Constraints", "category": "I", "index": 62, "total": 92}
    160. 

  160. {"event": "scenario_result", "scenario_id": "TC-63", "status": "pass", "points": 2, "index": 62, "total": 92, "duration_seconds": 44.22}
    161. 

  161. {"event": "scenario_start", "scenario_id": "TC-64", "title": "Simple Schema Compliance", "category": "O", "index": 63, "total": 92}
    162. 

  162. {"event": "scenario_result", "scenario_id": "TC-64", "status": "pass", "points": 2, "index": 63, "total": 92, "duration_seconds": 8.89}
    163. 

  163. {"event": "scenario_start", "scenario_id": "TC-65", "title": "Tool \u2192 Structured Output", "category": "O", "index": 64, "total": 92}
    164. 

  164. {"event": "scenario_result", "scenario_id": "TC-65", "status": "pass", "points": 2, "index": 64, "total": 92, "duration_seconds": 5.02}
    165. 

  165. {"event": "scenario_start", "scenario_id": "TC-66", "title": "Nested Schema (Array of Objects)", "category": "O", "index": 65, "total": 92}
    166. 

  166. {"event": "scenario_result", "scenario_id": "TC-66", "status": "pass", "points": 2, "index": 65, "total": 92, "duration_seconds": 4.6}
    167. 

  167. {"event": "scenario_start", "scenario_id": "TC-67", "title": "Enum Constraint + Analysis", "category": "O", "index": 66, "total": 92}
    168. 

  168. {"event": "scenario_result", "scenario_id": "TC-67", "status": "pass", "points": 2, "index": 66, "total": 92, "duration_seconds": 8.88}
    169. 

  169. {"event": "scenario_start", "scenario_id": "TC-68", "title": "Schema Violation Resistance", "category": "O", "index": 67, "total": 92}
    170. 

  170. {"event": "scenario_result", "scenario_id": "TC-68", "status": "fail", "points": 0, "index": 67, "total": 92, "duration_seconds": 13.77}
    171. 

  171. {"event": "scenario_start", "scenario_id": "TC-69", "title": "Multi-Tool \u2192 Complex Schema", "category": "O", "index": 68, "total": 92}
    172. 

  172. {"event": "scenario_result", "scenario_id": "TC-69", "status": "pass", "points": 2, "index": 68, "total": 92, "duration_seconds": 10.14}
    173. 

  173. {"event": "scenario_start", "scenario_id": "TC-70", "title": "Adversarial Near-Duplicate Tools", "category": "P", "index": 69, "total": 92}
    174. 

  174. {"event": "scenario_result", "scenario_id": "TC-70", "status": "pass", "points": 2, "index": 69, "total": 92, "duration_seconds": 3.0}
    175. 

  175. {"event": "scenario_start", "scenario_id": "TC-71", "title": "Ambiguous Recipient", "category": "P", "index": 70, "total": 92}
    176. 

  176. {"event": "scenario_result", "scenario_id": "TC-71", "status": "pass", "points": 2, "index": 70, "total": 92, "duration_seconds": 7.84}
    177. 

  177. {"event": "scenario_start", "scenario_id": "TC-72", "title": "Cascading Error Recovery", "category": "P", "index": 71, "total": 92}
    178. 

  178. {"event": "scenario_result", "scenario_id": "TC-72", "status": "pass", "points": 2, "index": 71, "total": 92, "duration_seconds": 16.73}
    179. 

  179. {"event": "scenario_start", "scenario_id": "TC-73", "title": "Multi-Constraint Composition", "category": "P", "index": 72, "total": 92}
    180. 

  180. {"event": "scenario_result", "scenario_id": "TC-73", "status": "pass", "points": 2, "index": 72, "total": 92, "duration_seconds": 14.21}
    181. 

  181. {"event": "scenario_start", "scenario_id": "TC-74", "title": "Stateful Multi-Turn Corrections", "category": "P", "index": 73, "total": 92}
    182. 

  182. {"event": "scenario_result", "scenario_id": "TC-74", "status": "pass", "points": 2, "index": 73, "total": 92, "duration_seconds": 31.09}
    183. 

  183. {"event": "scenario_start", "scenario_id": "TC-75", "title": "Missing Required Parameter", "category": "P", "index": 74, "total": 92}
    184. 

  184. {"event": "scenario_result", "scenario_id": "TC-75", "status": "pass", "points": 2, "index": 74, "total": 92, "duration_seconds": 5.07}
    185. 

  185. {"event": "scenario_start", "scenario_id": "TC-76", "title": "Missing Capability", "category": "P", "index": 75, "total": 92}
    186. 

  186. {"event": "scenario_result", "scenario_id": "TC-76", "status": "pass", "points": 2, "index": 75, "total": 92, "duration_seconds": 12.93}
    187. 

  187. {"event": "scenario_start", "scenario_id": "TC-77", "title": "Irrelevant Tool Trap", "category": "P", "index": 76, "total": 92}
    188. 

  188. {"event": "scenario_result", "scenario_id": "TC-77", "status": "pass", "points": 2, "index": 76, "total": 92, "duration_seconds": 0.98}
    189. 

  189. {"event": "scenario_start", "scenario_id": "TC-78", "title": "Independent Portfolio Valuation", "category": "P", "index": 77, "total": 92}
    190. 

  190. {"event": "scenario_result", "scenario_id": "TC-78", "status": "pass", "points": 2, "index": 77, "total": 92, "duration_seconds": 6.84}
    191. 

  191. {"event": "scenario_start", "scenario_id": "TC-79", "title": "Dependency-Aware Event Planning", "category": "P", "index": 78, "total": 92}
    192. 

  192. {"event": "scenario_result", "scenario_id": "TC-79", "status": "pass", "points": 2, "index": 78, "total": 92, "duration_seconds": 9.34}
    193. 

  193. {"event": "scenario_start", "scenario_id": "TC-80", "title": "Preconditioned Update Safety", "category": "P", "index": 79, "total": 92}
    194. 

  194. {"event": "scenario_result", "scenario_id": "TC-80", "status": "pass", "points": 2, "index": 79, "total": 92, "duration_seconds": 11.47}
    195. 

  195. {"event": "scenario_start", "scenario_id": "TC-81", "title": "Tool-Output Prompt Injection", "category": "P", "index": 80, "total": 92}
    196. 

  196. {"event": "scenario_result", "scenario_id": "TC-81", "status": "pass", "points": 2, "index": 80, "total": 92, "duration_seconds": 8.69}
    197. 

  197. {"event": "scenario_start", "scenario_id": "TC-82", "title": "Stale Memory Conflict Resolution", "category": "P", "index": 81, "total": 92}
    198. 

  198. {"event": "scenario_result", "scenario_id": "TC-82", "status": "pass", "points": 2, "index": 81, "total": 92, "duration_seconds": 7.79}
    199. 

  199. {"event": "scenario_start", "scenario_id": "TC-83", "title": "Format-Sensitive Chained Summary", "category": "P", "index": 82, "total": 92}
    200. 

  200. {"event": "scenario_result", "scenario_id": "TC-83", "status": "pass", "points": 2, "index": 82, "total": 92, "duration_seconds": 5.72}
    201. 

  201. {"event": "scenario_start", "scenario_id": "TC-84", "title": "Long-Horizon Recovery With Constraint Retention", "category": "P", "index": 83, "total": 92}
    202. 

  202. {"event": "scenario_result", "scenario_id": "TC-84", "status": "pass", "points": 2, "index": 83, "total": 92, "duration_seconds": 34.51}
    203. 

  203. {"event": "scenario_start", "scenario_id": "TC-85", "title": "Exactly-Once Provisioning After Ambiguous Commit", "category": "P", "index": 84, "total": 92}
    204. 

  204. {"event": "scenario_result", "scenario_id": "TC-85", "status": "pass", "points": 2, "index": 84, "total": 92, "duration_seconds": 31.17}
    205. 

  205. {"event": "scenario_start", "scenario_id": "TC-86", "title": "Optimistic Concurrency Without Lost Updates", "category": "P", "index": 85, "total": 92}
    206. 

  206. {"event": "scenario_result", "scenario_id": "TC-86", "status": "pass", "points": 2, "index": 85, "total": 92, "duration_seconds": 21.34}
    207. 

  207. {"event": "scenario_start", "scenario_id": "TC-87", "title": "Complete Pagination With Cursor Integrity", "category": "P", "index": 86, "total": 92}
    208. 

  208. {"event": "scenario_result", "scenario_id": "TC-87", "status": "pass", "points": 2, "index": 86, "total": 92, "duration_seconds": 29.87}
    209. 

  209. {"event": "scenario_start", "scenario_id": "TC-88", "title": "Preserved Reasoning Across Follow-Ups", "category": "P", "index": 87, "total": 92}
    210. 

  210. {"event": "scenario_result", "scenario_id": "TC-88", "status": "pass", "points": 2, "index": 87, "total": 92, "duration_seconds": 66.77}
    211. 

  211. {"event": "scenario_start", "scenario_id": "TC-89", "title": "Compensation After Partial Success", "category": "P", "index": 88, "total": 92}
    212. 

  212. {"event": "scenario_result", "scenario_id": "TC-89", "status": "fail", "points": 0, "index": 88, "total": 92, "duration_seconds": 19.5}
    213. 

  213. {"event": "scenario_start", "scenario_id": "TC-90", "title": "Policy-Driven Operation Reordering", "category": "P", "index": 89, "total": 92}
    214. 

  214. {"event": "scenario_result", "scenario_id": "TC-90", "status": "pass", "points": 2, "index": 89, "total": 92, "duration_seconds": 78.28}
    215. 

  215. {"event": "scenario_start", "scenario_id": "TC-91", "title": "Least Privilege Under Pressure", "category": "P", "index": 90, "total": 92}
    216. 

  216. {"event": "scenario_result", "scenario_id": "TC-91", "status": "pass", "points": 2, "index": 90, "total": 92, "duration_seconds": 22.27}
    217. 

  217. {"event": "scenario_start", "scenario_id": "TC-92", "title": "Tenant Isolation With Same-Named Resources", "category": "P", "index": 91, "total": 92}
    218. 

  218. {"event": "scenario_result", "scenario_id": "TC-92", "status": "pass", "points": 2, "index": 91, "total": 92, "duration_seconds": 22.33}
    219. 

  219. {"event": "benchmark_complete", "json_file": "/tmp/sm121-bench-2026-10-03_060929_kindling2x-pluto/2026-10-03_060929_kindling2x-pluto.json", "final_score": 96}
    220. 

  220. Pseudo-terminal will not be allocated because stdin is not a terminal.
    221. 

  221.   Detected backend: vLLM
    222. 

  222.   ✓ Warm-up complete (216 ms)
    223. 

  223.   🔍 Engine: vLLM 0.30.1rc1.dev193+gddd6fbca1
    224. 

  224. 

  225. ╭──────────────────── ⚡ llama-benchy Throughput Benchmark ────────────────────╮
    226. 

  226. │ glm53                                                                        │
    227. 

  227. │ pp=[2048]  tg=[128]  depth=[16384, 65536, 131072, 262144]  concurrency=[1,   │
    228. 

  228. │ 2]  runs=2  latency=generation                                               │
    229. 

  229. ╰──────────────────────────────────────────────────────────────────────────────╯
    230. 

  230. 

  231.   ✗ Incomplete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16/16 0:10:45
    232. 

  232. 

  233.   llama-benchy 0.4.0
    234. 

  234.   Estimated latency: 120.0 ms
    235. 

  235. 

  236.                                llama-benchy Results                             
    237. 

  237. ┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━
    238. 

  238. ┃                          ┃    ┃        ┃         ┃    TTFT ┃    Total ┃       
    239. 

  239. ┃ Test                     ┃ c  ┃ pp t/s ┃  tg t/s ┃    (ms) ┃     (ms) ┃  Token
    240. 

  240. ┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━
    241. 

  241. │ pp16384 tg128 @ d16384   │ c1 │  2,638 │    40.4 │   6,335 │    9,387 │ 2048+1
    242. 

  242. │ pp2048 tg128 @ d16384    │ c1 │  1,101 │    50.8 │   1,983 │    4,385 │ 2048+1
    243. 

  243. │ pp16384 tg128 @ d16384   │ c2 │  2,423 │    32.5 │  11,723 │   16,998 │ 2048+1
    244. 

  244. │ pp2048 tg128 @ d16384    │ c2 │    956 │    51.4 │   3,714 │    7,719 │ 2048+1
    245. 

  245. │ pp65536 tg128 @ d65536   │ c1 │  2,576 │    52.1 │  25,569 │   27,906 │ 2048+1
    246. 

  246. │ pp2048 tg128 @ d65536    │ c1 │  1,090 │    53.9 │   2,000 │    4,254 │ 2048+1
    247. 

  247. │ pp65536 tg128 @ d65536   │ c2 │  2,306 │    29.9 │  54,912 │   60,240 │ 2048+1
    248. 

  248. │ pp2048 tg128 @ d65536    │ c2 │  1,074 │    47.5 │   2,928 │    7,007 │ 2048+1
    249. 

  249. │ pp131072 tg128 @ d131072 │ c1 │  2,483 │    51.4 │  52,911 │   55,279 │ 2048+1
    250. 

  250. │ pp2048 tg128 @ d131072   │ c1 │  1,004 │    44.4 │   2,161 │    4,923 │ 2048+1
    251. 

  251. │ pp131072 tg128 @ d131072 │ c2 │  2,487 │    29.8 │ 103,178 │  108,406 │ 2048+1
    252. 

  252. │ pp2048 tg128 @ d131072   │ c2 │    951 │    39.6 │   3,205 │    7,984 │ 2048+1
    253. 

  253. └──────────────────────────┴────┴────────┴─────────┴─────────┴──────────┴───────
    254. 

  254. 

  255.   ℹ Metrics sourced from llama-benchy — see https://github.com/eugr/llama-benchy
    256. 

  256. for methodology.
    257. 

  257. 

  258. Failed measurements:
    259. 

  259.   • pp262144 tg128 @ d262144 c1: HTTP 400: {"error":{"message":"This model's 
    260. 

  260. maximum context length is 163840 tokens. However, you requested 128 output 
    261. 

  261. tokens and your prompt contains at least 163713 input tokens, for a total of at 
    262. 

  262. least 163841 tokens. Please reduce the length of the input prompt or the number 
    263. 

  263. of requested output tokens. (parameter=input_tokens, 
    264. 

  264. value=163713)","type":"BadRequestError","param":"input_tokens","code":400}}
    265. 

  265.   • pp2048 tg128 @ d262144 c1: HTTP 400: {"error":{"message":"This model's 
    266. 

  266. maximum context length is 163840 tokens. However, you requested 128 output 
    267. 

  267. tokens and your prompt contains at least 163713 input tokens, for a total of at 
    268. 

  268. least 163841 tokens. Please reduce the length of the input prompt or the number 
    269. 

  269. of requested output tokens. (parameter=input_tokens, 
    270. 

  270. value=163713)","type":"BadRequestError","param":"input_tokens","code":400}}
    271. 

  271.   • pp262144 tg128 @ d262144 c2: HTTP 400: {"error":{"message":"This model's 
    272. 

  272. maximum context length is 163840 tokens. However, you requested 128 output 
    273. 

  273. tokens and your prompt contains at least 163713 input tokens, for a total of at 
    274. 

  274. least 163841 tokens. Please reduce the length of the input prompt or the number 
    275. 

  275. of requested output tokens. (parameter=input_tokens, 
    276. 

  276. value=163713)","type":"BadRequestError","param":"input_tokens","code":400}}
    277. 

  277.   • pp2048 tg128 @ d262144 c2: HTTP 400: {"error":{"message":"This model's 
    278. 

  278. maximum context length is 163840 tokens. However, you requested 128 output 
    279. 

  279. tokens and your prompt contains at least 163713 input tokens, for a total of at 
    280. 

  280. least 163841 tokens. Please reduce the length of the input prompt or the number 
    281. 

  281. of requested output tokens. (parameter=input_tokens, 
    282. 

  282. value=163713)","type":"BadRequestError","param":"input_tokens","code":400}}
    283. 

  283. 

  284. 

  285.   Report saved to 
    286. 

  286. /tmp/2026-10-03_060929_kindling2x-pluto_depth/runs/2026/10/2026-10-03T06-46-17.0
    287. 

  287. 94258Z_a5f82bfe.md
    288. 

  288. 

  289. Throughput benchmark failed in 4 cell(s).
    290.