Wrong Qwen3.6-35B-A3B Benchmark
I wanted to point out an issue with the benchmarks cited in your README file for comparing your model with others. I noticed that the benchmark scores for some competing models are incorrect. For example, the numbers for the Qwen3.6-35B-A3B model differ from what is shown in the official image
Thank you for raising the question and your careful observation.
Evaluation method. All metrics presented in the table are reproduced in-house: we download the public model checkpoints, deploy them via vLLM or SGLang, and evaluate under a unified standardized pipeline. No officially reported results of the respective models are directly adopted in this table. Each model is tested only once on each evaluation set; retests are conducted only if obvious errors are found.
Evaluation configuration.
SWE-bench Verified / Multilingual / Pro, KAT-Code-Bench: agent=claude_code@2.1.195, pass@k=1, temperature=1.0, top_p=0.95, 256k ctx.
Terminal-Bench 2.1: agent=terminus-2 / claude_code, pass@k=1, temperature=0.7, top_p=1.0, 256k ctx.
PinchBench: agent=openclaw@2026.3.13, pass@k=1, temperature=0.7, top_p=1.0, 256k ctx.
Scicode: pass@k=1, temperature=0.6, top_p=1.0, 256k ctx.
- Anomaly description.
Qwen3.6-35BA3B: We found that on the SWE-bench Verified, SWE-bench Multilingual, and SWE-bench Pro test sets, our test results this time have an approximate 10 pp gap compared with the official results. We believe this is mainly caused by the harness version and some optimizations made to the test sets by the Qwen team, and it should not be an issue with the model itself.
Qwen3.5-35BA3B: We observed frequent hallucinations during evaluation, including attempts to invoke the unavailable MultiEdit tool under the current agent environment, which negatively impacts the final metric.
Gemma4-26B-A4B-it: Two main factors degrade evaluation performance: context overflow (exceeding the 256k context limit) and hallucinated calls to the unsupported MultiEdit tool in this evaluation setup.
The above deviations arise from mismatches between model tool preference and the allowed toolset in the evaluation harness, rather than inherent capability limitations of the models.
My daily experience, aligns more with results in the model description than published by Qwen team, there is no way 35b Qwen (both 3.5 and 3.6) is better 31b Gemma, and 26b Gemma is very weak as agent.