# Qwen3.6 local benchmark on DGX By [nanobro](https://blog.nanobro.co) ยท 2026-06-23 benchmark, llm, local-ai, qwen, lmstudio --- We almost benchmarked Qwen3.6 wrong on DGX ========================================== Most local model benchmark posts jump straight to the scoreboard. Ours almost did too, and that would have been a mistake. We were testing Qwen3.6 GGUF models on our DGX through LM Studio, with `lm-evaluation-harness` on top. At first the numbers looked ugly. One GSM8K run came back around 10 percent exact match for a 35B-class reasoning model. That kind of score is a red flag, but not always for the model. Sometimes it is the benchmark path telling you your setup is lying. That turned out to be the real story. After tracing the full pipeline, we found that the weak score was mostly caused by the evaluation path, not the model itself. LM Studio was returning reasoning separately from final answer content, and our token budget was too small for these reasoning-heavy Qwen3.6 variants. In other words, the model was spending its budget thinking, then getting cut off before it could finish the answer in the field that the scorer actually reads. Once we fixed the adapter and gave GSM8K enough output budget, the same stack started producing sane results. This article is the write-up of that process, what we fixed, what we trust now, and what we still do not trust yet. The setup --------- We are benchmarking local Qwen3.6 models on DGX with this stack: * Runtime: LM Studio * Model format: GGUF * API path: LM Studio OpenAI-compatible endpoint * Evaluation harness: `lm-evaluation-harness` * Normalization layer: a thin local proxy that strips `reasoning_content` from the scored output path and preserves only final assistant text for scoring The main models in this round are: * `qwen3.6-27b-mtp-pi-reasoning` * `qwopus3.6-27b-coder-mtp` * `qwopus3.6-27b-coder-compat-mtp` * `qwen3.6-35b-a3b-mtp` as the faster MoE reference Actual results from the first full run -------------------------------------- Here is what the completed DGX run gave us on the tasks we trust in this stack right now. Model GSM8K exact match IFEval inst loose IFEval inst strict IFEval prompt loose IFEval prompt strict qwen3.6-27b-mtp-pi-reasoning 0.00 0.375 0.375 0.40 0.40 qwopus3.6-27b-coder-mtp 1.00 0.625 0.625 0.80 0.80 qwopus3.6-27b-coder-compat-mtp 1.00 0.625 0.625 0.80 0.80 That does not mean the reasoning variant is useless. It does mean that, in this benchmark path, it still struggled to emit stable scoreable final answers, while the two coder variants were much more benchmark-stable. This is exactly why the benchmark-method story matters. Without fixing the stack first, it would be too easy to tell the wrong story about all three models. The first lesson: benchmark plumbing matters more than people want to admit --------------------------------------------------------------------------- Our first bad GSM8K result looked like a model failure. It was not. Here is what was happening: 1. LM Studio returned reasoning separately from final answer content. 2. `lm-eval` only scores the final answer text it sees in the expected field. 3. With `max_tokens=1024`, some Qwen3.6 reasoning runs used the whole budget before final answer text was emitted. 4. The scorer then saw empty output, treated it as invalid, and the benchmark cratered. That is not a fair test of the model. It is a test of whether your adapter and token budget are configured correctly. Once we switched to a normalized proxy and increased GSM8K to `max_tokens=4096`, the benchmark stopped producing obviously broken results. What we fixed ------------- We now use a small LM Studio proxy in front of `lm-eval`. Its job is simple: * forward requests to LM Studio `/v1/chat/completions` * preserve only final assistant `content` in the scored response path * keep reasoning in a side field for debugging * present a clean OpenAI-style response shape to the harness That gave us a stable interface for chat-style evaluation without waiting on upstream support for these exact Qwen3.6 GGUF architectures in other runtimes. The second fix was even more important than the first: we raised the output budget for GSM8K. For these reasoning-heavy Qwen3.6 models, `1024` was not enough. `3072` was much better. `4096` was the first setting that looked consistently trustworthy in our smoke tests. What we trust now ----------------- ### 1\. Latency, load time, and VRAM comparisons These are straightforward and already useful. From our direct LM Studio runs, the broad picture was clear: Model Load time VRAM Quick reasoning/code latency qwen3.6-27b-mtp-pi-reasoning ~7.6s ~14.8 GiB ~9s qwopus3.6-27b-coder-mtp ~7.6s ~15.6 GiB ~9 to 10s qwopus3.6-27b-coder-compat-mtp ~8.4s ~15.6 GiB ~9 to 10s qwen3.6-35b-a3b-mtp ~9.4s ~27.0 GiB ~3s The surprising result is that the 35B A3B MoE reference was much faster than the 27B dense models, despite using far more VRAM. If you have the memory budget, that model is the clear speed winner in our early tests. ### 2\. GSM8K through the proxy, if configured correctly This now looks good enough to use. The practical rule is simple: * use the full chat endpoint URL * normalize LM Studio responses through the proxy * set `max_tokens=4096` for GSM8K on these reasoning models * log samples so bad outputs can be inspected later That gets you from misleading numbers to something you can actually compare. ### 3\. IFEval through the same proxy path IFEval also works through the normalized chat-completions route. We are keeping it in the stack because it gives us a useful second signal beyond math-word-problem accuracy. GSM8K tells us whether the model can get to the right answer. IFEval tells us whether it actually follows instructions cleanly. That combination is already more useful for local model selection than raw speed charts alone. What we do not trust yet ------------------------ This is the important part, because a lot of benchmark posts skip it. We do **not** currently trust these tasks through the LM Studio backend: * ARC Easy * HellaSwag Why not? Because `lm-eval` scores these through loglikelihood, which means the backend needs to expose usable token logprobs. LM Studio's current OpenAI-compatible completions path does not give us reliable token logprobs for this workflow. In practice that means you can run the benchmark, but you should not trust the resulting score. So should we improve the benchmark to include ARC Easy and HellaSwag? Yes, but not by pretending the current path is good enough. The right move is one of these: 1. wait until the runtime exposes usable logprobs for these tasks 2. switch those specific tasks to a backend that supports trustworthy loglikelihood scoring 3. keep GSM8K and IFEval as the production benchmark path for LM Studio GGUF runs today Our recommendation is option 3 for now. It is the honest path. Why this matters beyond benchmarking ------------------------------------ This is bigger than one Qwen run. A lot of teams are about to make content, buying, and deployment decisions off local benchmark stacks that are only half verified. When the setup is wrong, a strong model can look weak, a weak model can look usable, and a pretty chart can say less than a single well-inspected sample log. That is why we now treat local benchmarking in three layers: ### Layer 1: infrastructure truth * Can the model load reliably? * How much VRAM does it use? * How long does it take to answer? ### Layer 2: benchmark path truth * Is the adapter preserving the final scored answer? * Is the token budget high enough? * Are failed samples inspectable? ### Layer 3: model truth * Does it solve the task? * Does it follow instructions? * Is it good enough for the real workflow we care about? If you skip the first two layers, the third layer is mostly theater. The benchmark stack we will use from now on ------------------------------------------- For Qwen3.6 GGUF models on DGX, our benchmark path now looks like this: * LM Studio for inference * local proxy for response normalization * `lm-evaluation-harness` on top of the proxy * GSM8K with `max_tokens=4096` * IFEval with a smaller but still safe generation budget * direct latency, load-time, and VRAM measurements alongside eval scores * sample logging always on for debug runs This gives us a benchmark system that is practical, repeatable, and honest about what it can and cannot measure today. Early takeaway -------------- The first useful conclusion is not even about which 27B variant wins. It is this: benchmark setup can dominate benchmark outcome. If you run reasoning-heavy local models through a stack that drops the final answer, the benchmark is grading your plumbing, not your model. Once we fixed the path, the numbers started making sense. That is the only point where model comparison becomes worth discussing. What comes next --------------- We now have a real benchmark workflow for these local Qwen3.6 runs, and that alone is useful content because it answers a question a lot of people are quietly struggling with: How do you benchmark local GGUF reasoning models without fooling yourself? The next post should be the scoreboard post: * all three Qwen3.6 27B variants * GSM8K comparison * IFEval comparison * latency and VRAM side by side * a practical recommendation for reasoning, coding, and speed-per-VRAM That is the piece most people expect first. In our case, it should come second. The first story was getting the benchmark to tell the truth. --- *Originally published on [nanobro](https://blog.nanobro.co/qwen36-local-benchmark-on-dgx)*