Most local model benchmark posts jump straight to the scoreboard. Ours almost did too, and that would have been a mistake.

We were testing Qwen3.6 GGUF models on our DGX through LM Studio, with lm-evaluation-harness on top. At first the numbers looked ugly. One GSM8K run came back around 10 percent exact match for a 35B-class reasoning model. That kind of score is a red flag, but not always for the model. Sometimes it is the benchmark path telling you your setup is lying.

That turned out to be the real story.

After tracing the full pipeline, we found that the weak score was mostly caused by the evaluation path, not the model itself. LM Studio was returning reasoning separately from final answer content, and our token budget was too small for these reasoning-heavy Qwen3.6 variants. In other words, the model was spending its budget thinking, then getting cut off before it could finish the answer in the field that the scorer actually reads.

Once we fixed the adapter and gave GSM8K enough output budget, the same stack started producing sane results.

This article is the write-up of that process, what we fixed, what we trust now, and what we still do not trust yet.

We are benchmarking local Qwen3.6 models on DGX with this stack:

• Runtime: LM Studio

• Model format: GGUF

• API path: LM Studio OpenAI-compatible endpoint

• Evaluation harness: lm-evaluation-harness

• Normalization layer: a thin local proxy that strips reasoning_content from the scored output path and preserves only final assistant text for scoring

The main models in this round are:

• qwen3.6-27b-mtp-pi-reasoning

• qwopus3.6-27b-coder-mtp

• qwopus3.6-27b-coder-compat-mtp

• qwen3.6-35b-a3b-mtp as the faster MoE reference

Here is what the completed DGX run gave us on the tasks we trust in this stack right now.

That does not mean the reasoning variant is useless. It does mean that, in this benchmark path, it still struggled to emit stable scoreable final answers, while the two coder variants were much more benchmark-stable.

This is exactly why the benchmark-method story matters. Without fixing the stack first, it would be too easy to tell the wrong story about all three models.

Our first bad GSM8K result looked like a model failure. It was not.

Here is what was happening:

1. LM Studio returned reasoning separately from final answer content.

2. lm-eval only scores the final answer text it sees in the expected field.

3. With max_tokens=1024, some Qwen3.6 reasoning runs used the whole budget before final answer text was emitted.

4. The scorer then saw empty output, treated it as invalid, and the benchmark cratered.

That is not a fair test of the model. It is a test of whether your adapter and token budget are configured correctly.

Once we switched to a normalized proxy and increased GSM8K to max_tokens=4096, the benchmark stopped producing obviously broken results.

We now use a small LM Studio proxy in front of lm-eval.

Its job is simple:

• forward requests to LM Studio /v1/chat/completions

• preserve only final assistant content in the scored response path

• keep reasoning in a side field for debugging

• present a clean OpenAI-style response shape to the harness

That gave us a stable interface for chat-style evaluation without waiting on upstream support for these exact Qwen3.6 GGUF architectures in other runtimes.

The second fix was even more important than the first: we raised the output budget for GSM8K.

For these reasoning-heavy Qwen3.6 models, 1024 was not enough. 3072 was much better. 4096 was the first setting that looked consistently trustworthy in our smoke tests.

These are straightforward and already useful.

From our direct LM Studio runs, the broad picture was clear:

The surprising result is that the 35B A3B MoE reference was much faster than the 27B dense models, despite using far more VRAM. If you have the memory budget, that model is the clear speed winner in our early tests.

This now looks good enough to use.

The practical rule is simple:

• use the full chat endpoint URL

• normalize LM Studio responses through the proxy

• set max_tokens=4096 for GSM8K on these reasoning models

• log samples so bad outputs can be inspected later

That gets you from misleading numbers to something you can actually compare.

IFEval also works through the normalized chat-completions route.

We are keeping it in the stack because it gives us a useful second signal beyond math-word-problem accuracy. GSM8K tells us whether the model can get to the right answer. IFEval tells us whether it actually follows instructions cleanly.

That combination is already more useful for local model selection than raw speed charts alone.

This is the important part, because a lot of benchmark posts skip it.

We do not currently trust these tasks through the LM Studio backend:

• ARC Easy

• HellaSwag

Why not?

Because lm-eval scores these through loglikelihood, which means the backend needs to expose usable token logprobs. LM Studio's current OpenAI-compatible completions path does not give us reliable token logprobs for this workflow. In practice that means you can run the benchmark, but you should not trust the resulting score.

So should we improve the benchmark to include ARC Easy and HellaSwag?

Yes, but not by pretending the current path is good enough.

The right move is one of these:

1. wait until the runtime exposes usable logprobs for these tasks

2. switch those specific tasks to a backend that supports trustworthy loglikelihood scoring

3. keep GSM8K and IFEval as the production benchmark path for LM Studio GGUF runs today

Our recommendation is option 3 for now. It is the honest path.

This is bigger than one Qwen run.

A lot of teams are about to make content, buying, and deployment decisions off local benchmark stacks that are only half verified. When the setup is wrong, a strong model can look weak, a weak model can look usable, and a pretty chart can say less than a single well-inspected sample log.

That is why we now treat local benchmarking in three layers:

• Can the model load reliably?

• How much VRAM does it use?

• How long does it take to answer?

• Is the adapter preserving the final scored answer?

• Is the token budget high enough?

• Are failed samples inspectable?

• Does it solve the task?

• Does it follow instructions?

• Is it good enough for the real workflow we care about?

If you skip the first two layers, the third layer is mostly theater.

For Qwen3.6 GGUF models on DGX, our benchmark path now looks like this:

• LM Studio for inference

• local proxy for response normalization

• lm-evaluation-harness on top of the proxy

• GSM8K with max_tokens=4096

• IFEval with a smaller but still safe generation budget

• direct latency, load-time, and VRAM measurements alongside eval scores

• sample logging always on for debug runs

This gives us a benchmark system that is practical, repeatable, and honest about what it can and cannot measure today.

The first useful conclusion is not even about which 27B variant wins.

It is this: benchmark setup can dominate benchmark outcome.

If you run reasoning-heavy local models through a stack that drops the final answer, the benchmark is grading your plumbing, not your model.

Once we fixed the path, the numbers started making sense. That is the only point where model comparison becomes worth discussing.

We now have a real benchmark workflow for these local Qwen3.6 runs, and that alone is useful content because it answers a question a lot of people are quietly struggling with:

How do you benchmark local GGUF reasoning models without fooling yourself?

The next post should be the scoreboard post:

• all three Qwen3.6 27B variants

• GSM8K comparison

• IFEval comparison

• latency and VRAM side by side

• a practical recommendation for reasoning, coding, and speed-per-VRAM

That is the piece most people expect first. In our case, it should come second.

The first story was getting the benchmark to tell the truth.