Why my leaderboard has reshuffled: A field report on token hunger, silent responses, and the pitfalls of OpenRouter

Upgrading for a model that promises a lot

I recently upgraded my local stack, which is built around an Asus Ascent GX10. The reason was my quality standards. After switching a few months ago from my 24 GB Shared Memory MacBook M4 to the Asus GX10 with 128 GB Shared Memory, I could finally run larger models, from Qwen 3.6-27B to GPT-OSS-120B. The models ran faster, results improved, but the quality I expected was still missing.

Even the release of Qwen3.8-27B, a model that made a huge leap forward and excited the community, still left something to be desired. So I continued to depend on online models like GLM5.3 as a code reviewer. That, however, contradicted my goal of using LLMs in a privacy-compliant way on my own infrastructure.

In parallel, I was watching the results of the "large" local open weight model Qwen3.8-Flash-Next and the quantization efforts around my preferred model GLM5.3-Flash. This model, in its released size, was beyond the memory capacity of two clustered Nvidia Spark LLM servers. Resourceful quantizers ultimately managed to quantize the model down to 186 GB, however, allowing the community to run GLM5.3-Flash locally on a dual Spark setup.

Then August arrived, and hardware prices for local LLM setups exploded again. My Asus GX10, which had cost a steep 4,300 euros two months earlier, was now listed at 7,300 euros from the same vendor. That put my plans to expand my LLM hardware on hold. All the more welcome, then, was a clearance deal I found. It was still a good 1,000 euros more than a few days prior, but given the ongoing price chaos and the high prices expected to continue, I pulled the trigger.

The payoff of that investment was finally being able to run Qwen 3.8 Flash Next locally — the open weight preview model for Qwen 3.8 Flash sharing the same architecture as the upcoming Qwen 4 family — along with my particularly valued GLM 5.3 Flash. Both models represent, in my view, genuinely serious local intelligence. In the benchmark, Qwen3.8-Flash-Next scores notably ahead of GLM5.3-Flash. But comparing the technical dimensions of the two models quickly makes clear that GLM 5.3 Flash is the stronger model when it comes to reasoning depth and context window. Its larger parameter count also speaks for itself in terms of world knowledge. Ultimately, both models land near the top of the benchmark in quite good company.

The model that performed worse than it should have

After the setup, I wanted to know where things stood and ran both models through the benchmark. After the first test, GLM5.3-Flash scored significantly lower than Qwen3.8-Flash-Next. My benchmark is oriented toward everyday tasks — there are other benchmarks for intelligence and math tests — but the fact that GLM refused to break into the top 20 at all suggested a process error as well.

The answer lay in the model's token hunger. I first noticed this phenomenon about a year ago with the early GPT thinking models, and it hits a blind spot in CrucibleMark's original design. When I designed the benchmark, there were hardly any thinking models running locally, so I built in a capped token budget as a control mechanism.

My thinking was to make models comparable in terms of cost within a simulated production use case. Tokens have a price, and that price should be captured across my standardized question catalog.

Today the world looks different. Models with dense world knowledge can answer concisely and precisely, drawing on immense reasoning density while developing a very large appetite for memory. Running these models requires expensive data centers. Smaller, more frugal models that run on local hardware, however, often only reach peak performance through extensive reasoning. They discard, question, and restart before converging on a result. That produces enormous quantities of reasoning tokens, from the early GPT models to my GLM 5.3 Flash. And here was my design flaw: my benchmark was cutting off reasoning depth. That was a relic of the original design. What I find interesting is that even I prefer to take longer coffee breaks in exchange for a genuinely well-reasoned result, rather than settling for a half-baked quick answer that I'll have to pull apart again during refactoring.

My benchmark was therefore not measuring real model capability on my hardware — it was measuring the result within the constraints of the old token cap.

When 32,000 tokens vanish without a trace

Thinking models reason before producing the actual response. Normally the reasoning ends and the response text follows. Sometimes, though, the reasoning never stops: the model generates reasoning tokens endlessly until the budget is exhausted. The visible response text remains empty or consists of fragments — reasoning crumbs. The Judge then evaluates the emptiness, and the score collapses to one or two percent, even though the model is perfectly capable of the task.

The concrete trigger for the fix was the result of a specific investigation: GLM-5.3 burned through the entire budget of 32,000 tokens on a UX Writing task and achieved a score of 1.05 percent. The benchmark evaluated the result as technically a valid model response — the model simply had no chance to answer properly, since its output was cut off.

An outlier analysis across all versioned result files revealed just how large the problem actually was: 47 asset runs across 22 models suffered measurable score damage:

This was not an isolated incident but systematic behavior of an entire model class that my measurement framework had until then incorrectly interpreted as model failure. My earlier attempts at a fix had not addressed the root cause:

• A static override per model required knowledge that only emerges from measurements, and then acted as a special case alongside the model description.
• Preventive reasoning caps via OpenRouter also cap legitimate reasoning and throttle the models.
• A single-stage re-ask with an increased budget failed again at the cap in persistent cases.

None of the three approaches wrote the insight gained back into the model description, so every benchmark run repeated the same experience from scratch.

An update against forgetting

After this realization, I introduced an escalation ladder. Instead of a static cap, a retry after triggering climbs to the next token budget level — first 24,000, then 32,000 tokens. Absolute steps rather than multipliers, so that calibrated starting budgets do not explode. The escalation is triggered by four simultaneous conditions: truncation due to token limit, empty visible output, a detectable reasoning signal, and no probe context (the ladder only applies to real test tasks, not to a model's initial reasoning check).

Two properties turn this into measurement infrastructure rather than a patch:

• It learns. A successfully escalated level is written back into the Model Card as cot_budget_calibration. The next run starts there directly instead of climbing again. The calibration applies across modules, is visible in the run header, and lands as a CSV column in every report.

• It separates measurement limit from model failure. If the highest level is exhausted without visible output, the result is 0 percent, with full traceability in the audit log. The Judge remains strictly quality-focused and never evaluates the token load itself.

This is complemented by a last-resort mode: one final individual attempt with an opened budget of 48,000 tokens, deliberately without card calibration. It remains a documented exception with a warning block in the report and its own CSV columns — evaluable by the Judge, but produced under exceptional resource conditions.

One final gap was only closed in live operation: GLM-5.3, after burning through 32,000 tokens on a subsequent appearance, produced around 300 characters of response crumbs instead of nothing. This caused it to bypass both the ladder and the last-resort, because the trigger only reacted to empty content. The Judge evaluated the crumbs as a 1.05 percent response. Since then, the rule is: visible output under 500 characters on a token truncation plus reasoning signal counts as reasoning-only truncation. Substantive output above that threshold remains accepted. Comparability is preserved: no best-of-2, no retroactive grade improvement through mere repetition.

The result of the live verification on the historical trigger case (GLM-5.3, UX Writing, five tasks): all five tasks answered correctly, averaging 79.8 percent, total cost 33 cents. The former disaster task now sits at 78.75 percent instead of 1.05 percent, with zero false triggers in the remaining healthy runs. The entire problem class affected 22 of 152 models in the leaderboard — roughly 14 percent. For the large majority, nothing changes. For the affected minority, it is the difference between a random score catastrophe and a traceable, calibrated edge case.

Disillusionment when retesting the top 20

After the changeover, I systematically retested the entire top 20 and discovered behavior that had nothing to do with the token ladder. Models that had answered sensibly in earlier benchmarks reshuffled into different, lower positions when benchmarked again. That looked suspiciously like model degradation.

The decisive control point was my local comparison group: nine self-hosted models, fixed weights, identical vLLM config, identical Judge, identical assets. If the updated measurement pipeline runs stably, their positions must remain constant. And that is exactly what they did, apart from an absolute deviation of ±0.5 points, which I treat as pure noise:

The cloud models, by contrast, shifted by −3 to −4.6 points over the same period. The difference between the two groups was exactly the one variable that does not exist for my local models: the provider host.

The Judge itself, claude-haiku-4-5-20251001, remained identical across all runs, and around 50 assets across all models landed at exactly ±0.0 delta: identical responses produced identical scores. The higher budget itself was also not the cause. An A/B control test with three budget levels (20,000 / 25,000 / 32,000 tokens) on the same prompt reproduced the September phenomenon in none of the variants. The token ladder in fact rescued scores rather than lowering them — for example with mimo-v2.5 reasoning_5b: from 16.0 percent (August burn without ladder) to 82.8 percent (with escalation token ladder).

A provider I had never chosen

When I ran the original tests — usually shortly after a model's release — OpenRouter was still routing me directly to the model's own vendor. At that point, many open weight models had no alternative providers yet. Weeks later, additional providers had picked up the same models, and OpenRouter routes by default to the cheapest provider. In that competition, lower prices are often bought through adjusted serving settings. And that made a clearly visible difference in results.

As a fix, I repeated the tests with vendor routing pinned explicitly. Even then, I could observe notable variance with MiniMax M3. Whether server settings at MiniMax M3 had been changed was not something I could determine.

OpenRouter is a gateway: the same model is served by rotating upstream hosts. Xiaomi's Mimo-v2.5 ran across six hosts on the day of the re-benchmark (GMICloud, DeepInfra, Xiaomi, StreamLake, Novita, Venice); Minimax-M3 across as many as thirteen. A host-pinning test with an identical prompt showed just how differently these hosts actually respond:

On a German-language task, language fidelity ranged between 0.10 and 0.62, depending on which server happened to respond. MiniMax M3 showed the same effect in a concrete score drop: a CLI task (expecting concise Bash output) fell from 100 to 58 percent, because CoreWeave and GMICloud delivered the brief one-liner (115 and 106 characters respectively), while Venice, Together, and StreamLake produced a verbosely commented block (516 to 931 characters). Identical prompt, identical benchmark code. The score difference arose solely from the response style of whichever server happened to be serving the request.

Claude Opus 5 and the wall of refusals

An independent pattern affected Claude Opus 5, and here the cause was not OpenRouter but the provider Anthropic itself. In the Logical Reasoning module (11 assets from reasoning and Metacognition tasks), the Refusal rate on identical prompts rose from 2 to 8 refused responses. Newly refused were one reasoning task (previously 99.2 percent, now 0) and all five Metacognition tasks (previously 68 to 97 percent, now uniformly 0). This pushed the module score from 68.6 to 26.4 percent, while all other modules remained stable (Code 81.9, Documentation 86.0). This too is not a measurement or budget effect, but a server-side security tightening at Anthropic under an unchanged model name. It is the same type of cause as with OpenRouter, except that here Anthropic itself changed the behavior.

A deeper analysis of the Metacognition refusals afterward revealed the exact trigger. Not the task content, but the prescribed <thought> tag format instruction in the prompt triggered the refusal. This was confirmed by a further A/B test that applied across versions to both Opus 5 and 5.5.

What this means for the leaderboard

With these three independent but clearly distinguishable causes, the reshuffling of the scoreboard can be explained in a traceable way:

A uniform Judge shift would affect all modules equally. That is precisely what is not shown here: MiMo V2.5 loses on Documentation (−14.3) and Content (−11.3), while Logical Reasoning rises (+3.3). Minimax-M3 loses on CLI (−9.3), while Documentation rises (+3.3). The pattern shows changed response behavior, not changed evaluation.

I have therefore introduced vendor pinning in CrucibleMark as a rule, each time with allow_fallbacks: false, so that a test fails rather than silently running over a different host. In addition, a new CSV column upstream_provider now logs per request which host actually responded. Future routing events are thus traceable.

The pinned control re-run confirmed the diagnosis doubly: under the vendor host, burns, content-filter aborts, and language mixing disappeared entirely, and scores returned toward the August ratings. The remaining gap I attribute to normal variance between two benchmark runs drifting apart in time, plus the config delta between a calibrated 32k start and the old 20k cap.

Conclusion

My benchmark CrucibleMark has gone through several iterations over the past months. Where it originally set out to illuminate the black box of "LLM" and make its behavior traceable for the observer, the stated goal today is to determine performance as a function of measured behavior. The underlying question remains the same: which open weight model can serve as an alternative to expensive commercial offerings?

The leaderboard reshuffle has three faces: it shows where the token ladder finally measures token-hungry models fairly, where OpenRouter routing undermines comparability when the vendor host is not explicitly enforced, and where a provider itself — here Anthropic — changes model behavior without communicating this in the model name. For me, this means: a benchmark is never fully calibrated as long as the infrastructure beneath it continues to evolve. The token ladder with memory and vendor pinning are the current answer to that evolution — not as a one-time repair, but as part of the ongoing evolution of the measurement infrastructure.