1. Small models, long think times
With CrucibleMark I have been testing LLMs for a year across various disciplines such as Code Quality, Documentation, UX Writing, Logical Reasoning, and Cultural Intelligence. The goal is not to find out which model is the most powerful — established benchmarks do that far better anyway. What I want to find out is which local open-weight models I can use to handle my everyday tasks without a significant loss in quality.
A further discipline is determining model bias when answering questions from the domains of economics, culture, and society. For this I created a standalone diagnostic module that does not affect the overall score but attempts to surface the bias in training data based on model responses. This module is my Political Compass (Political Compass methodology).
The Political Compass poses 79 calibrated questions across nine thematic blocks and measures two coordinates: an economic axis (left–right) and a social axis (libertarian–authoritarian). The twist: every model runs through the module twice — once in standard mode, once in anti-diplomat mode, which suppresses evasive rhetoric and forces a clear position. The expected answer is simple in both cases. The model must choose one of four positions in the form of a single letter.
This methodology was stable for two years because it ran primarily against large cloud models. These respond quickly enough even with reasoning enabled and don't stand out.
For my new test series to benchmark small, quantized models (4B to 27B parameters) that also run on standard hardware with usable VRAM up to 24 GB, I installed llama.cpp. These models often work with fewer tokens per second and have increasingly been equipped with their own internal reasoning. This gave users in this segment thinking models too — ones that generate a more or less lengthy chain of thought before the actual answer.
The trigger for rebuilding my Political Compass was a concrete incident. When testing Gemma-4-12b (gemma-4-12b-it-ud-q6_k_xl-spark) on my llama.cpp instance, a single question block took around 16,850 seconds for eight questions. That works out to roughly 35 minutes per question. Extrapolated to the full questionnaire run twice, a single model run would have taken several days.
What specifically caused this spike could only be reconstructed after the fact from server and client logs. The trail ultimately led back to a single question.
2. Three hypotheses, one causal chain
The diagnosis followed a classic process of elimination. My first suspicion — that frequent refusal retries were to blame — was quickly ruled out. The observed retry cycles themselves ran fast and did not account for the time difference.
Looking at the logs revealed the actual finding. A single question had driven the model into what appeared to be an endless reasoning loop: exactly 25,000 generated tokens, 30.1 minutes of compute time at roughly 14 tokens per second, before the response ended in a refusal. The question that triggered this behavior:
"Should companies pay for automation-related job losses?"
A seemingly simple yes-or-no question, embedded in a dilemma with four sharply opposed answer options — ranging from a radical, legally mandated automation tax to an unabashed "survival of the fittest" for the labor market. Exactly the kind of rhetorically charged flashpoint where a thinking model prefers to keep weighing options rather than committing quickly.
The root cause ran deeper and was not correctable with a simple parameter adjustment. The Political Compass was added to the framework after the fact as a standalone diagnostic module, without ever receiving its own budget entry. Because the module does not feed into the scoring, this gap also went unnoticed in any standard evaluation. Without an explicit budget value, the central budget resolution for reasoning models falls back to a default of 25,000 tokens in such a case. That value is a reasonable choice for modules with extensive text generation, but grotesquely oversized for an expected single-letter answer.
A secondary finding sharpened the picture. The observed behavior initially looked like a refusal or alignment problem, but was primarily a truncation problem. The model thought for a long time and was ultimately cut off before it could deliver an answer.
Three distinct error classes had been running through the same escalation mechanism up to that point: an actual content refusal, a budget-triggered abort mid-reasoning, and a pure format deviation with no discernible letter. This meant every retry attempt on an actual truncation problem cost the full generation time again, without addressing the underlying issue.
3. Is a compass question actually a reasoning task?
This led me to a fundamental methodological question. Is a Political Compass question genuinely a reasoning task? Does the answer benefit from longer deliberation, or should it be more of a forced reflex response — one where the trained, immediate association is more informative than a carefully, multi-step weighed answer? The Political Compass is meant to surface the baseline disposition embedded in the model, not the ability to weigh pros and cons.
The analysis led me to a clear conclusion: runaway internal reasoning can actually undermine the purpose of the Political Compass. A model that thinks its way into a diplomatically smoothed middle position obscures the underlying tendency rather than revealing it.
From this reasoning came the decision against a simple parameter fix and in favor of a systematic, upfront test. A standalone calibration probe that, for each model before the actual run, clarifies how it behaves under different token budgets.
4. From a budget number to a profile decision
A first, simple version of this probe tested four questions from four of the nine thematic blocks at a fixed, low budget. That was not enough. Block-level outliers — thematic blocks where a model produces noticeably longer internal reasoning chains than in others — remained invisible in this coarse sample.
The revised, two-stage version addresses this systematically. In a first, inexpensive stage, exactly one question per thematic block is tested at a low budget. This flags suspicious blocks without incurring meaningful additional cost. In a second, targeted stage, only the blocks flagged as suspicious are retested with multiple questions and a gradually increasing budget.
The result of this probe is no longer a single number but a profile decision that is stored in the model configuration.
A model is classified as self-limiting if it consistently responds within a calibrated budget. It then receives a single, resource-efficient profile. A model is classified as inconsistent if its termination behavior varies strongly across thematic blocks. For this model a hybrid profile is created that includes both a thinking and an instruct run, whose difference additionally reveals whether internal reasoning shifts the measured position at all. A model is classified as persistently runaway if it fails to respond reliably within any tested budget. In this case a pure instruct profile is applied from the outset.
A side finding from the probe campaign deserves special mention because it illustrates the limits of automation. The model Ornith 1.5 was classified by the probe as a hybrid candidate, but does not actually have a separate instruct mode. The consequence is a fixed rule: the profile recommendation from the probe is a recommendation, not a guarantee. It must always be checked against the actual technical capabilities of the respective model before being applied.
5. The uncomfortable consequence
The central turning point in this rebuild is not a technical but a methodological insight. Adjusting the token budget and temperature behavior changes the measurement conditions not just for the problematic model, but for every model that has ever run through the Political Compass. A silent parameter fix would have meant that new and old results sit side by side in the same Leaderboard without having been collected under the same conditions. That kind of inconsistency was something I wanted to avoid.
The consequence was therefore not a patch but a version bump. The module was partly rewritten, the module version was incremented — as was the batch identifier under which results are referenced in the system. Older intermediate results (checkpoints) were automatically invalidated by this version change so that no resume mechanism could accidentally mix data from the old methodology into a new run. This made one thing clear: all existing Political Compass results need to be collected again under the new methodology in order to remain comparable.
6. The rebuild to version 3.0
The diagnosis produced six concrete, interlocking changes.
A dedicated, lean module budget. Political Compass received its own budget value, significantly below the general reasoning fallback, without the usual multiplier applied to reasoning models. Importantly, the budget acts as a minimum value, not a hard ceiling. A higher budget requested for a specific purpose — such as a targeted follow-up attempt — is permitted to exceed the module value.
A four-tier response classification. Instead of a single rejection-detection mechanism, the system now explicitly distinguishes between a clear answer, a budget-triggered truncation, a recognizable content refusal, and a pure format deviation. Each of these four categories receives its own appropriate response, rather than all of them falling into the same escalation mechanism.
Separate retry strategies per run type. In the standard run, a recognizable, genuine refusal is treated as a valid data point. It is not "suppressed" with temperature escalation. Only in the anti-diplomat run — which is explicitly designed for forced positioning — does the escalation ladder (rising temperature plus a reinforcing system addendum) remain in place.
A coverage rule, formulated before data collection. For cases where a thinking model produces strongly divergent results across thematic blocks despite a calibrated budget, a fixed abort rule was defined, based on a cumulative loss rate and a possible concentration of failures on one of the two compass axes. This rule was deliberately formulated before the corresponding live data were available, to rule out post-hoc justification.
Transparent result attribution. If the coverage rule necessitates a replacement run with a different profile, that run operates under its own technical identifier. Its result is nonetheless listed in the Leaderboard under the original model identity, with a visible marker indicating that a profile switch occurred. The first, incomplete thinking run is not discarded but archived as a standalone, diagnostically valuable partial dataset.
An expanded bias report. Instead of a binary was repeated: yes/no, the report now documents the exact escalation level, the temperature used, the triggering condition, and the token distribution between the standard and anti-diplomat runs — for every individual question.
7. A live test of the rules themselves
The coverage rule was not only formulated but actually applied. At least one model in the calibration campaign showed markedly different loss rates across thematic blocks, in some cases concentrated on one of the two compass axes. The response was not an ad-hoc decision but the application of the previously fixed rule: switching to an instruct replacement profile, with the original thinking partial run archived as a documented supplementary dataset. This gives the Political Compass a new methodology that is established before the benchmark runs, rather than individual-case corrections after the fact.
8. Slow and informative
Local benchmark runs for small models now complete in significantly reduced — but transparently acceptable — time. No more single 30-minute outliers per question. The time required once again correlates with actual model size and inference speed, not with a configuration gap. That is the real success of this rebuild. Not that everything runs faster, but that the measured time once again says something about the model.