CrucibleMark is my journey toward evaluating the capabilities of open-weight models in local and cloud hosting. This path is paved with ever-new ideas and aspects of model evaluation, and the resulting challenge of incorporating these new criteria into the overall score. With the tool-use benchmark I began capturing which model could use tools how well in agentic environments. Unfortunately, this evaluation had not yet affected the score. Until now.
The starting point
The tool-use module has been running as a standalone benchmark since its introduction: it checks whether a model actually calls external tools such as web search or HTTP fetch and can evaluate the results, or merely hallucinates them. From my perspective, a critical test for agent pipelines. The module delivered reliable scores from the start, but did not feed into the Total Score. This was intentional at first, as I assumed only a few LLMs would be capable of using tools. Since an extensive test of all benchmark models led to a different conclusion, it became increasingly important for evaluating LLMs in production use that this score also be included in the overall calculation.
Another point also grew more important as the benchmark increased in complexity: models that perform as strong all-rounders in everyday use sometimes land behind pure coding specialists on the Leaderboard. This could not be fixed with a bugfix. The cause lay in the nature of the Total Score calculation itself.
The new path to the score
Starting with version 5, the tool-use module officially enters the overall formula. At the same time, what a model fails to deliver now counts as well: missing or erroneous results flow into the Total Score as a moderate deduction, rather than simply being excluded from the calculation.
The calculation core
The actual formula is deliberately kept simple: a weighted average across all modules:
$$ \text{Score} = \frac{\sum (\text{Modul-Score} \times \text{Modul-Gewicht})}{\sum \text{Modul-Gewicht}} $$
The complexity comes not from the formula itself, but from what must be decided beforehand. Which number is actually inserted into the numerator and denominator of the formula for each module?
The classification step
Before the score is calculated, a classification checks per model and module which of four states a result falls into:
| State | Meaning | Effect on formula |
|---|---|---|
present |
Test ran successfully | Score counts normally in numerator and denominator |
missing / unknown |
Test was scheduled but returned no result | Score = 0 in numerator, weight remains in denominator (penalty) |
incapable |
Model technically does not support the test domain | Excluded entirely, neither numerator nor denominator |
rolling_out / not_deployed |
Module is new, not yet active for all models | Excluded for all models (transition protection) |
This chain of if-then conditions is at its core nothing more than a filter that decides, before the calculation, which values are actually applied.
Why four states instead of a simple yes/no check
A simple yes/no logic would conflate two fundamentally different situations:
- "The model is technically incapable of using tools" is fair and should not be penalized.
- "The model did not complete the benchmark even though it could have" is unfair and should be reflected in the score. This distinction between
incapableandmissingis precisely the added value of the four states. The calculation core remains the same simple weighted average throughout.
A real-world example
The difference becomes concrete with the model "Command A+". With the same raw data (numerator 464.23), two entirely different scores result depending on the logic applied:
$$ \text{Malus-Formel (aktuell, korrekt):} \quad \frac{464.23}{7.5} = 61{,}90 $$
$$ \text{Exempt-Formel (alt, fehlerhaft):} \quad \frac{464.23}{6.5} = 71{,}42 $$
The penalty formula is the new, correct logic: the missing module remains in the denominator with its weight of 1 and acts as a penalty. The original exempt formula shows the old logic that applied before the fix, treating the missing module as if it had never existed. The average score of "Command A+" was artificially inflated. This difference of roughly ten points was precisely what triggered the revision of the calculation.
Why strong specialists do not always rank at the top
Anyone who uses a model daily for complex, multi-layered tasks will sometimes wonder why it sits behind a pure coding specialist on the Leaderboard. The obvious assumption would be that code quality is weighted more heavily in the Total Score than other areas — but that is not the case. Every full module, whether Code Quality, UX Writing, Cultural Intelligence, Reasoning, or now Tool Use, enters the formula with the same weight: no module pulls the score more strongly in one direction than another.
Why do LLMs known for coding still tend to rank near the top? Because the Total Score is not a specialist ranking — it measures consistency across all areas. A model that performs solidly to strongly in six out of seven modules wins on average against a model that excels in one area but falls noticeably short in others, even if that one area matters most in everyday use. Daily usage is almost always linear and focused: one works predominantly in one or two domains, not simultaneously across all seven that the Total Score maps evenly.
That is precisely why the Total Score is deliberately intended as an entry point, not a final verdict. It provides the first read on a model's overall strength. Anyone wanting to know how it performs specifically in coding, documentation, or another use case will find the relevant sub-scores individually accessible in the expanded Leaderboard. The Total Score opens the door to the rankings; the sub-scores lead deeper into the respective use case.
A related point concerns agentic models, which are often considered particularly capable in everyday use yet do not always rank near the top of the Leaderboard. The real strength of orchestrating LLMs lies in the ability to monitor many sub-processes simultaneously, weigh them against each other, and direct other models or tools. CrucibleMark does not currently test this orchestration capability. Every model competes exclusively as an executing LLM that answers a single, linearly formulated task directly. A model that excels in everyday use as the conductor of an entire agent system is therefore evaluated here only as an individual musician, not as an orchestra director. This aspect should be kept in mind when interpreting the rankings of models with a strongly agentic orientation.
What changes for the rankings
For most models, practically nothing changes. Models that were tested cleanly and completed all modules remain largely stable in their ranking position.
- Top and middle of the field: scores remain largely unchanged (varying in the tenths range)
- Lower positions: this is where the most noticeable shifts appear, because data gaps existed there that should have counted against the affected models
- Excluded remain: models with demonstrably missing capability for a test domain
This is no coincidence — it is the price of a more honest basis for comparison between local, open, and commercial models, all measured against the same questions. The best score in the end is not simply the highest value, but a result that a model has genuinely earned through complete test coverage.