CrucibleMark is my journey toward evaluating the performance of open-weight models in local and cloud hosting. This path is paved with ever-new ideas and aspects of model evaluation, and the resulting challenge of incorporating these new assessment criteria into the overall score. With the tool-use benchmark I began capturing which model could use tools how well in agentic environments. Unfortunately, this evaluation had not yet affected the score. Until now.
The starting point
The Tool-Use module had been running as a standalone benchmark since its introduction: it checks whether a model actually calls external tools such as web search or HTTP fetch and can evaluate the results, or merely hallucinates them. From my perspective, a critical test for agent pipelines. The module delivered reliable scores from the start, but did not feed into the Total Score. This was intentional at first, since I assumed that only a few LLMs would be capable of using tools. However, because a comprehensive test of all benchmark models led to a different conclusion, it became increasingly important for productive use-case evaluation that this score also flow into the overall rating.
Another point grew more pressing as the benchmark's complexity increased: models that excel as strong all-rounders in everyday use sometimes end up behind pure coding specialists on the Leaderboard. This could not be fixed with a bug fix. The cause lay in the nature of the Total Score calculation itself.
The new path to the score
Starting with Version 5, the Tool-Use module officially enters the overall formula. At the same time, what a model fails to deliver now counts as well: missing or erroneous results flow into the Total Score as a moderate penalty, rather than simply being excluded from the calculation.
The calculation core
The actual formula is deliberately kept simple: a weighted average across all modules:
$$ \text{Score} = \frac{\sum (\text{Module Score} \times \text{Module Weight})}{\sum \text{Module Weight}} $$
The complexity comes not from the formula itself, but from what must be decided beforehand. Which number is actually inserted into the numerator and denominator of the formula for which module?
The classification step before
Before the score is calculated, a classification checks, per model and per module, which of four states a result falls into:
| State | Meaning | Effect on formula |
|---|---|---|
present |
Test ran successfully | Score counts normally in numerator and denominator |
missing / unknown |
Test was scheduled but produced no result | Score = 0 in numerator, weight remains in denominator (penalty) |
incapable |
Model technically does not support the test domain | Excluded entirely, neither numerator nor denominator |
rolling_out / not_deployed |
Module is new, not yet active for all models | Excluded for all models (transition protection) |
This chain of if-then conditions is, at its core, nothing more than a filter that decides before the calculation which values are actually applied.
Why four states instead of a simple yes/no check
A simple yes/no logic would conflate two fundamentally different situations:
- "The model is technically incapable of using tools" is fair and should not be penalized.
- "The model did not complete the benchmark even though it could have" is unfair and should be reflected in the score. This distinction between
incapableandmissingis precisely the added value of the four states. The calculation core always remains the same simple weighted average.
A real-world example
The model "Command A+" illustrates the difference concretely. With the same raw data (numerator 464.23), two entirely different scores result depending on the logic applied:
$$ \text{Penalty formula (current, correct):} \quad \frac{464.23}{7.5} = 61{,}90 $$
$$ \text{Exempt formula (old, incorrect):} \quad \frac{464.23}{6.5} = 71{,}42 $$
The penalty formula is the new, correct logic: the missing module remains in the denominator with its weight of 1 and acts as a penalty. The original exempt formula shows the old logic that applied before the fix, treating the missing module as if it had never existed. The average score of "Command A+" was artificially inflated. This roughly ten-point difference was precisely what triggered the revision of the calculation.
Why strong specialists do not always rank at the top
Anyone who uses a model daily for complex, multi-layered tasks may sometimes wonder why it sits behind a pure coding specialist on the Leaderboard. The obvious assumption would be that code quality is weighted more heavily in the Total Score than other areas — but that is not the case. Every full module, whether Code Quality, UX Writing, Cultural Intelligence, Reasoning, or now Tool Use, enters the formula with the same weight: no module pulls the score more strongly in one direction than another.
Why do LLMs known for coding still tend to rank near the top? Because the Total Score is not a specialist ranking — it measures consistency across all areas. A model that performs solidly to strongly in six out of eight modules wins on average against a model that excels in one area but falls noticeably short in others, even if that one area is what matters most in daily use. Day-to-day usage is almost always linear and focused: work happens predominantly in one or two domains, not simultaneously across all eight that the Total Score maps equally.
That is precisely why the Total Score is deliberately conceived as an entry point, not a final verdict. It provides the first read on a model's overall strength. Anyone wanting to know how it performs specifically in coding, documentation, or another use case will find the relevant sub-scores individually accessible in the expanded Leaderboard. The Total Score opens the door to the rankings; the sub-scores lead deeper into each specific use case.
A related point concerns agentic models, which are often regarded as particularly capable in everyday use yet do not always rank near the top of the Leaderboard. The real strength of orchestrating LLMs lies in the ability to monitor many sub-processes simultaneously, weigh them against each other, and direct other models or tools. CrucibleMark does not currently test this orchestration capability! Every model competes exclusively as an executing LLM that answers a single, linearly formulated task directly. A model that excels in daily use as the conductor of an entire agent system is therefore evaluated here only as an individual musician, not as the orchestra director. This aspect should be kept in mind when interpreting the rankings of strongly agentic models.
What changes for the rankings
For most models, practically nothing changes. Models that were tested cleanly and completed all modules remain largely stable in their ranking positions.
- Top and middle of the field: scores remain largely unchanged (varying in the tenths range)
- Lower positions: this is where the most noticeable shifts appear, because data gaps existed there that should have counted against the affected models
- Excluded remain: models with demonstrably missing capability for a test domain
This is no coincidence — it is the price of a more honest basis for comparison between local, open, and commercial models, all measured against the same questions. The best score in the end is not simply the highest value, but a result that a model has genuinely earned through complete test coverage.