There are decisions you keep putting off until everyday life makes them for you.
I had been eyeing the NVIDIA DGX Spark for a while. As a compact local inference system with 128 GB of unified memory, it was theoretically exactly what I would have needed to finally run server-class models privately. Theoretically. In practice, two things consistently held me back: the price of the device and the steadily rising cost of memory, which reliably keeps pushing this segment higher. I was hoping for better times. But better times never came.
Then June 2026 arrived.
What has happened to API and subscription prices over the past few weeks thoroughly upended my calculations. The final push came from GitHub Copilot's pricing adjustment. I use the tool almost daily — it has shaped the way I work as a freelancer for years — and I had always weighed the expensive DGX Spark against the monthly subscription costs. In early June 2026, that price increased by a factor of 8 for the same usage (link to article). Not gradually. Just like that. That fundamentally changed my attitude toward investing in the Spark. Anyone who uses LLMs productively the way I do, without a large corporate AI budget, is currently watching the ground fall out from under their cost calculations. And I don't think I'm alone in that.
So I ran the numbers. And then I ordered the Spark.
The logic behind it was less optimism than pragmatism. Either memory prices keep rising and the device will be even more expensive in six months — in which case now was the right time to get in. Or prices normalize and I have a solid piece of hardware with real residual value that I can resell if needed. Both outcomes are acceptable to me. What was not acceptable: simply continuing to pay a monthly bill that was rising sharply, without at least having looked for an answer.
Before new models could be tested, the foundation had to be right
In parallel with the hardware decision, CrucibleMark underwent an extensive refactoring. The benchmark originally started as a simple CSV list. The technical debt that had accumulated across several versions made a clean break necessary. The entire score and review mapping was migrated to an ID-based system. Each model now has a single source of truth: a unique ID through which all scores, reviews, and metadata are consistently assigned. That sounds like housekeeping. But it is the prerequisite for keeping results reliably comparable as the model base grows as fast as it currently is.
And it is growing. New models worth taking seriously are appearing almost weekly. I am now also testing server-class models intensively — models that can run on the Spark with its 128 GB of RAM, models that simply would not run on my M4 Mac and had previously fallen outside my testing range. That is the actual announcement of this post. The benchmark is gaining a new dimension.
What CrucibleMark now aims to discover
The original guiding question was: at what point can a locally run model hold its own against established commercial models like Claude Sonnet or GPT? That question remains valid. It just no longer suffices.
Three new dimensions are added to that.
1. Local server models in everyday use. With the DGX Spark, I now run three local inference servers: the M4 Mac, the Spark, and optionally a gaming PC with an RTX 4070 Ti that has so far only been used for testing. Early on, I noticed that the same model can feel completely different on different hardware or at different quantization levels. Different response quality, different timing, sometimes a noticeably different tool-use reliability. I will systematically document these infrastructure comparisons going forward. For anyone who wants to use local inference productively, they are more relevant than raw score values.
2. Chinese frontier models as a commercial alternative. Models like MiniMax M3, GLM 5, or Kimi K2.6 have shown over the past few weeks that they can seriously compete with Anthropic and OpenAI in certain task areas — at a fraction of the cost. A relevant option that I intend to test and evaluate explicitly. But not a neutral one. The use of these models must be weighed against questions of data security and legal frameworks. Especially in the European and German context, that is not a footnote but a genuine requirement. CrucibleMark already accounts for this in its evaluation.
3. Cost efficiency as a standalone benchmark category. What a model can do is one thing. What it costs under real working conditions — in time, in tokens, in API bills — is another. Over the past few weeks I have experienced firsthand how a promising model in a complex development context does not solve problem after problem, but for every problem solved generates one or two new ones. The score on paper is fairly irrelevant in that situation. The real question is: for what kind of tasks is this model actually productive to use, without spending half the saved API costs on debugging hours?
A niche benchmark — and why that need not be a disadvantage
CrucibleMark is not an academic benchmark and not a commercial product. It is a tool I built for myself. As a freelancer who works with LLMs every day, I want to know which model is actually reliable in which situation. The basis of my tests is not the strengths that vendors communicate in their press releases. It is productive scenarios from my own daily work. CrucibleMark is a complement to the large established benchmarks, not a competitor.
Whether that is enough, the results will have to show. I am working on it.
The updated benchmark results are available in the [Scoreboard](/scoreboard/). New tests and comparisons are added on an ongoing basis.