Part 1: How I accidentally built a framework
Prologue: A question that set everything in motion
Ever since discovering ChatGPT, I had been genuinely blown away by the possibilities AI systems offered me for writing and research. From that point on, I used AI assistance almost daily. Early on I discovered the advantages of Perplexity for day-to-day research, ChatGPT for drafting and revising web copy, BlackboxAI in the context of my web development work with the CMS Drupal. But my real enthusiasm for this abstract entity began with a naive question: How far can you trust it? How much can you rely on it? How hard can I push it?
At that point I knew nothing about synthetic benchmarks used to evaluate LLM performance — and I had no desire to. I wasn't interested in a math olympiad or an essay on Kant. I had discovered AI through my work as a UX and frontend designer, and what I wanted to know was whether it would genuinely make my work easier. Or whether it would just create additional correction overhead. The doubt remained.
The actual question was more concrete: Can I trust the thing when I let it solve production problems at midnight? Will it tell me in thirty seconds how to set a symlink, deploy a Docker container, anchor an alias persistently in .zshrc — or will it spin in circles through pages of output that look like a lot and deliver little?
But there was something else. As a UX and frontend designer, I have worked closely with developers for years — I know the effort involved, I know the dependencies, and I know the moments when I had to simplify layouts because the resources weren't there to implement a simple JavaScript feature. From the beginning, AI was also this for me: the chance to move into areas that had previously been closed off. Not by learning, but by doing.
That was my starting point. No research project. No planned framework. An experiment driven by doubt — and by the quiet hope that my skills could genuinely expand through it.
And the experiment had a clearly defined end: I would test it, get my answer, and be done.
But it didn't work out that way.
Phase 1: The first benchmark — and what it showed
The early tests were simple. Local models via Ollama on my consumer hardware — the fact that alongside the major commercial services there were also open models to run at home fit perfectly with my enthusiasm for open source. The key was Ollama, and I'll write more extensively about my experiences with it elsewhere. I enthusiastically installed local versions of big names: Llama, DeepSeek, Qwen, Mistral. But which model in which size for which purpose? Was there one for everything — or did I need different ones for different tasks? And at what point did the time factor for response generation outweigh the actual benefit?
I put together a handful of manually assembled tasks from my real working life. No sophisticated scoring, no Judge, no Leaderboard. Just: does it do what I say?
The results were sobering and fascinating in equal measure. Models that shone in public benchmarks failed at fundamental CLI tasks. Others that nobody had on their radar — Ministral, for example — delivered reliably and quickly. Until then I had never heard of Ministral. That changed. The first real aha moment was surprisingly simple: the metrics by which the AI industry evaluates models have almost nothing to do with my work.
Later I learned there's a name for this: Goodhart's Law — when a measure becomes a target, it ceases to be a good measure. Whether horsepower, processor performance, or megapixels — benchmark results are readily sold in marketing as proof of quality, even though they often measure synthetic scenarios that don't appear in my daily work. DeepSeek V3.2 was the earliest documented example in my setup: academically strong, 27 timeouts out of 43 tasks in practice.
And instead of wrapping up my experiment there, I wrote the first real test assets.
Phase 2: The crucible takes shape
The name CrucibleMark didn't come about by chance. Since LLMs somehow resembled theoretical black boxes to me, I wanted to illuminate them from every angle, stress them, and push them to their limits — in order to understand them better. In a crucible, materials are heated and tested under pressure to reveal what is truly inside them. That was exactly what I wanted to do with these models: place them in a controlled context and observe how they behave — not in an exam situation, but under working conditions.
The modularity emerged later, not from a plan but from necessity. First Code Quality. Later CLI Operations as well. Then Logical Reasoning, because I quickly noticed there is a difference between a model that knows the right command and one that understands why the command is right. Then UX Writing, because in my daily work I write microcopy constantly and wanted to know which model could genuinely support me in that — and which was merely going through the motions.
The framework didn't grow according to a master plan. It grew according to the questions that the work raised. And with every new question I noticed: I'm already asking them differently than I was a few weeks ago.
Phase 3: The regex problem and the big break
Somewhere between the third and fourth module, the limits of rule-based scoring began to show.
The original evaluation system combined keyword matching, regex checks, and semantic similarity analysis. It was precise for structured outputs — tables, code segments, CLI responses with a clear right-or-wrong answer. But it was blind to quality and had to be manually recalibrated for every new model — a maintenance burden that no longer scaled with a growing model pool. A model that technically hit all the keywords yet produced nonsense received a good score. A model that solved the task elegantly but chose different words was penalized.
Hermes 3 was the first model to bring this problem into sharp focus. The rule-based scorer had rated it positively for a long time — too positively, as it turned out. CLI tasks with clear input-output structure: 82.7 points. Reasoning across multiple steps: 51.3 points. The gap was there — but the scoring system didn't see it fully.
The solution was conceptually clear once I had formulated the question: What if a second, independent language model evaluated the answers? It reads the original task, the response of the model being tested, and a Golden Standard — representing the ideal reference answer — and assigns a score with a mandatory justification. Chain of Thought before the score, not after — an insight I hadn't come up with myself but learned from the research. For the first time I was researching not for a piece of writing, but for a system.
That was the break. The shift from the rule-based hybrid scorer to the LLM-as-a-Judge system.
The night the framework was built
What sounded simple in theory was a complex engineering problem in implementation. And here something happened that I hadn't anticipated: collaborating with an AI agent — Claude Sonnet in agentic mode — fundamentally transformed the process.
The requirements document emerged from a long conversation with Perplexity, which had become very important to me as a brainstorming and research assistant. The conversation wasn't about implementation details but about architectural principles, risks, and design decisions. The uncomfortable questions: What happens when the Judge and the model under test are in RAM at the same time? What happens with an API timeout in the middle of an overnight benchmark? And what data does a cloud Judge actually get to see — and what happens to it? The fact that a model like Claude Haiku as Judge evaluates the outputs of its competitors is not just a data privacy question. It's one I only fully thought through later: What if the judge learns from what it sees?
These questions don't come from a specification. They come from experience — and growing curiosity.
The next day began an experiment that proved pivotal for me: based on the requirements document, I estimated the approximate effort a developer would have needed for the implementation — in time and money. The regional hourly rate for freelance developers was quickly established. I then handed the finished document to GitHub Copilot in VS Code, with Claude Sonnet 4.6 in agentic mode. 15 minutes later: 15 new files. 5 modified files. 35 new tests. All green. After another three hours of fine-tuning, the Judge was running. An experienced senior developer would realistically have needed 6–8 working days for the same scope.
The cost for the entire implementation of the Judge module: $2.08.
That's not an exaggeration. That was the moment I understood what I was dealing with here. And it explains why a small experiment became a framework that occupied me for months — sometimes almost obsessively. The productivity explosion made things feasible that I would never have tackled alone before. And something happened that I hadn't counted on: the tool I had built to answer questions generated new questions faster than I could answer old ones. The more the framework delivered, the more I wanted to test. The more I tested, the clearer it became what was still missing. The test had become a project — I just hadn't noticed yet.
Saturation as the next problem
With a functioning Judge system and a growing model pool, the next strategic question arose almost on its own: What happens when all models eventually land at 95% or above? Then the benchmark becomes too smooth, too comfortable, too easy to game. It loses its discriminative power — as I had by now learned to call it.
I had that discussion in passing with various AI assistants, which was itself a meta-commentary on the whole experiment. The direction was clear: don't harden globally, but modularly. Don't overwhelm all models with the same harder tasks — tighten the screws selectively. The first lever was obvious: make the Judge prompt stricter and determine the best Judge for this purpose. A more pedantic judge pushes the scores of all models uniformly downward without a single new question needing to be written. So the actual calibration began.
The result was a framework that had grown far beyond the original plan: seven modules, a hybrid scoring system that calculates Instruction and Reasoning scores separately; an LLM Judge with provider abstraction; a Leaderboard with 61 tested models across three categories. Version 3.5.9. Pylint average 9.15 out of 10.
What had begun as an experiment had become a tool. And at some point I realized that it wasn't only the framework that had grown. The UX designer who had started all of this out of curiosity had become someone who no longer merely operated this system, but understood it better and better, shaped it more and more precisely, and grew a little further with it with each iteration. In the end I had become not just the observer of the whole thing, but its architect. But the most interesting part was still ahead of me.