Part 1: How I accidentally built a framework
Prologue: A question that set everything in motion
Ever since I discovered ChatGPT, I had been genuinely blown away by the possibilities AI systems gave me for writing and research. From that point on, I used AI assistance almost daily. Early on I discovered the advantages of Perplexity for day-to-day research, ChatGPT for drafting and revising web copy, and BlackboxAI in the context of my web development work with the CMS Drupal. But my real enthusiasm for this abstract entity began with a naive question: How far can you trust it? How much can you rely on it? How hard can I push it?
At that point I knew nothing about synthetic benchmarks used to evaluate LLM performance — and I had no desire to. I wasn't interested in a math olympiad or an essay on Kant. I had discovered AI through my work as a UX and frontend designer, and what I wanted to know was whether it would actually make my work easier. Or whether it would just create additional correction overhead. The doubt remained.
The real question was more concrete: Can I trust the thing when I let it solve production problems at midnight? Will it tell me in thirty seconds how to set a symlink, deploy a Docker container, or anchor an alias persistently in .zshrc — or will it spin in circles through pages of output that look like a lot and deliver little?
But there was something else. As a UX and frontend designer, I have worked closely with developers for years — I know the effort involved, I know the dependencies, and I know the moments when I had to simplify layouts because the resources weren't there to implement a simple JavaScript feature. From the start, AI was also this for me: the chance to push into areas that had previously been closed off. Not by learning, but by doing.
That was my starting point. No research project. No planned framework. An experiment driven by doubt — and by the quiet hope that my skills could genuinely expand through it.
And the experiment had a clearly defined end: I would test it, get my answer, and be done.
But things turned out differently.
Phase 1: The first benchmark — and what it showed
The early tests were simple. Local models via Ollama on my consumer hardware — the fact that alongside the big commercial services there were also open models for running at home fit perfectly with my enthusiasm for open source. The key was Ollama, and I'll write more about my experiences with it elsewhere. I enthusiastically installed local versions of big names: Llama, DeepSeek, Qwen, Mistral. But which model, in which size, for which purpose? Was there one for everything — or did I need different ones for different tasks? And at what point did the time factor for response generation outweigh the actual benefit?
I put together a handful of manually assembled tasks from my real working life. No elaborate scoring, no Judge, no Leaderboard. Just: does it do what I say?
The results were sobering and fascinating in equal measure. Models that shone in public benchmarks failed at fundamental CLI tasks. Others that nobody had on their radar — Ministral, for example — delivered reliably and quickly. I had never heard of Ministral before. That changed. The first real aha moment was surprisingly simple: the metrics by which the AI industry evaluates models have almost nothing to do with my work.
Later I learned there's a name for this: Goodhart's Law — when a measure becomes a target, it ceases to be a good measure. Whether horsepower, processor performance, or megapixels — benchmark results are readily sold in marketing as proof of quality, even though they often measure synthetic scenarios that don't appear in my daily work. DeepSeek V3.2 was the earliest documented example in my setup: academically strong, 27 timeouts out of 43 tasks in practice.
And instead of wrapping up my experiment there, I wrote the first real test assets.
Phase 2: The crucible takes shape
The name CrucibleMark didn't come by accident. Since LLMs felt to me like theoretical black boxes, I wanted to illuminate them from every angle, put them under stress, and push them to their limits — in order to understand them better. In a crucible, materials are heated and tested under pressure to reveal what is really inside them. That was exactly what I wanted to do with these models: place them in a controlled context and observe how they behave — not in an exam situation, but under working conditions.
The modularity emerged later, not from a plan but from need. First Code Quality. Later CLI Operations. Then Logical Reasoning, because I quickly noticed there is a difference between a model that knows the right command and one that understands why the command is right. Then UX Writing, because in my day-to-day work I write microcopy constantly and wanted to know which model could genuinely support me in that — and which was merely going through the motions.
The framework didn't grow according to a master plan. It grew according to the questions that the work raised. And with every new question I noticed: I was already asking them differently than I had a few weeks before.
Phase 3: The regex problem and the big break
Somewhere between the third and fourth module, the limits of rule-based scoring began to show.
The original evaluation system combined keyword matching, regex checks, and semantic similarity analysis. It was precise for structured outputs — tables, code segments, CLI responses with a clear right-or-wrong. But it was blind to quality and had to be manually recalibrated for every new model — a maintenance burden that no longer scaled with a growing model pool. A model that technically hit all the keywords but produced nonsense got a good score. A model that solved the task elegantly but chose different words was penalized.
Hermes 3 was the first model to bring this problem into sharp focus. The rule-based scorer had rated it positively for a long time — too positively, as it turned out. CLI tasks with clear input-output structure: 82.7 points. Reasoning across multiple steps: 51.3 points. The gap was there — but the scoring system didn't see it fully.
The solution was conceptually clear once I had formulated the question: What if a second, independent language model evaluated the answers? It reads the original task, the tested model's response, and a Golden Standard — representing the ideal reference answer — and assigns a score with a mandatory justification. Chain of Thought before the score, not after — an insight I hadn't come up with myself but learned from the research. For the first time I was researching not for a piece of writing, but for a system.
That was the break. The switch from the rule-based hybrid scorer to the LLM-as-a-Judge system.
The night the framework was born
What sounded simple in theory was a complex engineering problem in implementation. And here something happened that I hadn't expected: collaborating with an AI agent — Claude Sonnet in Agentic Mode — fundamentally transformed the process.
The requirements document emerged from a long conversation with Perplexity, which had become very important to me as a brainstorming and research assistant. The conversation wasn't about implementation details but about architectural principles, risks, and design decisions. The uncomfortable questions: What happens when the Judge and the tested model are in RAM at the same time? What happens with an API timeout in the middle of an overnight benchmark? And what data does a cloud Judge actually get to see — and what happens to it? The fact that a model like Claude Haiku, acting as Judge, evaluates the outputs of its competitors is not just a data privacy question. It's one I only fully thought through later: What if the judge itself learns from what it sees?
These questions don't come from a specification. They come from experience — and growing curiosity.
The next day began what was, for me, a landmark experiment: based on the requirements document, I estimated the approximate effort a developer would have needed for the implementation — in time and money. The regional hourly rate for freelance developers was quickly established. I then handed the finished document to GitHub Copilot in VS Code, with Claude Sonnet 4.6 in agent mode. Fifteen minutes later: 15 new files. 5 modified files. 35 new tests. All green. After another three hours of fine-tuning, the Judge was running. An experienced senior developer would realistically have needed 6–8 working days for the same scope.
The cost for the entire implementation of the Judge module: $2.08.
That's not an exaggeration. That was the moment I understood what I was dealing with. And it explains why a small experiment became a framework that occupied me for months — sometimes almost obsessively. The productivity explosion made things feasible that I would never have tackled alone before. And something happened that I hadn't anticipated: the tool I had built to answer questions generated new questions faster than I could answer the old ones. The more the framework delivered, the more I wanted to test. The more I tested, the clearer it became what was still missing. The test had become a project — I just hadn't noticed yet.
Saturation as the next problem
With a functioning Judge system and a growing model pool, the next strategic question arose almost on its own: What happens when all models eventually land at 95% or above? Then the benchmark becomes too smooth, too comfortable, too easy to game. It loses its ability to differentiate — its discriminative power, as I had by then learned to call it.
I had this discussion in passing with various AI assistants, which was itself a meta-commentary on the whole experiment. The direction was clear: don't harden globally, but modularly. Don't overwhelm all models with the same harder tasks, but tighten the screws selectively. The first lever was obvious: make the standard prompt stricter and identify the anti-diplomat mode for this purpose. A more pedantic judge pushes the scores of all models uniformly downward without a single new question needing to be written. So the actual calibration began.
The result was a framework that had grown far beyond the original plan: seven modules, a hybrid scoring system that calculates instruction score and reasoning score separately; an LLM Judge with provider abstraction; a Leaderboard with 61 tested models across three categories. Version 3.5.9. Pylint average 9.15 out of 10.
What had begun as an experiment had become a tool. And at some point I realized that it wasn't only the framework that had grown. The UX designer who had started all this out of curiosity had become someone who no longer merely operated the system, but understood it ever more deeply, shaped it ever more precisely, and grew a little further with it through each iteration. In the end I had become not just the observer of the whole thing, but its architect. But the most interesting part was still ahead of me.