LLM Model Review
· Instruction-Tuned
Hermes 4 405B achieves an overall score of 68.4 percent and carries the speed profile Interactive Tool Expert. That label fits the character of this model surprisingly well: a large, dense server-class generalist with 405 billion parameters — direct, format-conscious, often useful on the first attempt, but without the authority one would reasonably expect at this scale. The tested deployment was a cloud Open Weights instance via OpenRouter running the endpoint’s factory default behavior; no switchable thinking mode was available in this run, even though the architecture supports Extended Thinking in principle. Sovereign Risk: LOW: Nous Research is a US-based company and subject to the CLOUD Act; however, the weights are publicly available and can be run locally, so no third-party API access is required.
Header Notes: Stability and Reliability
| Metric | Value | Rating | Analysis |
|---|---|---|---|
| Timeout Rate | 1/49 | Sporadic | The model shows sporadic dropouts that would require retries in practice. For a cloud Open Weights endpoint, this is not background noise — it is a concrete API stability risk. |
| P95 Response Time | 51.12 s | Acceptable | Isolated outliers, still tolerable for interactive use. The tail is visible enough, however, to occasionally slow down workflows and agent pipelines. |
Architecture and Expectations
The metadata Instruct, Thinking-Optional explains more about Hermes 4 405B than the name suggests. As an Instruct model it is calibrated for direct command execution: clear answers, minimal unnecessary reasoning, solid format discipline. As a Thinking-Optional model it does bring multi-step inference in principle, but that capability was not explicitly activated during the benchmark. What is being evaluated is therefore the behavior a regular API user actually gets without any special switches. That is methodologically clean and, in practice, even more relevant than any lab-based reasoning assessment.
Add to that the editorial classification as a Generalist, Server-class, Dense architecture. This matters because it keeps the list of excuses short. A dense 405B model activates its full capacity with every response. There is no MoE argument available here along the lines of “only a small fraction of the weights are active.” A model that enters the ring at this scale has to deliver across the board. Hermes 4 405B does that in places. But it does not feel like a model that converts its sheer mass into precision. More like a large aggregate running in second gear.
Performance and Cost Profile
The speed profile Interactive Tool Expert certifies that Hermes 4 405B is positioned for interaction-oriented use. In practical terms: not a batch machine for kilometer-long documents, but a model that is supposed to remain dialogue-capable in tool and assistant scenarios. The measured generation speed is qualitatively decent for that purpose, but not spectacular. An important note for the reader: for this cloud Open Weights model, throughput is a benchmark of the OpenRouter endpoint and its underlying cloud infrastructure — not an abstract property of the weights file in a vacuum.
On the positive side: token economy. Hermes 4 405B behaves token-economically; no module exceeds the expected verbosity range. In CLI, Code Quality, Documentation, Content Transformation, Cultural Intelligence, and UX Writing it stays below the fleet median in each case. The model does not talk too much on principle. Especially in API use, that is a genuine advantage, because brevity here is not just a stylistic preference but cost control. Only in Reasoning does it become clear that brevity does not automatically mean sharpness.
Code Quality and Security: Good Checklist, Mediocre Security Advisor
In the code and audit domain, Hermes 4 405B shows one of its more useful sides. It delivers structured tables, identifies the right order of magnitude of issues in a PHP security audit, and maintains clean formatting throughout. That sounds trivial, but it is not. Many models fail at the basic task of keeping analysis and output format aligned. Hermes 4 405B does not.
The qualitative catch is depth. In the security audit the model does identify 19 vulnerabilities — hitting the correct count — but explanations and fixes frequently stay at the surface. “SQL Injection” becomes a correct but abstracted diagnosis. What is missing are concrete attack paths, real payload examples, and actionable remediation suggestions with context. The judge describes this aptly as a useful checklist, but not a sufficient explanation of actual exploit vectors. For beginners that is still workable. For a team that needs to genuinely prioritize and close a finding, it is too thin.
That is precisely where the difference between large and great becomes visible. Hermes 4 405B sees the problem areas, but it does not walk the reader through the building. It rates Path Traversal too low, and Session Fixation likewise. Particularly problematic is the imprecise decomposition of individual vulnerabilities in the admin-delete path. The model recognizes the pattern but does not dissect it with the precision one would expect from a server-class generalist of this size. In security matters, half-knowledge is not a charming quirk — it is a liability risk in text form.
Reasoning and Logic: Correct Destination, Unnecessarily Winding Route
The Reasoning module is where Hermes 4 405B most clearly demonstrates what the Thinking-Optional category means in default behavior. It can solve logical tasks, but the answers do not feel cohesive. On the classic guard riddle the model arrives at the correct solution, but the path there is unnecessarily meandering, with self-corrections and repeated recalculation of the same logic. This is not a reasoning collapse — more like an internal stumble at full speed.
More significant is the format issue. In three metacognitive reasoning tasks the model responded predominantly in English despite an explicit German-language instruction. This is not an isolated outlier. Across multiple tasks in the Reasoning module the model shows a consistent pattern: when language, format, and reasoning instructions are given simultaneously, the language instruction is the first constraint to be dropped. In production environments with a fixed target language, that is not a minor inconvenience — it is a direct compliance failure.
Metacognition Compliance (Reasoning): The model does not refuse the use of explicitly requested <thought> tags in 3/5 metacog tests with a consistent policy statement, but it fails systematically on format and language compliance. The reasoning content is partially correct on substance — the score deduction here results primarily from poor instruction-following, not from complete logical failure. CrucibleMark deliberately evaluates native zero-shot instruction compliance as a real-world characteristic. That is precisely where Hermes 4 405B loses points unnecessarily.
There is also a practical note: this module was where the only timeout of the entire run occurred, and the tail was particularly long here. No need to read too much into that. It simply shows that the model tends toward outliers at the logical boundary. For interactive puzzle tasks that is annoying. For agentic systems that depend on tight response windows, it quickly becomes a tactical problem.
Content Transformation: Strong Restructuring, Not Always Strict Enough
In the Content Transformation module, Hermes 4 405B demonstrates why one might want to work with it. The rework of a YouTube script succeeds with clean German prose, a coherent analysis phase, realistic pacing, and a surprisingly good density of production notes such as screen annotations, music cues, and pattern interrupts. The model understands not just the text but the mechanics behind it. That is more than pretty rewriting.
Particularly encouraging is that Hermes 4 405B avoids the typical trap many large models fall into — turning every transformation into a novel. It stays comparatively disciplined, builds complete structure, and hits a conversational tone without sliding into rambling. The judge flags minor structural deviations, such as embedding troubleshooting content rather than giving it a dedicated section. That is legitimate criticism, but not a demolition.
A core character trait of the model is visible here: when the task calls for creative recombination within clear guardrails, Hermes 4 405B performs noticeably more convincingly than in strict analytical deep-work. It is better at restructuring than at proving.
Cultural Intelligence and UX: Solid, but with Social Rough Edges
In the Cultural Intelligence domain, Hermes 4 405B delivers usable to good results. Defusing toxic or exclusionary phrasing works at its core. The model replaces problematic expressions meaningfully and keeps the language natural. The catch is the last mile: in an HR-adjacent rewrite it reintroduces a gendered noun — “Expertin” — after having already neutralized language elsewhere. That is not a gross error, but exactly the kind of inconsistency that damages the professional impression in sensitive text types.
Tonally, Hermes 4 405B also tends to remain somewhat transactional. Where the better solution would be warm and inviting, Hermes more often sounds like a clean requirements list. That is typical Instruct: precise, concise, dutiful. But in recruiting, UX, or community copy, the winner is not the correct bureaucrat — it is the author with human insight. Hermes 4 405B is solid here, not perceptive.
Tool Use and Hallucinations: Strong Foundation, Two Red Cards
The numerical Tool Use score looks strong on paper. The qualitative record, however, does not make it unqualified praise. In two tool-related tasks the model hallucinated content that was not present in the retrieved tool output. That is not a minor slip — it is exactly the kind of error that disqualifies research and reporting workflows. When a model invents something during tool-assisted work, it breaks the implicit contract with the user: I fetched data for you, so stay with the data.
That finding deserves its own heading.
Hallucinations and Factual Integrity
Hermes 4 405B shows a genuine weakness in factual fidelity on tool-assisted tasks. In two cases it generated content that did not originate from the tool output; scoring explicitly capped the value there via a hallucination cap. For content-critical tasks such as research, situation reports, or any form of fact-bound summarization, this is a disqualifying signal.
These hallucinations carry more weight because they do not arise from a free creative task — they occur in a setting with an external data source. That is precisely where a model must be able to distinguish between source and embellishment. Hermes 4 405B cannot do that reliably enough. Anyone deploying it in tool workflows needs validation layers. Otherwise you are building an assistant that works with files and still fabricates.
Privacy and Data Sovereignty
For the model source itself, the situation is comparatively clear: Nous Research is a US company headquartered in San Francisco. The weights are available under the Meta Llama Community License as Restricted Weights; according to the available card data, Nous does not operate its own public API. The vendor card therefore lists the applicable law for deployment as “N/A (local/self-hosted)”, data location as “Local or third-party hoster”, data retention as 0 days, and no GDPR DPA.
For this specific benchmark, however, the decisive factor is that Hermes 4 405B ran as a cloud Open Weights model via OpenRouter. This shifts the data protection question from the model vendor to the actual hosting and routing path. Based on the available card data, a calculated Sovereign Risk: LOW results, because the weights are openly available and no mandatory API access to Nous itself is required. For companies in Germany or the EU, a practical caveat remains: without reliable, verified provider data on the specific deployment infrastructure and without a DPA, a clean GDPR assessment for enterprise use cannot be completed. The CLOUD Act remains fundamentally relevant for US vendors. Even if data were physically located in Europe, US authorities can demand access under certain conditions.
Conclusion
Hermes 4 405B is an interesting contradiction. As a large, dense server-class generalist it arrives with the aura of a heavyweight, but often behaves like a sensible mid-range assistant with good manners and limited bite. It is token-economical, format-stable, often convincing in restructuring and production tasks, and initially pleasantly direct in tool handling. Especially as an Instruct model in default behavior, that directness is not a deficiency — it is part of the design.
Its limitations are too pronounced to explain away, however. Security analyses remain too shallow, Reasoning suffers from unnecessary roundabout thinking and weak language compliance, and the hallucinations in tool-assisted tasks are a serious warning signal for fact-critical use. Add to that only acceptable real-world stability with sporadic API dropouts. That is not a total write-off. But it is also not the kind of 405B model you hand the keys to your production system without a second thought.
Recommendation: well suited for structured transformation, script reworks, general assistance, and tool-adjacent interaction with human review. Only conditionally suitable for security audits requiring analytical depth, format-strict multilingual reasoning tasks, and any research work where fabricated details are not negotiable. In short: Hermes 4 405B has form. It is just missing a measure of judgment.
This evaluation was generated automatically based on the benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.
