Hermes 4 405B

With an overall score of 68.4%, Hermes 4 405B is not a complete failure, but it is certainly not the sweeping open-weights power demonstration one might reflexively expect from 405 billion dense parameters in the Frontier class. The model was tested as a cloud Open Weights offering via OpenRouter in standard mode without Extended Thinking enabled; the Speed Profile badge reads “Interactive Tool Expert,” targeting interactive tool and structured tasks with solid but not breathtaking responsiveness. The character is clear: direct, mostly disciplined, often useful, but too erratic in logic and linguistic precision for blind trust. Sovereign Risk: LOW — the weights come from Nous Research in the US, are openly available, and require no mandatory third-party API access at the source, even though running it in the cloud via a third-party provider naturally brings that provider’s jurisdiction into play.

Header Notes: Stability and Reliability

For a Frontier-class cloud model, this is no badge of honor, but it is not yet cause for alarm. A single timeout across 49 runs is, above all, a reliability finding about the service itself when dealing with an API or proxy endpoint. This is precisely what matters for production agent systems: not the quality of a single successful run, but whether the endpoint responds reliably under load. Hermes 4 405B remains usable here, but not without caveats.

Architecture and Expectations

The classification as an instruct model with optional thinking captures the character fairly well. Out of the box, Hermes 4 405B responds more like an obedient, goal-directed assistant than a demonstratively deliberate reasoning system. In the benchmark, this is not a flaw — it is the correct measuring stick: standard mode was tested because that is exactly how a typical user first encounters the model via the API. The fact that Hermes is supposed to support extended thinking in principle was not separately activated in this run; the mode is accordingly set to n/a, meaning standard behavior of a cloud endpoint without a thinking toggle.

More important is the second classification: Generalist, Frontier, Dense. Generalist means excuses based on specialization do not apply. Frontier means expectations are at their maximum. And Dense means all 405 billion parameters are active for every response. There is no MoE discount here, no “only a few experts were actually active.” When a model like this delivers only mediocrity in core disciplines, that can and should be stated without flinching.

Performance and Cost Profile

The “Interactive Tool Expert” badge is reasonably apt. Hermes 4 405B does not feel like a sprint model — more like an assistant that is structured enough for tool and formatting tasks while remaining interactive. The measured generation speed should be read as an infrastructure value from OpenRouter, not as an abstract property of the weights model alone. With Cloud Open Weights, speed is always also a product of the provider, its routing, and its backend capacity. Readers should therefore understand these values as an endpoint profile, not as some immutable characteristic of the model.

One positive observation: Hermes 4 405B works in a token-economical manner. Not a single module exceeds the expected verbosity range. In fact, CLI, Code Quality, documentation, UX, and cultural tasks all come in below the fleet median. The model does not ramble. It saves words where other systems first put their own thinking on display. This is welcome for cost and latency. It does not, however, rescue a weak answer.

Code Quality and Security: Good Checklist, Rarely a Real Audit

In the code and security domain, Hermes 4 405B shows perhaps the most sympathetic and simultaneously most frustrating side of its profile. Sympathetic, because it works in a structured manner, produces clean tables, and does not systematically miss relevant vulnerabilities. Frustrating, because the analysis frequently stops at the surface, right where a serious auditor would already be reaching for the screwdriver.

The exemplary security audit of a vulnerable web application illustrates this very clearly. Hermes identifies 19 vulnerabilities, which at first glance hits the expected level of completeness. That is not nothing. The table is readable, the categorization is broadly usable, and the most important classics — from SQL Injection to XSS to IDOR — all appear. Anyone wanting a quick triage gets actionable material.

But then comes the second look, and it is less flattering. The explanations frequently remain abstract, almost textbook-like. Instead of a concrete payload for SQL Injection, the model delivers formulations like “direct insertion of user input into SQL query.” That is factually correct and operationally thin. A penetration tester does not need to be told that fire is hot. They need to know where it is burning, how it spreads, and which door closes first. That is precisely where Hermes 4 405B leaves points on the table. Fix recommendations often stay at the buzzword level — “prepared statements,” “whitelist,” or password_hash() — where other models already provide reproducible patch context.

Even more critical is the severity assessment itself. Several vulnerabilities are rated too leniently. Path Traversal only as high rather than critical, Session Fixation too low, Type Juggling without the necessary sharpness. This is not an academic cosmetic flaw — it changes priorities in real-world operations. Whoever sorts risks incorrectly manages them incorrectly. Hermes 4 405B here is more a competent junior-level analyst than an experienced incident responder.

That said, the module should not be written off entirely. The rule-based values show solid error detection alongside weak solution depth. Put differently: the model sees a lot but explains too little. For initial analyses, review checklists, and broad vulnerability inventories, it is useful. For remediation-grade security work, human follow-up or a sharper model is required.

Logic and Reasoning: Correct Result, Restless Mind

In reasoning, the downside of combining an instruct character with optional thinking becomes particularly apparent. Hermes 4 405B can solve classic logic problems. It answers the guard post puzzle correctly. The mathematical structure is understood, the standard solution is right. The problem is not that the model ends up wrong. The problem is the path it takes to get there.

The reasoning trace is long, meandering, and self-doubting. The model repeatedly works through the same logical path, briefly contradicts itself, and then recovers. This reads less like cool deduction and more like someone thinking very loudly in their head, hoping to catch the right turn in time. For simple logic, this may be tolerable. For complex multi-step tasks, it is a warning sign.

There is also a structural language problem. In several tasks in the metacognition section, Hermes 4 405B responded in English despite an explicit German-language instruction. This language failure is not an isolated outlier. Across multiple reasoning tasks, the model shows a consistent pattern: when simultaneous constraints on language, length, and format are given, it drops the language constraint first. Specifically in metacognition tasks with a <thought> structure and a German target output, Hermes reliably switches to English. For production environments with a fixed output language, this is a genuine instruction-following problem, not a mere cosmetic flaw.

Metacognition Compliance (Reasoning): The model refuses to use the explicitly requested <thought> tags in 3/5 metacog tests, citing a consistent policy statement. The reasoning content itself is partially correct — the score deduction results from the format refusal, not from reasoning errors. For comparison: in the tag-free reasoning_5* tests, the model achieves an average score of approximately 61.7%, consistent with its general performance level. CrucibleMark deliberately evaluates native zero-shot instruction compliance as a real-world characteristic — this deduction is methodologically intentional.

In addition, there are the automatic constraint violations in the same module. Three metacognition tasks were flagged for language mismatch. The pipeline treats this as a hard violation of the language instruction. Rightly so: if a system reads “respond in German” and delivers English, the quality of the content is a secondary concern. In compliance-bound workflows, the response fails at the gate before anyone can applaud its reasoning.

Content Transformation: Strong at Restructuring, Not Always Elegant

In transformation tasks, Hermes 4 405B shows one of its better faces. The long YouTube script on setting up 2FA demonstrates that the model understands production logic. It separates analysis from transformation, works with timecodes, sets visual stage directions, incorporates pattern interrupts, and keeps the tone conversational. The result is usable enough that, with human editing, it could actually become a publishable script.

Noteworthy is how precisely Hermes hits the production language. Cues like [SHOW], [CLICK], [CIRCLE], [B-ROLL], [MUSIC], or [TEXT ON SCREEN] are not scattered decoratively but used in context. Add to that retention hooks, a CTA, and even an Easter egg for community engagement. This is not a literary achievement, but it understands the format. Some models write about video. Hermes writes more in video.

The weaknesses lie in the fine-tuning. The troubleshooting passage is integrated functionally but not as cleanly presented as its own distinct block as in the reference. Individual sentences occasionally overshoot the desired spoken-word brevity. These are not serious defects. They simply show that in this module, Hermes commands production craft more than editorial elegance.

UX Writing, Documentation, and Language Sensitivity: Functional, Often Somewhat Cold

The benchmark score in UX Writing is only moderate, and the qualitative logs offer a plausible explanation. Hermes 4 405B frequently formulates correctly, but not with particular finesse. Especially in German-language, tone-sensitive tasks, it often lacks the final nuance between “formally correct” and “communicatively smart.” The model tends to write like a list where people expect an invitation.

This is also evident in the inclusive rewrite of a job posting. Hermes reliably removes toxic or exclusionary phrasing. “Ninja” becomes “dedicated professional,” martial buzzwords become more neutral competencies. So far, so adequate. But then it makes the classic mistake many models make: after a neutral opening, it reintroduces a gendered marker with “Expertin.” This is exactly the kind of carelessness that immediately stands out in diversity-sensitive texts. Not malicious — just imprecise. And imprecision is already the error in tasks like these.

There is also the matter of tone. Hermes formulates in a transactional rather than inviting manner. The prose feels like a checklist with a polite surface. Modern HR communication, however, lives on the implicit signal “you are welcome here,” not merely on the semantic removal of problematic words. The model sanitizes language but does not truly curate it. That is the difference between text laundering and editing.

Cultural Intelligence: Decent Instincts, Limited Warmth

In the Cultural Intelligence module, Hermes 4 405B does not perform crudely. The rule-based values are solid, linguistic correctness is high, and problematic original phrasings are mostly defused cleanly. The model understands the assignment of producing culturally sensitive or inclusive language. That is the good news.

The bad news: it frequently stops at technical compliance. The best version of these tasks demands not just defusing but social intelligence in tone. Hermes produces clean, somewhat bloodless compromises. For internal revisions or as a first draft, this is useful. For external communications where tone and stance are part of the message, a human should still review it.

Tool Use and Hallucinations: This Is Where It Gets Serious

The hardest substantive flaw in this run lies not in stylistic questions but in hallucination resistance on tool tasks. Two tool-use assets were explicitly flagged as hallucinations. The model generated content that did not originate from the retrieved tool result but was fabricated. The score was consequently capped by a hallucination cap. This is not an ordinary point deduction — it is a disqualifying signal for content-critical applications.

Precisely because Hermes 4 405B is positioned as a controllable, tool-friendly model, this finding carries significant weight. A tool model may be creative when creativity is called for. In research, extraction, or fact-bound reporting, it may not. A model that supplements tool outputs at will is no longer an assistant — it is an improv theater with JSON syntax.

This does not mean Hermes is unusable in every tool case. The general tool execution score is high enough to acknowledge structural capability. But the qualitative finding is clear: as soon as external factual sources are authoritative, this model needs tight guardrails. Verification, source mirroring, and a second review step when in doubt are mandatory.

Data Privacy and Data Sovereignty

The situation here is unusual but important for European users. The calculated Sovereign Risk is LOW because Nous Research makes the weights openly available and, based on available information, does not operate its own public API as a mandatory intermediary. This significantly reduces the source risk of the model weights.

For the specific deployment in this test, however, the following applies: Hermes 4 405B ran as a Cloud Open Weights model via OpenRouter. This means actual data sovereignty depends on the third-party provider’s deployment infrastructure, not on Nous Research alone. No verified provider data was available here regarding that deployment infrastructure. The Nous vendor card lists applicable law as N/A (local/self-hosted), data location as “Local or third-party hoster,” data retention as 0 days, and no GDPR DPA. For organizations with strict GDPR obligations, this is not an automatic disqualifier, but it is not a free pass either. Without reliable information from the specific cloud provider regarding jurisdiction, DPA, and data residency, the compliance picture remains incompletely documented.

Conclusion

Hermes 4 405B is an interesting but contradictory Frontier model. As a dense 405B generalist, it should carry itself with effortless authority. Instead, it comes across as a highly capable assistant that plays to its size in some disciplines while leaving a surprising amount of precision on the table in others. Its strengths are structured outputs, solid content transformation, reliable tool orientation, and a pleasingly economical use of tokens. Its weaknesses are at least equally clear: security explanations that are too shallow, inconsistent language compliance in reasoning, mediocre UX instincts, and — most critically — documented hallucinations in tool-bound tasks.

So what is it actually suited for? Hermes 4 405B is well suited for drafting work, structured text production, broad analysis checklists, and interactive tasks where a human reviews the output regardless. It is less suited for unsupervised fact-critical agents, multilingual strictly regulated workflows, and security work where not just the right heading but correct prioritization and concrete countermeasures are what count.

In short: Hermes 4 405B has substance, but no free pass. A large open model that is often useful and must therefore be judged rigorously. Frontier does not automatically mean sovereign. Here it means, above all: capable, promptable, controllable — but not yet trustworthy enough without oversight.

This evaluation was generated automatically based on the benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.