Skip to content
CrucibleMark

Hermes 4.3 36B (Thinking)

Hermes 4.3 36B (NVFP4) is the compressed variant of NousResearch’s Open Weights model based on ByteDance Seed-OSS-36B, fully decentralized post-trained via the Psyche network. Fireworks compressed the BF16 checkpoint to NVFP4, significantly reducing the memory footprint of the weights. 36 billion dense parameters, 512,000 tokens of native context, optional thinking mode, and an Apache 2.0 license for free commercial use.

NousResearch Version 4.3 Commercial use permitted Dense 36 B 512 K Context 01/2025 locally tested

  • Open Weights
  • Server
  • VSPK
  • Text
  • 36B
  • NVFP4
  • Compressed-Tensors
  • 512K-Context
  • Long Context
  • Agentic Orchestrator
  • Batch

Sovereign Risk: MEDIUM Firworks/Hermes-4.3-36B-nvfp4 is an NVFP4 quantization (LLM-Compressor, long-seq calibration on Rombo-Org/Optimized_Reasoning) of the NousResearch base weights (Hermes-4.3-36B on ByteDance Seed-OSS-36B-Base, decentrally post-trained via Psyche). Apache-2.0 license, publicly available on HuggingFace. NousResearch is subject to the US CLOUD Act; with fully local vLLM deployment there is no cloud data egress, risk remains moderate.

Tool-use profile: 6 assets in detail

Comparison of asset performance (P1/P2/Combined) against the fleet average

Asset performance (radar)

Score Breakdown vs. Fleet Average

Tool-use details

Asset performance, reliability, and runtime profile

Name P1 P2 Combined
EU License Research 100 60 76
HTTP Fetch & Extract 80 35 57.5
Tool Failure Handling (404) 80 60 73
Web Search & Tool Selection 100 35 70.8
URL Construction & Fetch 80 60 70
Multilingual Search & Synthesis 100 15 57.5

CrucibleMark evaluates tool use across 6 independent tests. Click a test name for details.

Reliability

  • Tool Call Valid: No
  • Retry: Not required
  • Hallucination: Detected

Reliability measures how consistently a model actually executes tool calls: Tool Call Valid schema and format accepted, Retry Required successful only after retry, Hallucination Flag fabricated tools or parameters detected. All three green means production-ready.

Runtime profile

Call 1
5.22
First Request
MCP
1.23
Protocol Latency
Synthesis
39.48
Response Generation
Total
275.54
Sum of All Phases
Token
8775
Input + Output
Cost
$0
Cost per Run

The runtime profile shows the latency and cost metrics for the model run: Call 1 First Request, MCP Protocol Latency, Synthesis Response Generation, Total sum of all phases, supplemented by Token input and output and Cost cost per run.

Tool-use review

Created · 36B · NVFP4 · Compressed-Tensors · 512K-Context · Long Context · Agentic Orchestrator

Deployment Verdict

Conditional deploy, because tool execution is strong, but the invalid tool call and detected hallucination limit confidence in production MCP pipelines. The overall impression is usable, but not cleared for unsupervised decisions.

Tool Execution Profile

Hermes 4.3 36B demonstrates genuine tool intelligence rather than mere template usage. In the Web Search & Tool Selection test — which checks the choice between search and direct retrieval without an explicit hint — it correctly identifies that web_search is needed first. This speaks to workable orchestration in open-ended tasks. In the URL Construction test, which measures the derivation of a target URL from prior knowledge and the subsequent fetch, it is still solid but less precise. P1 80 here means: functional, but not deterministic enough for fragile fetch paths.

The tool_call_valid=false finding is critical. This does not mean the model fundamentally misunderstands tools. It means protocol compliance in the specific run was not stable enough. Since no retry was required, this looks more like a singular formatting or call error than a recurring loop problem. For MCP infrastructures, it nonetheless remains an integration risk.

Synthesis Fidelity

How well does it condense tool results? Only limitedly reliable. P2 performance is the clear weak point. HTTP Fetch & Extract and Multilingual Search & Synthesis in particular show that the model does not consistently translate retrieved content into reliable response text. It often finds the source, but loses details, priorities, or cross-lingual accuracy during condensation.

Does it stay within the tool result or fall back on training? In the Honeypot EU License Research test — which checks whether current license restrictions are actually retrieved from web sources — it stays within acceptable bounds and does not hallucinate. This is the most important trust signal. The global hallucination finding remains a safety risk, however: once a model outputs fabricated facts as a tool result, it undermines the reliability of the entire pipeline.

Error Resilience

In the 404 test, which distinguishes transparent error communication from fabricated page content, the model responds acceptably. It does not hallucinate substitute content despite the failure. P2 60 is not elegant, but production-viable. For operational systems, this kind of honesty matters more than linguistic polish.

Operational Profile

Total 275.54s per run. Call 1: 5.22s. MCP latency: 1.23s. Call 2: 39.48s. Operated locally, so direct runtime costs are not relevant. Given the only moderate overall performance, this is slow.

Conclusion & Recommendation

Suitable for locally operated research and orchestration pipelines where a model needs to select tools and report errors transparently, but every content synthesis is still validated by validators, schema checks, or human review. Not suitable for compliance, policy, or executive summary pipelines where the model must precisely condense tool results and reproduce them without any fabricated additions. Those deploying Hermes 4.3 36B should treat it as a tool dispatcher with downstream verification — not as a trusted final authority.

This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.