Skip to content
CrucibleMark

Hermes 4.3 36B

Hermes 4.3 36B (NVFP4) is the compressed variant of NousResearch’s Open Weights model based on ByteDance Seed-OSS-36B, fully decentralized post-trained via the Psyche network. Fireworks compressed the BF16 checkpoint to NVFP4, significantly reducing the memory footprint of the weights. 36 billion dense parameters, 512,000 tokens of native context, optional thinking mode, and an Apache 2.0 license for free commercial use.

NousResearch Version 4.3 Commercial use permitted Dense 36 B 512 K Context 01/2025 locally tested

  • Open Weights
  • Server
  • VSPK
  • Text
  • 36B
  • NVFP4
  • Compressed-Tensors
  • 512K-Context
  • Long Context
  • Agentic Orchestrator
  • Batch

Sovereign Risk: MEDIUM Firworks/Hermes-4.3-36B-nvfp4 is an NVFP4 quantization (LLM-Compressor, long-seq calibration on Rombo-Org/Optimized_Reasoning) of the NousResearch base weights (Hermes-4.3-36B on ByteDance Seed-OSS-36B-Base, decentrally post-trained via Psyche). Apache-2.0 license, publicly available on HuggingFace. NousResearch is subject to the US CLOUD Act; with fully local vLLM deployment there is no cloud data egress, risk remains moderate.

Tool-use profile: 6 assets in detail

Comparison of asset performance (P1/P2/Combined) against the fleet average

Asset performance (radar)

Score Breakdown vs. Fleet Average

Tool-use details

Asset performance, reliability, and runtime profile

Name P1 P2 Combined
EU License Research 100 60 76
HTTP Fetch & Extract 80 35 57.5
Tool Failure Handling (404) 80 80 80
Web Search & Tool Selection 100 35 70.8
URL Construction & Fetch 80 35 57.5
Multilingual Search & Synthesis 100 15 57.5

CrucibleMark evaluates tool use across 6 independent tests. Click a test name for details.

Reliability

  • Tool Call Valid: No
  • Retry: Not required
  • Hallucination: Detected

Reliability measures how consistently a model actually executes tool calls: Tool Call Valid schema and format accepted, Retry Required successful only after retry, Hallucination Flag fabricated tools or parameters detected. All three green means production-ready.

Runtime profile

Call 1
6.11
First Request
MCP
1.53
Protocol Latency
Synthesis
50.1
Response Generation
Total
346.45
Sum of All Phases
Token
9540
Input + Output
Cost
$0
Cost per Run

The runtime profile shows the latency and cost metrics for the model run: Call 1 First Request, MCP Protocol Latency, Synthesis Response Generation, Total sum of all phases, supplemented by Token input and output and Cost cost per run.

Tool-use review

Created · 36B · NVFP4 · Compressed-Tensors · 512K-Context · Long Context · Agentic Orchestrator

Deployment Verdict

Deploy conditionally, because tool execution is strong, but synthesis fidelity — given the hallucination finding and invalid tool call — does not carry enough confidence for unattended production pipelines.

Tool Execution Profile

Hermes 4.3 36B demonstrates genuine tool intelligence, not merely rigid call patterns. On the Web Search & Tool Selection test, it reliably recognizes — without explicit prompting — that a search is required before a direct fetch. This points to usable planning logic in dynamic MCP workflows. On the URL Construction test, which measures correct URL derivation followed by a subsequent fetch, it remains usable but not deterministic enough. P1 80 means here: mostly functional, but not precise enough for fragile pipelines with hard URL dependencies. The overall finding that at least one tool call was invalid remains critical. Since no retry was required, this looks less like a mere formatting issue and more like a localized protocol or execution uncertainty.

Synthesis Fidelity

How well does it consolidate tool results? Only with limited reliability. The P2 score of 43.33 is the clear weak point of this model. Particularly on HTTP Fetch & Extract, Web Search & Tool Selection, URL Construction & Fetch, and especially Multilingual Search & Synthesis, it consolidates results too coarsely, loses detail, or draws the wrong conclusions. For architectures in which the model is expected to translate tool output into reliable working answers, this represents an operational risk.

Does it stay within the tool result or fall back on training? On the Honeypot EU License Research test — which checks whether current license restrictions are genuinely drawn from web sources — the model stays on the safe side. It does not hallucinate there and produces a usable trust signal. The global hallucination finding nevertheless remains a security risk: once a model outputs fabricated content as purported tool results, it is not just a single answer that degrades — the reliability of the entire tool infrastructure is compromised.

Error Resilience

On the 404 test, which checks for transparent behavior when a tool call fails, Hermes 4.3 36B responds in a production-appropriate manner. It communicates the error openly and does not fabricate page content. This is acceptable for production systems and matters more than cosmetic response quality.

Operational Profile

Total 346.45s per run. Call 1 6.11s, MCP latency 1.53s, Call 2 50.10s. Clearly slow for the quality delivered. Costs are local. Economically viable only when Open Weights operation, data control, or 512K context outweigh throughput requirements.

Conclusion & Recommendation

Suitable for locally operated MCP pipelines with human-in-the-loop, transparent error handling, and a clear separation between tool execution and final sign-off. Not suitable for autonomous research, compliance, or multilingual pipelines in which the response itself must serve as a reliable consolidation of tool output. If you deploy it, use it as an orchestrator with downstream verification — not as the final authority for factual summarization.

This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.