Skip to content
CrucibleMark

GLM-5.1

GLM-5.1 is Z.AI’s post-training upgrade with 754 billion total and 40 billion active parameters in a MoE architecture, optimized for long-horizon agentic coding workflows with up to eight hours of autonomous execution. The context window spans 200,000 tokens, and the weights are available as an Open Weights model under the MIT license.

Zhipu AI Version 5.1 Commercial use permitted MoE 754 B (40 B active) 200 K Context 12/2025 $1.05 / $3.5 per 1M

  • Open Weights
  • Frontier
  • OpenRouter
  • Text
  • Instruction-Tuned
  • Agentic Orchestrator
  • Batch

Sovereign Risk: HIGH Z.AI (formerly Zhipu AI) is a Chinese company and subject to China’s National Security Law (NSL), which can enable state access to data. In February 2025, Germany’s BSI explicitly warned against the use of Chinese AI cloud services (BSI reference: Warning DeepSeek, 04.02.2025); this risk assessment applies analogously to all Chinese cloud AI providers that process user data on Chinese servers. With purely local inference, the Cloud Act-equivalent risk does not apply.

Tool-use profile: 6 assets in detail

Comparison of asset performance (P1/P2/Combined) against the fleet average

Asset performance (radar)

Score Breakdown vs. Fleet Average

Tool-use details

Asset performance, reliability, and runtime profile

Name P1 P2 Combined
EU License Research 100 40 64
HTTP Fetch & Extract 80 35 57.5
Tool Failure Handling (404) 40 60 44
Web Search & Tool Selection 100 80 91
URL Construction & Fetch 80 60 70
Multilingual Search & Synthesis 100 60 80

CrucibleMark evaluates tool use across 6 independent tests. Click a test name for details.

Reliability

  • Tool Call Valid: No
  • Retry: Not required
  • Hallucination: Detected

Reliability measures how consistently a model actually executes tool calls: Tool Call Valid schema and format accepted, Retry Required successful only after retry, Hallucination Flag fabricated tools or parameters detected. All three green means production-ready.

Runtime profile

Call 1
6.27
First Request
MCP
2.78
Protocol Latency
Synthesis
45.12
Response Generation
Total
325.02
Sum of All Phases
Token
18177
Input + Output
Cost
$0
Cost per Run

The runtime profile shows the latency and cost metrics for the model run: Call 1 First Request, MCP Protocol Latency, Synthesis Response Generation, Total sum of all phases, supplemented by Token input and output and Cost cost per run.

Tool-use review

Updated · Instruction-Tuned · Agentic Orchestrator

Deployment Verdict

Conditional deploy, because tool execution is strong, but one invalid tool call and detected hallucination limit confidence in production MCP pipelines. The combined finding is therefore only viable for controlled tool infrastructures.

Tool Execution Profile

GLM-5.1 demonstrates genuine tool intelligence rather than mere schema imitation. On the Web Search & Tool Selection test — which checks whether the model selects search over fetch without being prompted — it makes the correct decision reliably. That is a strong signal for agentic orchestration. On the URL Construction test, which measures independent derivation of the target URL, it performs adequately but not precisely enough for deterministic fetch pipelines. The P1 score remains high; protocol adherence does not. The fact that the overall finding comes out as tool_call_valid=false is therefore relevant: the model frequently finds the right working path but does not consistently produce formally valid calls. Retry was not required, so the issue lies more in isolated call validity than in systematic MCP format failure.

Synthesis Fidelity

How well does it consolidate tool results? Only with limited reliability. P2 performance at 59.17 is the clear weak point. Particularly on HTTP Fetch & Extract — where precise extraction from real page content is required — the model loses accuracy. It can merge results, but not stably enough for pipelines in which proper nouns, years, or license details must be reproduced verbatim.

Does it stay within the tool result or fall back on training data? On the Honeypot EU License Research test, which checks whether current license restrictions are answered from web sources rather than prior knowledge, it does not hallucinate. That is a positive finding. Nevertheless, the global hallucination result remains a security risk: once a model in a tool pipeline outputs fabricated facts as retrieved results, the entire infrastructure loses its auditability.

Error Resilience

On the 404 test, which forces transparent handling of a failed tool call, GLM-5.1 stays on the acceptable side. It does not fabricate page content and thus responds to obvious tool failures in a production-viable manner. The P2 score of 60 also indicates only average clarity in error communication here. For robust systems that is sufficient; for heavily automated downstream steps without human oversight it is not ideal.

Operational Profile

Total 325.02s. Call 1 6.27s. MCP latency 2.78s. Call 2 45.12s. Slow. Cost per run: local. Pricing: $1.05/1M input, $3.5/1M output. For the performance shown, tail latency is the critical operational disadvantage.

Conclusion & Recommendation

Suitable for supervised research and orchestration pipelines in which the model is expected to select tools, span search paths, and consolidate multilingual results. Not suitable for compliance, extract-and-trust, or fully automated fetch pipelines in which every tool result must be called formally correctly and condensed with precision. Anyone deploying GLM-5.1 should enforce strict tool call validation, response verification, and tight guardrails ahead of downstream actions.

This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.