Skip to content
CrucibleMark

Mistral Medium 3.5

Mistral Medium 3.5 is Mistral AI’s Frontier model focused on agentic workflows and coding. With 128 billion parameters, the model operates with a context window of 256,000 tokens and supports multimodal inputs for text and image. Available as an Open Weights model under a Modified MIT license for local use or via cloud API, from a European provider environment with GDPR compliance.

Mistral AI Version 3.5 Commercial use permitted Dense 128 B (128 B active) 256 K Context 12/2025 $1.5 / $7.5 per 1M

  • Open Weights
  • Frontier
  • API
  • Text
  • Vision
  • Instruction-Tuned
  • Real-Time

Tool-use profile: 6 assets in detail

Comparison of asset performance (P1/P2/Combined) against the fleet average

Asset performance (radar)

Score Breakdown vs. Fleet Average

Tool-use details

Asset performance, reliability, and runtime profile

Name P1 P2 Combined
EU License Research 100 40 64
HTTP Fetch & Extract 80 100 90
Tool Failure Handling (404) 80 60 73
Web Search & Tool Selection 100 80 91
URL Construction & Fetch 80 35 57.5
Multilingual Search & Synthesis 100 40 70

CrucibleMark evaluates tool use across 6 independent tests. Click a test name for details.

Reliability

  • Tool Call Valid: Yes
  • Retry: Not required
  • Hallucination: Detected

Reliability measures how consistently a model actually executes tool calls: Tool Call Valid schema and format accepted, Retry Required successful only after retry, Hallucination Flag fabricated tools or parameters detected. All three green means production-ready.

Runtime profile

Call 1
0.57
First Request
MCP
0.94
Protocol Latency
Synthesis
2.67
Response Generation
Total
25.09
Sum of All Phases
Token
10645
Input + Output
Cost
$0
Cost per Run

The runtime profile shows the latency and cost metrics for the model run: Call 1 First Request, MCP Protocol Latency, Synthesis Response Generation, Total sum of all phases, supplemented by Token input and output and Cost cost per run.

Tool-use review

· Instruction-Tuned

Deployment Verdict

Created on: 14.06.2026, 16:15:27

Deploy conditionally, because tool execution is strong and protocol-clean, but synthesis fidelity drifts away from verifiable tool content too often, limiting confidence in downstream decisions.

Tool Execution Profile

Mistral Medium 3.5 behaves production-ready on the MCP side. Tool calls are valid, retry was not required, and the model shows no signs of pure schema-following. On the Web Search & Tool Selection test — which checks whether the model selects a search tool instead of fetch without being prompted — it makes the correct tool choice reliably. This speaks to genuine tool intelligence in open pipelines.

Weaker is the precision on the URL Construction test, which checks whether the model can correctly derive a target URL from its own knowledge and then execute fetch. Here, execution is sufficient for usable results, but not for deterministic flows with hard expectations around exact endpoints. The pattern is clear: when the environment provides the right tool and target resolution is open-ended, the model performs strongly. When it must derive a concrete URL itself, reliability drops.

Synthesis Fidelity

How well does it condense tool results? Only partially reliable. The P2 score of 59.17 fits the asset picture: perfect condensation on HTTP Fetch & Extract, but notable weaknesses on EU License Research, URL Construction & Fetch, and Multilingual Search & Synthesis. The model can cleanly extract facts from available content. It is less reliable, however, at consolidating multiple tool results closely against the evidence base and clearly flagging uncertainty.

Does it stay within the tool result or fall back on training? On the honeypot EU License Research test — which checks exactly this behavior — it does not hallucinate, and the verification status is strong. Nevertheless, the low P2 score is a warning signal: the model stays formally within safe territory but does not condense the researched license information precisely enough for compliance-adjacent use. Since hallucination was detected globally, this is a security risk, not merely a quality shortcoming. In a tool pipeline, fabricated or overextended output undermines the evidence chain of the entire infrastructure.

Error Resilience

On the 404 test, the model responds acceptably. It communicates the tool error transparently and does not fabricate page content. The P2 score of 60 indicates that the error message is not always condensed ideally, but the behavior remains production-capable. For operations, what matters most here is that no substitute facts are constructed when a retrieval fails.

Sovereignty Profile

Locally deployable and therefore attractive for sovereign deployments. At the same time, the model sits only 1.37 points below the fleet average of 67.84. For an open-weights, locally deployable Server model, that is a strong operational argument.

Conclusion & Recommendation

Suitable for MCP-backed pipelines where tool selection, retrieval, and structured extraction matter more than high-precision final synthesis: research assistance, technical information gathering, pre-processing for human review. Not the first choice for compliance, policy interpretation, multilingual evidence synthesis, or other flows where every result must stay tightly grounded in tool evidence. Deploy only with response grounding, source display, and a validator at the final synthesis stage.

This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.