Mistral Medium 3.5

Mistral Medium 3.5 is Mistral AI’s Frontier model focused on agentic workflows and coding. With 128 billion parameters, the model operates with a context window of 256,000 tokens and supports multimodal inputs for text and image. Available as an Open Weights model under a Modified MIT license for local use or via cloud API, from a European provider environment with GDPR compliance.

Mistral AI Version 3.5 Commercial use permitted Dense 128 B 256 K Context 12/2025 $1.5 / $7.5 per 1M

Open Weights
Server
Mistral AI
Text
Vision
Instruction-Tuned
Real-Time

Sovereign Risk: LOW Mistral AI is headquartered in France. As an EU company, the storage and access to the weights are not subject to US CLOUD Act provisions or Chinese security laws.

Tool-use details

Asset performance, reliability, and runtime profile

Name	P1	P2	Combined
EU License Research	100	40	64
HTTP Fetch & Extract	80	35	57.5
Tool Failure Handling (404)	80	60	73
Web Search & Tool Selection	100	35	70.8
URL Construction & Fetch	80	80	80
Multilingual Search & Synthesis	100	60	80

Name

Combined

EU License Research

100

HTTP Fetch & Extract

57.5

Tool Failure Handling (404)

Web Search & Tool Selection

100

70.8

URL Construction & Fetch

Multilingual Search & Synthesis

100

CrucibleMark evaluates tool use across 6 independent tests. Click a test name for details.

Reliability

Tool Call Valid: No

Retry: Not required

Hallucination: Detected

Reliability measures how consistently a model actually executes tool calls: Tool Call Valid schema and format accepted, Retry Required successful only after retry, Hallucination Flag fabricated tools or parameters detected. All three green means production-ready.

Runtime profile

Call 1: 0.64; First Request

MCP: 2.04; Protocol Latency

Synthesis: 3.84; Response Generation

Total: 39.11; Sum of All Phases

Token: 9593; Input + Output

Cost: $0; Cost per Run

The runtime profile shows the latency and cost metrics for the model run: Call 1 First Request, MCP Protocol Latency, Synthesis Response Generation, Total sum of all phases, supplemented by Token input and output and Cost cost per run.

Deployment Verdict

Conditional deploy, because tool execution is strong, but one invalid tool call and a detected hallucination limit confidence in unsupervised MCP pipelines. The overall impression is good, but not robust enough for high-trust automation.

Tool Execution Profile

Mistral Medium 3.5 demonstrates genuine tool-selection competence, not just rigid procedural behavior. On the Web Search and Tool Selection test — which checks whether the model chooses between search and direct fetch without an explicit hint — it selects the correct tool confidently. This points to usable orchestration logic. On the URL Construction test, which measures autonomous derivation of a target URL followed by a fetch, it remains usable but less precise. This is the typical point where production pipelines need deterministic guardrails.

The critical finding is that the tool call was not consistently valid. Since no retry was required, this does not look like a mere formatting issue followed by self-correction, but rather a one-off protocol or parameter slip that the runtime did not catch. For MCP this means: good tool intelligence, but no consistently reliable protocol hygiene.

Synthesis Fidelity

How well does it condense tool results? Only limitedly reliable. The P2 score of 59.17 is the clear weak point. Particularly on EU License Research — which is supposed to consolidate current license restrictions from web sources — and on HTTP Fetch & Extract, which measures precise factual accuracy from fetched content, the model loses accuracy and condensation sharpness. It finds material, but the last mile of dependable summarization is not consistently strong.

Does it stay within the tool result or fall back on training data? On the Honeypot EU License Research test, which checks exactly this behavior, it does not hallucinate. That is the most important trust anchor in this report. At the same time, the global hallucination finding is a security risk: once a model outputs fabricated facts as a tool result inside a tool pipeline, it is not just one answer that goes wrong — the infrastructure itself becomes unreliable.

Error Resilience

On the 404 test — which checks whether a failed tool call is communicated transparently or whether page content is invented — the model responds acceptably. Despite the error, it does not hallucinate substitute content. Error communication is therefore production-ready, even if it is not particularly well-condensed or actionable.

Operational Profile

Total 39.11s per run. MCP latency 2.04s. Model calls 0.64s and 3.84s. Slow for the quality delivered. Cost: local. Inexpensive to operate, but time per run is high for interactive tool pipelines.

Conclusion & Recommendation

Suitable for research-oriented MCP pipelines with human oversight, for multilingual sourcing, discovery steps, and tool selection ahead of downstream validation. Not suitable for compliance, license review, regulatory responses, or any other path where the synthesis itself serves as a dependable final finding. Deploy only with schema validation, tool-call guardrails, and a second instance for result verification.

This analysis was generated automatically based on benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.