Tool-use review
Created · Long Context
Deployment Verdict
Created on: 14.06.2026, 16:16:02
Conditional deploy, because Devstral 2 showed no hallucinations but produced invalid tool calls and, with a combined score of 48.50, remains clearly below the threshold for trustworthy standard orchestration.
Tool Execution Profile
The core weakness lies not in accessing tools per se, but in the selection and form of the call. In the Web Search & Tool Selection test — which checks without explicit hints whether web_search is needed instead of fetch — the model only unreliably recognizes the required tool switch. This argues against genuine tool intelligence in open pipelines. In the URL Construction test, which measures deriving a target URL from internal knowledge and then executing a fetch, it performs noticeably better. The pattern therefore does not look like general tool incapability, but rather like a somewhat rigid approach: known direct retrieval paths work; context-dependent tool selection does not hold up consistently. Since a retry was required and the tool call is marked as invalid, this points more to a protocol or format issue than to a pure content-understanding problem. For MCP pipelines this is nonetheless critical, because orchestrators require deterministic call structures.
Synthesis Fidelity
How well does it consolidate tool results? Weakly. P2 at 40 shows that Devstral 2 only partially merges retrieved content cleanly. This is most visible in HTTP Fetch & Extract, which measures precise fact extraction from real page content, and in Multilingual Search & Synthesis, which tests cross-language research with a German-language summary. In both cases, consolidation essentially breaks down.
Does it stay within the tool result or fall back on training data? Here the trust signal is better than the synthesis quality. In the Honeypot EU License Research test — which checks whether current license restrictions are actually retrieved from web sources — the model stayed bound to the tool path. No hallucination, Content Verification State A. For compliance-adjacent workflows this matters more than the raw P2 score.
Error Resilience
In the 404 test, which measures whether a failed tool call is handled transparently, Devstral 2 responds in a production-appropriate way. It communicates the error rather than fabricating page content. P2 80 and no hallucination despite a 404 are a reliable positive signal. For production systems this is acceptable, because a visible error can be corrected — fabricated fallback content cannot.
Operational Profile
Total 57.71s per run. Model calls 4.74s and 4.51s, MCP latency 0.37s. Overall slow. Cost 0.004757 per run. Inexpensive. The price fits; the performance does not.
Conclusion & Recommendation
Suitable for tightly controlled coding or retrieval pipelines with strong external tool steering, hard schema validation, and automatic retries. Not suitable for open MCP setups in which the model must independently switch between search, fetch, and synthesis. Anyone deploying it should make tool selection outside the model and treat the response as downstream processing — not as reliable agent control.
This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.
